Dependencies - Blessing or Bane? And possible solutions?

[Twyla]

Allow me to begin with:

A ) Request:

This can readily be a hot-button topic within the Sim City community, so I must reiterate that this is a public forum.  As such, the discussion needs to remain civil - in spite of how strongly one may be entrenched with either extreme.

B ) Recognition:

This discussion is in no way intended to be regarded as a slight against those who so readily offer up their hard work to the Sim City community - be it as LOTs, MODs, BATs, or whatever incarnation their contribution takes.  Mere words cannot fully express my respect and gratitude regarding the skill, artistry, and generosity of contributors.

And now, the discussion:

Will Wright's innocent little game, Sim City, has grown light-years beyond all possible expectations in the quarter century of its existence and turned an unknown fledgling game developer, Maxis, into a legend of the gaming industry.  The game succeeded for the very reason it almost never was - there simply is no other game like it!  Numerous communities, with countless members spanning the globe, remain glued to their monitors in support of a game that is older than many of them are - the newest incarnation is approaching a decade in age and is likely the oldest thing most of us own.  But until Maxis/EA releases a worthy successor, Sim City 4 will remain in our hearts the de-facto standard of what a Sim game should be.

Until a true Sim City 5 comes to pass, we have a plethora of options in expanding the bounds of Sim City 4 - far too many to even attempt to list.  We, as a community, indeed owe a great debt of gratitude to those who so generously invest their time and efforts developing both the expansions themselves and the tools to create them.

But herein lies the rub:

With so many expansion possibilities, a great deal of care must be taken on all fronts to:

• Avoid internal conflicts (ie - duplicated resource addresses)
• Minimize overhead
• Eliminate redundancies
• Simplify accessibility for the less-technically-inclined

in addition to maintaining the communal spirit which has allowed Sim City 4 to thrive as it has.  Yet, in some ways, we're victims of our own successes. 

Countless Mega-Packs of fabulous textures and intricate props enable individuals to create fantastic new content for the game - but only if the users already have these Mega-Packs.  If not, the user needs to find and acquire these dependencies - in some cases adding more than 20MB (a sizeable load for SC4's internal coding) in order to properly implement a 65KB lot.  For the average user, even one with a considerable amount of custom content, it's fair to say that far less than 30% of any particular Mega-Pack dependency ever gets utilized.

The simplest solution would be that content creations being offered include these dependencies - though this leads to other problems.  Increasing the likelihood of internal conflicts and redundancies, in addition to a lack of proper credit where it is due.


Given the availability of open-source installers and utilities (many of which have their own dependency headaches), it shouldn't be that difficult for us to develop a one-size-fits-all distribution installer/manager.

Such would actually consist of two programs: the developer utility and the user utility, each of which would be available as readily-locatable separate downloads:


The Developer Utility:
This program would organize all required assets (props, textures, models, etc) from their respective sources and compress it all into a single, compact file for distribution.  In doing so, it would preserve the hierarchy of those sources.

It would also distinguish between unique assets (those exclusive to this particular expansion) and common assets (ie - those distributed in Mega-Packs and reused by multiple expansions).

The User Utility:
This program would read the distribution file produced by the Developer Utility to reconstruct the source files, complete with reproducing the source hierarchy.  In doing so, it would check each asset in turn and, if absent, install it - if a particular asset already exists, it goes to the next.  This functionality should also include an over-write option (just in case).

The User Utility would:

⚫	Install the unique assets in the appropriate location
⚫	Notify the user that it uses a given dependency - giving the creator(s) proper credit

•	Check to see if the given dependency exists

⚬	If not, it creates the file with all necessary assets
⚬	If so, it checks each asset in turn - adding those not present (or overwriting, if enabled)

⚫	Repeat for each additional dependency

While probably not the best potential solution, it seems to cover all the bases:

• Eliminates having to hunt down and manually install multiple dependencies
• Eliminates cluttering things up with extraneous assets (minimizing overhead)
• Polices itself in regards to installing assets (eliminating duplications)
• Streamlines the distribution of custom content
• Simplifies things enormously for the user, making individual expansions far more appealing

If I were more adept with programming (as well as SC4's inner workings), I'd take a crack at this myself.  Though I'm sure there are dozens (if not hundreds) in this forum who could write these utilities in less time than I've spent describing them - preferably as self-sufficient programs without dependencies of their own.

[GDO29Anagram]

Your topic of discussion: Streamlining the process of downloading plugins.

The Developer Utility:
This program would organize all required assets (props, textures, models, etc) from their respective sources and compress it all into a single, compact file for distribution.  In doing so, it would preserve the hierarchy of those sources.

Kinda sounds like the DatPacker. Nifty for making prop packages or texture packages, for those wannabe or seasoned developers out there. Or to just cram all the junk in your Plugins folder into one single compact file so that it doesn't make a mess out of things.

Oftentimes, there's a readme that comes with these dependency packages, which includes things such as the developers in question, details on the items, and a link to a forums thread to report problems to or to suggest new things. The readme says, "READ ME!!!" (Why else is it called a readme? )

The User Utility:
This program would read the distribution file produced by the Developer Utility to reconstruct the source files, complete with reproducing the source hierarchy.  In doing so, it would check each asset in turn and, if absent, install it - if a particular asset already exists, it goes to the next.  This functionality should also include an over-write option (just in case).

Sounds like the Reader. Many of the plugins made for Simcity 4 (And many of the files that originally came with the game) are packaged into single files: DAT files. The Reader's able to read an entire DAT and organise it into the individual files that make it up. Plus you can edit the values set in them. (Actually it doesn't matter how they're organised within the DAT; The game can tell what's a texture, what's an exemplar, and what's a prop and so on; Organisation is for when you're looking THROUGH the DAT's contents.)

If you're DatPacking a number of plugins, their readmes (and all other irrelevant files) will be excluded, and the resulting DAT is placed in a "plugins_compressed" folder in the plugins folder (if you didn't specify another location). Some plugins (like the NAM) cannot be DatPacked because it can screw up any update processes. Others don't need any packing, as they're small enough already or deemed unnecessary.

---
Many people would balk at the sheer number of dependencies needed for a specific download. Problem is that in some cases, a certain plugin (Let's say it's a landmark) made by, for example purposes, myself, may use props made by one person, and textures made by another, and models made by yet another person. Two ways to simplify the download process:

- The unrecommended way: Copy all three things needed, pack it, and submit it on the file exchange in question, and face potential termination.

- The "Coalitionist" way: The three people and myself may unite into a group, unify their respective models, props, textures, et cetera, into a combined DAT and submit the Dependency packages. That way, if I (or another in that group) make more lots that require their props, people won't be burdened by downloading things they already have or have to download again. Dependency packages are one-time downloads, unless it needs to be updated; Everything you need is already available for everything that group makes. I guess that best describes what goes down for BSC and SFBT.

Doesn't work best when there are solo developers that like to use "deps" made by two different groups, unless you already have the dep packages made by the two groups. I see what you mean there.

There's the CAM for example: It adds more growth stages to the game, but has no "CAMeLOTs" except for the ones derived from the game's existing buildings; You'll have to go fish for them yourself. There's a lot of them anyway, but there exists several "starter packs" and yes, they have their own dependencies. Chances are, you will have already downloaded many or all of the prop and texture dependencies, leaving only the model (The actual building) to be downloaded. It's just as intense as downloading individual CAMeLOTs, only there are readmes you could use as checklists. (That was an absolute nightmare for me to do... It's just that I have incredible determination.)

There's one more thing:

There's a SC4 custom-created tool, Cleanitol, that can identify what files you have and don't have, and can also delete them if you're replacing them with an updated one. (I never tried that tool before; I just run on full faith that I have all the plugins I need, until I get brown boxes in the game, then I just read the readme to determine what's missing. Plus, I just use the Search feature in Windows Explorer as a close-enough equivalent.)

Also, I've never seen a tool that required a dependency, except for the BAT (It requires GMAX) and the Lot Editor (I think it's more of a patch).

Actually, I don't see anything wrong with the current system; You just gotta read the readmes and use some special tools. And a REALLY sharp eye.

-----
I actually want to compare this to the Network Addon Mod, just because I'm more familiar with it.

Does the NAM require a dependency? No, unless you don't have Simcity 4 Deluxe or Rush Hour. Hole-digging lots and slope mods are more like accessory mods; Technically not required, but highly recommended.

Is the NAM a dependency? Yes; Many NAM plugins (RHW, NWM, HSR, SAM, Rural Roads) require it to work properly, and special TE lots (GLR stations, for example) also require it. (The NAM Retexture and Cosmetic Mod only modifies some of the game's original textures; It can technically work without the NAM.)

Who makes the NAM? A large group of people, and a number of others who test it. (What else would all those names in the NAM readmes be for?) Many people make special and different features for the NAM (Superhand's NAM RCM for example; It's pretty much him who made the entire RCM); Think of it as the individual prop/model/texture makers in my aforementioned coalition example. It's essentially specialisation and division of labour that occurs here, and I would bet it also happens in all the other SC4 groups here.

Actually, only the NAM has ever had such a distributor made for it: You tell it what plugins you want, and it creates the entire NAM folder containing what you requested. But what happens if the NAM creates new content, like with the currently in-development (and currently stalled) RHW v4.2? Yes, the distributor will then be out of date, until it obtains the updated files.

Now that I bring it up, a sort of "Omni-Distributor" would have to keep up with hundreds of files being updated, and who would ever maintain such a distributor anyway? At least, in theory, it could be applied in small-scale. Proof of concept: The NAM distributor.

Another idea that just struck me. Have you ever been on an online shopping site and the things you ordered are in a "cart"? Perhaps if the STEX had such a feature, it would make downloading easier. The downside is when your "cart" gets "full". But even with the "cart" size limit being X amount of megabytes, what's three large downloads compared to 30 smaller ones? At least then, the ability to download individual plugins would be an option, as is the cart option.

[Twyla]

To hit some of the highlights in response:

You specifically listed DatPacker and iLive's Reader, which are indeed wonderful tools - if you have the technical background to understand their functioning.  Not everyone does.  And let's face it... Reader is an incredibly powerful tool - and one innocent change could inadvertently trickle down like Apollo 13.  v1.4 is an immense improvement over its predecessor, but is less-readily obtainable - I forget the reasoning (I think it has to do with the filesize) as to why it's not available through the usual channels - and it's far too easy to mess something up in v0.9.

As for the "Read Me" files...   I consider myself fairly tech-savvy and some of them have me wondering if they were written by Machiavelli.  That's more on the individual creators than 'the system', as that degree of technicality is more representative of bad technique.  And, as you said yourself, you have to keep a sharp eye out - and you're far more tech-savvy than the average player.

As for Cleanitol, I can't comment one way or the other - most every version I've found in the repositories has been locked for one reason or another.  As with the numerous dependencies, some of the comments I've seen in the locked versions have dissuaded me from investigating this program further.


But the source availability of these and other programs is why I feel a unified distribution method is a reasonable 'next step'.  In some form or another, most all the work has already been done - it just needs someone properly knowledgeable to put it together.

Many people would balk at the sheer number of dependencies needed for a specific download. Problem is that in some cases, a certain plugin (Let's say it's a landmark) made by, for example purposes, myself, may use props made by one person, and textures made by another, and models made by yet another person. Two ways to simplify the download process:

- The unrecommended way: Copy all three things needed, pack it, and submit it on the file exchange in question, and face potential termination.

- The "Coalitionist" way: The three people and myself may unite into a group, unify their respective models, props, textures, et cetera, into a combined DAT and submit the Dependency packages. That way, if I (or another in that group) make more lots that require their props, people won't be burdened by downloading things they already have or have to download again. Dependency packages are one-time downloads, unless it needs to be updated; Everything you need is already available for everything that group makes. I guess that best describes what goes down for BSC and SFBT.

Doesn't work best when there are solo developers that like to use "deps" made by two different groups, unless you already have the dep packages made by the two groups. I see what you mean there.

Which is but one facet to the problem.

I may be misunderstanding things slightly but, to me, the numerous assets available are being offered under a GPL-like license - the example you offered being akin to a 'derivative work'. 

So long as you make it clear that a given expansion includes external components - as a common example, textures made by BSC - and convey proper credit to their sources, there shouldn't be any problem with including them in the distribution itself.  With the utilities I'm proposing, such recognition would inherently be encoded into the distribution (similar to the common .CAB file) and displayed to the user when installed - they cover these bases themselves.

Also, I've never seen a tool that required a dependency, except for the BAT (It requires GMAX) and the Lot Editor (I think it's more of a patch).

I've encountered quite a few which require various MS Runtimes (VB & VC being the most common), along with ones demanding specific versions of .NET.

I actually want to compare this to the Network Addon Mod, just because I'm more familiar with it.

Personally, I categorize NAM itself more akin to Rush Hour - it's a fundamental improvement/expansion to the game itself (as are most traffic/pathfinding improvements).  Most of the NAM's core is stuff that should have been in SC4 to begin with.

Yes, additional NAM components (RHW, NWM, SAM, etc) require the NAM Core but, again, more comparable to Rush Hour requiring SC4.  Not to mention that they're specifically 'marketed' as add-ons to NAM.

Actually, only the NAM has ever had such a distributor made for it: You tell it what plugins you want, and it creates the entire NAM folder containing what you requested. But what happens if the NAM creates new content, like with the currently in-development (and currently stalled) RHW v4.2? Yes, the distributor will then be out of date, until it obtains the updated files.

Now that I bring it up, a sort of "Omni-Distributor" would have to keep up with hundreds of files being updated, and who would ever maintain such a distributor anyway? At least, in theory, it could be applied in small-scale. Proof of concept: The NAM distributor.

What I'm proposing is similar, yes, but differs in several significant ways:

• It would be offered as a freely-distributed tool (vs 'in-house'), with both developers and users having full access to the utilities themselves (even if the sources are kept proprietary)
• It wouldn't be as 'compartmentalized' as NAM's installer - checkboxes would only be offered on a distribution-basis.
• Rather than being an end-all 'catalog', it installs whatever distributions (created by the Developer Utility) it finds.  ie - Each .DST (or whatever extension used) found has a single checkmark (see below)

Simply as an example (using random offerings from the LEX):


As you can see, one could download whatever LOTs/MODs/BATs they desired and install them all at once without having to worry about dependencies.

And since the User Utility can throw a pop-up or other notice when a dependency is encountered, the creator(s) of that dependency still receives full and proper credit for their work.

Seems a very equitable solution to me.

[riiga]

Making it like the package manager in many Linux distros would simplify things. You can for starters create a command-line tool and then allow people to build GUIs on top of it, all open-source of course.

[Twyla]

Making it like the package manager in many Linux distros would simplify things. You can for starters create a command-line tool and then allow people to build GUIs on top of it, all open-source of course.

A true package manager would be nice, but that would also involve creating a catalog/index of every add-on and plugin available (and maintaining it indefinitely).  While, yes, this would be the ideal, this would have an inherent tendency to 'squeeze out the little guys' and lend itself to favoritism.  Not implying the bias would be intentional (though it possibly could be), but it's simply the nature of the beast.



Does an actual licensing arrangement for custom content even exist?  So far as I know, it's mainly an issue of common courtesy (along with site-specific ToUs regarding uploads).

[superchad]

i came up with a somewhat similar idea here, http://sc4devotion.com/forums/index.php?topic=13126.0

i dont see how it could squeeze the little guys out, make it so they system would be a snap to register, were if your on the LEX you can register easily and get a unique ID for your dependency.

if an auto system could be developed that would be great, auto install, auto update, and easy registration for new content.

[JoeST]

Heh, isn't it great to have such optimism

I'm currently looking at doing something similar, but rather than changing the current system, I'm looking into ways of wrapping the current exchanges with api's so that developers can query them easily, and then building simple, UNIXy tools that can be combined easily to do stuff like automatic dependency gathering, reporting duplicate TGI's (similar to what DATpacker already does, but without the packing, so it can be used with other tools)

RE a package manager, my ideas are what you'd initially need to do, since the Exchange databases already contain much data, they dont contain dependencies though, so that would have to be user stipulated/contributed#

The 'include dependencies in everything' approach is non-viable, the creators stipulate non-GPL licenses usually, the click-through license on BSC (and other) installers isnt GPL, and PEG has a weird, very restrictive EULA/license. I think it's also true that SimCity/Maxis's EULA also stipulates some conditions on custom content, though I dont know.  I think the particulars of the most common license is 'personal modification is allowed, but you cannot distribute without explicit permission from the creator, in any form'/

The Tool dependencies are things that cannot be helped by these kinds of systems because the libraries they use are usually proprietary/closed source/licensed in such a way as to disallow distribution though inclusion.

The current system is used and understood by many, most of which are set in their ways (somewhat unfortunately).

Good ideas, if a little unwieldy. I will be attempting to attempt my ideas when I'm free from exams, if I remember by then. I'm willing to hear and to try other's ideas, if any of mine get off the ground...

Joe

[Twyla]

i dont see how it could squeeze the little guys out, make it so they system would be a snap to register, were if your on the LEX you can register easily and get a unique ID for your dependency.

See, here's the problem...

Trying to maintain a complete database cross-linking dependencies is an onerous task - even for a single group.  Trying to do so for multiple groups, plus all the 'independents', gets deep into the realm of diminishing returns - exponentially increasing work for exponentially decreasing returns.  Not to mention the fact that the hoops a small-time contributor would need to leap through in order to submit would be discouraging.  In short - you're creating a Union.

I hate Unions.

The current system is used and understood by many, most of which are set in their ways (somewhat unfortunately).

And that's hitting the nail between the eyes.

The .DST approach works fully within the existing system - it's an additional tool vs the replacement a package manager would be.

Maybe I'm just too old-school, being from the days of ye olde 640K barrier - long before bloatware.  I simply do not see the return of installing thousands of additional resources (every one of which puts a strain on SC4's old-school coding) when an offering utilizes less than a score of them.  I've seen offerings (here and elsewhere) with a dozen (or more) resource dependencies - that's a LOT of dead weight.  I took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.

Admittedly, that's more on the LOT creators than those who provide Mega Packs - but the point remains.

As to some of the 'restrictive EULAs' you mentioned...  Rather than start a war, lemme use the analogy of why I hate Pepsi.

It's not because of the product (I actually like a good ol' Pepsi Throwback once in a while), but because of their 'marketing strategy':
1 ) They buy up popular fast-food chains and force them to serve/sell their products exclusively vs allowing them to follow customer preference
2 ) They offer at-cost distributing to retailers willing to not carry any competing products (yes, this is FACT)
3 ) They use these 'forced purchases' to artificially inflate their sales statistics - using half-truths vs actual consumer preferences

The whole reason the SC4 community has grown as large as it has is because of the openness of it.  A lot of the offerings of additional and improved content exist because of the availability of Mega Packs.

But, again, there's only so much baggage SC4 can handle.  And only so much benefit utilities like DAT packers can offer.


There is also one additional aspect which needs to be taken into consideration:

Let's say Person-X (who could as easily be a small group as an individual) has a fair amount of resources available - props, textures, models, etc.  Good stuff.  Stuff that a lot of people make use of.

And, for some reason, they decide to pull their content...  Maybe they got in an argument with someone here, or found a different SC4 site, or just got their panties in a wad for no particular reason, etc.

Now what?

If you don't already have these resources, where do you get them?

Even without this, some dependencies are pretty hard to track down - some have been around a long while, and the sheer volume of offerings frequently makes any given file a needle in a haystack.  Some creators automatically assume that everyone already has them (anyone worth speaking of), etc.

Not to mention a number of 'legacy' dependencies...  Their creators have moved on - making their inclusion in a Linux-type distribution manager a tentative affair, at best.


Believe me...  I've studied just about every angle out there - SNAFUs, FUBARs, and other red tape included.  As much as I would truly LOVE a full-bore package manager, it simply isn't a feasible solution - partially owing to the huge investment required to establish/maintain an exhaustive database (as well as certain offerings being locked/removed for whatever reasons), but mainly owing to the limitations of SC4's internal coding.

I don't profess to be a person of many talents, though one of my modest 'claims to fame' is a bent for combining seemingly contradictory ideas into a cohesive whole in such a manner as it seems inevitable.

Not saying that there aren't other solutions, or better ones, but none that I've seen or heard that I believe to be more compatible with the existing situation.

[Diggis]

I've ummed and ahhed about responding to this topic.  It comes up every few months, people have great ideas, but no one is willing or able to actually do anything about it.

As I understand it, you want someone to develop a program that everyone will upload their dependencies into. This program will break each part of the mega pack into individual models and textures and ID them. 
Then creators will add a dependency file to their upload which will link to this program/depository and download only the necessary ones, and only if you don't have it?

I can foresee a large number of issues....
- There are already somewhere in the vicinity of 860 dependency packs out there.  I know cos I prepared the list of them.  This was up to December 09, so there are more now obviously. Many of these are already compiled.  I don't know if you have ever tried to pull one model out of a compiled pack, but it's not easy. Textures are easier, but only marginally.

- Many of the creators of these packs are no longer active in the community. Who takes responsibility for their work?  Or do we ignore it and start from now?  While there are many great artists still working out there I doubt we could NOT include SG's stuff for example.  Now, BSC can manage anything by one of their creators, but solo artists or smaller groups could struggle.

- I took from your opening post that you can't write this package yourself, so would be requiring someone else to write this.  There aren't many people in this community I know of that could write such a programme, and non that I can think of that are active.

- There are 3 major exchanges plus a number of minor ones in the community.  Relations between members of all these sites are always on great terms.  Who hosts this?  What if someone is banned from one of the other sites?

While I appreciate that the current system isn't great I don't believe it's as bad as you make it out.  Most responsible lotters use common packs, such as the BSC Mega Packs, and they try to minimise the number they reference by looking for alternatives.  Not only that, they only time you'll get a brown box is if you don't have the building model. Missing props just don't show up so the lot is a little bare.  If it's too bare, you need to find the dependencies.

I created the dependency list because I was sick of not knowing what I had or where it was.  Now I check off what I have downloaded and then I don't have to redownload it.  It also links to the latest version of each pack so you know you are getting the latest version.

[superchad]

in my idea the program has no database, there is a server database, and a small database of needed files that the downloaded content needs, the database on the server is a mere reference to files on the LEX, the database for the user tells the server what is needed based on ID numbers

[JoeST]

Adding a separate manifest-like file to every item on the LEX listing anything it uses is as big a job as maintaining a distinct database of links. Especially since the updating would have to be done by the owners of the items (or the LEX staff), rather than just anyone, which stops them from having time to do other stuff, and all for a system that is new/untested.

A hybrid approach could be a third party make these manifest files for each file on the LEX/STEX/etc... but that's just the same as a database in a way....


Actually, the Hybrid might work: given an id of a lot (such as STEX-2780 or LEX-443) we can make a file:

Code (STEX-2780.dst) Select Expand


{
"name":"Keaton Plaza 1",
"author":"DuskTrooper",
"description":"a skyscraper blahhhh<h1>OMG THIS IS AWESOME</h1>",
"dependencies":[
"LEX-443",
"http://sc4devotion.com/csxlex/lex_filedesc.php?lotGET=583"
]
}

which is actually kind of what I would have had the API display for each item, if I could work the dependencies database in somewhere behind the API. Maybe having a site with a collection of these in an editable form, so that anyone can come and edit the dependencies at least, maybe with a thing that makes sure someone 'in charge' can check over edited ones, or something.

Maybe I'm just too old-school, being from the days of ye olde 640K barrier - long before bloatware.  I simply do not see the return of installing thousands of additional resources (every one of which puts a strain on SC4's old-school coding) when an offering utilizes less than a score of them.  I've seen offerings (here and elsewhere) with a dozen (or more) resource dependencies - that's a LOT of dead weight.  I took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.

This might be solved by having some smart software similar to datpacker, that analyses LOT's and things, and strips out anything it finds that isn't used by anything else.

The people who stop us indexing their contributions, or those who have left (unless they've said it is ok to do so) should not be included. That is their choice,

Sorry I forgot to make this clear, I wasn't intending to build software that downloaded items from exchanges, merely create tools to make dependency tracking simpler. All my tool might do is provide a link to where to find the items it knows you need. That's the power of UNIX philosophy: simple tools, easy interfaces. It would mean that I could interface this tool easily with another one that could download things from the LEX (similar to Wou's download tool), or check to see if you already have it installed, or some other yet to be thought of application. The same goes for a duplicate detector, or an unused detector. Simple, extensible, awesome!

Joe

EDIT: Indeed Diggis' dependency list is a fantastic resource, what I am attempting/suggesting/etc. is merely an extension to this and stuff. Thank you diggis, you're awesome

[Diggis]

EDIT: Indeed Diggis' dependency list is a fantastic resource, what I am attempting/suggesting/etc. is merely an extension to this and stuff. Thank you diggis, you're awesome

The problem with the dependency list is it relies on someone (read: Me  )to update it. I tend to get a bug every maybe year to do that.  I can't see an automated way to do it without having it linked to the Exchanges.

[Twyla]

in my idea the program has no database, there is a server database, and a small database of needed files that the downloaded content needs, the database on the server is a mere reference to files on the LEX, the database for the user tells the server what is needed based on ID numbers

Umm...  that's a lot of databases for a program that doesn't need a database...

As I understand it, you want someone to develop a program that everyone will upload their dependencies into. This program will break each part of the mega pack into individual models and textures and ID them.  Then creators will add a dependency file to their upload which will link to this program/depository and download only the necessary ones, and only if you don't have it?

Nope.

The utilities I'm proposing are more akin to the way a typical archive manager works, treating the dependency files (by name) more-or-less as folders:
~ The Developer Utility would build a compressed 'archive' which includes all assets utilized by the offering - pretty much the same as it is on their system
~ The User Utility would rebuild that structure of the user's system - adding if it doesn't exist, with the option to over-write (for updates, etc)

I can foresee a large number of issues....
- There are already somewhere in the vicinity of 860 dependency packs out there.  I know cos I prepared the list of them.  This was up to December 09, so there are more now obviously. Many of these are already compiled.  I don't know if you have ever tried to pull one model out of a compiled pack, but it's not easy. Textures are easier, but only marginally.

- Many of the creators of these packs are no longer active in the community. Who takes responsibility for their work?  Or do we ignore it and start from now?  While there are many great artists still working out there I doubt we could NOT include SG's stuff for example.  Now, BSC can manage anything by one of their creators, but solo artists or smaller groups could struggle.

- I took from your opening post that you can't write this package yourself, so would be requiring someone else to write this.  There aren't many people in this community I know of that could write such a programme, and non that I can think of that are active.

- There are 3 major exchanges plus a number of minor ones in the community.  Relations between members of all these sites are always on great terms.  Who hosts this?  What if someone is banned from one of the other sites?

As you've pointed out - the voice of experience - even just trying to maintain a simple listing of dependencies is a monumental task; comprehensively cross-referencing them (as a full-blown distribution manager would need to), not to mention keeping them up-to-date, would be a full-time task for multiple people.

What I'm proposing bypasses that mess entirely - by simply packaging the individual dependencies with the offering.

And it isn't that I CAN'T write it myself...  I just don't have a great deal of experience tinkering with SC4's inner workings - and experience counts!  Someone already knowledgeable could likely churn out the entire program in less time than it would take me to track down and establish the necessary data fields.

[Diggis]

The utilities I'm proposing are more akin to the way a typical archive manager works, treating the dependency files (by name) more-or-less as folders:
~ The Developer Utility would build a compressed 'archive' which includes all assets utilized by the offering - pretty much the same as it is on their system
~ The User Utility would rebuild that structure of the user's system - adding if it doesn't exist, with the option to over-write (for updates, etc)
As you've pointed out - the voice of experience - even just trying to maintain a simple listing of dependencies is a monumental task; comprehensively cross-referencing them (as a full-blown distribution manager would need to), not to mention keeping them up-to-date, would be a full-time task for multiple people.

What I'm proposing bypasses that mess entirely - by simply packaging the individual dependencies with the offering.

OK, sorry you are using terms that have lost me.  The developers utility builds an archive of all the dependences, yes? How do they get there? How do new ones get there as we go on?
And the User Utility then interfaces with this and downloads the necessary elements of the archive to the users computer, yes? How does it know what they are?  Does a lotter have to list the ID's of the elements? 

Bear in mind we already have people who don't list dependencies with links.

Also, who would host the archive?

[Lowkee33]

I like the ideas coming out here, so I don't want to halt the conversation, but.

I'll throw my "stuck in the old ways" idea out there.

I would say that all of this can be done without changing the current format.  The work is re-doing every exemplar so that models across every pack share the same Building/Prop Families.  Each MEGA pack has "exemplar packs" associated with it (one for buildings, another for props...).  A set of core lots are also made.

The player would download the Lot pack.  Say this person wants to use CP vol 01, so this person downloads that MEGA pack and the exemplar packs.  Since all of the exemplars are separated, no lot is "dependent" on the player having a specific MEGA pack, only the more packs the more models (the same amount of families), and the less repetition.

As far as programmed lists of things:  I would like to see some database that works the other way around.  When a dependency is downloaded, so too is every exemplar associated with it.  The models take up the most file size, so getting the most of them is more important to me.

[superchad]

Twyla i mean by that the program doesnt have its own database, the database is provided with custom content, within a file included in the lot, the file just has the ID numbers that reference in the server database, the server database then link to the correct files on the exchange, it would have a system so content that has been merged links to the merged packs, and a system for determining if existing content is up to date, maybe the plugin folder has a database of installed dependencies, and updates ones needed updating. the program wouldnt have free access to the exchange, you would have to login to the exchange on the program.

regardless if you take my approach or another approach i belive we need to come up with simpler ways of tracking down dependencies, installing them, and updating them.

[Zaphod]

Its 2011 so I think most people have something better than dial-up for downloading, and have a better computer than the 256 mb RAM Windows ME dinosaur they started playing SC4 with in 2003.  It's my opinion that if ain't broke, don't fix it, or things get even more complicated. I have virtually all the major packs on this site, I keep some in an inactive backup plugins folder, etc. The real solution is for creators to think about what they are doing first, and avoid using one-off dependencies if it means their creation could otherwise be dependency light or free.

I took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.

To be fair, some lots are really good. Some dependency packs with far more bits and pieces than any one thing could use. They seemingly exist for this purpose. Jestarr's industrial props and their use towards building railyard lots and  industrial complexes are a great example.

[jmyers2043]

@ superchad

regardless if you take my approach or another approach i believe we need to come up with simpler ways of tracking down dependencies, installing them, and updating them.

Interesting.    I don't find them hard to track down at all. I can see a scenario when someone goes on a massive download spree that they may have some dependency searching and extra work afterwards.     

@ Twyla. I'd say go for it if you want to write a program. If not? Then the discussion should properly fizzle out and will have served no purpose. 


My rules of thumb are to use the lot as is. Delete. Fix it in the LE. Download the dependency when critical to the function or look of the lot. Four choices and none require programming knowledge, time searching the past, or future upkeep. 



- Jim

[jacksunny]

In my honest opinion, I would have to say that this entire converstion is useless. 

I mean sure this will benefit new players to the game who have to download all these dependencies but for those of us who already have them it won't matter. Besides once you download them, you won't have to download them all again and with all these dependency superpacks (like BSC Essentials). Before you know it you won't have to download anything ever again. I have to agree with Zaphod :

It's my opinion that if ain't broke, don't fix it, or things get even more complicated.

I believe that doing this will just cause major problems, but that's just my thought. Which is better, spending a huge amount of time to create a program and database, never mind updating it, or spending a few minutes just to go to a dependency and download it?

If you decide to go ahead with this then I wish you all the best of luck. 

Regards,
Jack

[superchad]

i dont see how a system could hurt

jmyers2043

that is the major scenerio that happens alot, people dont just download one or two things, i always downloads large amounts of content, and extract it into a single folder, label the folder say May162011 run the install files, and hunt for dependencies if needed

it would make doing this much simpler, and updating them simpler too

also sometimes people have to rebuild there plugins, say from a hard drive crash/corruption