Thoughts on Content Delivery in SC4 - please discuss!

[7up]

Why hello there.  I realize none of you will recognize me or my username on these forums. I am classic lurker, gleaning information and content but not contributing much discussion or feedback.  I intend to change this; I think the size and quality of the SC4 community is incredible for a 8 year old game! If you are a member of ST and SC4D you will find this post at both sites not only for the sake of exposure but also because of the cross dependency these sites have and thus the need for cooperation between them. My apologies for the redundancy.

This post (novel?) has been brewing in my brain for a while but my recent frustrations with site upgrades and such have prompted me to type. Please don't take any of this as derogatory towards any of the SC4 exchanges.  I have nothing but gratitude and respect for the sites and people who run them that keep this game and its community alive!
Let me start with some software experiences in the Linux world, my main operating environment. If you are already familiar with these systems, please bear with me.  I have to assume that most SC4 players are exclusive Windows users, naturally.

When I started using Linux about 10 years ago, I ran into a situation that many users were battling at that time - dependency hell. In Unix and Unix-like systems, software tends to be smaller and more purposeful (do one thing, do it well) and much of its functionality depends on linking to other programs and libraries that can do their particular mission well, like connecting to a network or writing to a jpeg, etc. The same thing of course occurs in Windows software, but distributors tend to package everything the programs needs within the program itself.  This results in much redundant code and the inevitable 'software bloat' that comes with it.  But, I digress.
By now, this all should seem quite familiar as it is the exact situation with delivering and coordinating custom content for, say, Simcity 4.  Groups of files that need to downloaded and installed in specific locations that may (and often do) depend on other files to be installed. It may not be a surprise to you that the solution to this problem is already developed and mature.

To solve this problem of software dependency and delivery, Linux systems use what called a package manager. The system I have the most familiarity with is the APT system developed by Debian and used in all Debian variant OSs including the most popular distribution, Ubuntu.  APT and similar systems have the ability to install software 'packages' on to users' machines by downloading them from a online repository, checking for needed dependency packages, checking for conflicts, etc.  The system keeps track of installed software, upgradable software, dependencies, conflicts - the whole nine yards. 

I should be noted that these packages are nothing more than files and folders on a machine.  The APT system uses simple archives (think zip files) and plain text files to organize and identify these packages. 
Let me give an example of how a package manager is used.  I open the program (typically GUI nowadays) and I can browse through categories or search through package names or descriptions to find what I want to install.  Installing the program (package) is as easy as checking a box next to it in the manager.  Perhaps after checking the box, I get a screen that says this program requires additional packages in order to operate - do I want to selected these automatically to install as well?  Sure.  I also see that a package I previously have installed has an icon next it to indicating that a new version is available. With a click I can mark this package for upgrade.  When I have selected all the software I want, I hit 'apply' and all of the needed files are downloaded and automatically installed. Cool.
Compare this scenario with your own experiences with the current state of content installation for Simcity 4.  It is not uncommon for me to download 20 or more BATs in a sitting and then location and download any dependencies that they have.  The files come in the form of a zip archive that may contain the needed files plainly, or may contain an installer. 

With all due respect to all the amazing content creators in the Simcity community, the installer is the worst possible way to install a few files in the plugin directories!  I find myself clicking through hundreds of "Next" buttons and empty eula screens to install the couple dozen buildings I desire. Almost always, a readme pops up and redirects the computer focus to the browser requiring me to reactivate the installer screen just so I can hit "Exit". Then do that 19 more times.  Sound familiar?

In short, I realize that this sort of proselytizing is late in coming to the Simcity 4 community, but the software and solutions freely exist  to enable this sort of content delivery for our own use and our own community. Who's ready for a change? Lets discuss!

[Tarkus]

First off, welcome to SC4D, 7up! 

There have been quite a few discussions on this front and related topics, including one relatively recently, and it's been an ongoing dialogue for years.  Are there drawbacks to the current "status quo"?  Yes.  But are there drawbacks to change in this instance?  The answer there is also yes. 

I certainly can see the advantages of some sort of a "package manager" type of setup, but there are some considerable obstacles to consider as well, namely the interface of the exchange, and the status of installers as part of the SC4 delivery system.  In order to effectively accomplish a package manager, if I am understanding everything correctly, it would require a complete rewrite of the exchange software of the major sites in order to properly interface with the package manager.  It is my understanding that the developer who wrote the LEX software does not really have time to work on it further, and Dirk over at ST is just finishing up a major revamp of that entire site.  The infrastructure isn't in place where all the files are, and it'd take substantial effort to put it in place.  One could, theoretically, start anew, but it would defeat the purpose if that "new start" didn't have a substantial amount of quality content that would give people a reason to use it.

With regards to the installers, yes, it can be a lot of clicking "Next" and "Exit", but as simple as many of the installers are, that doesn't take up that much time in the grand scheme of things.  Should people be competent with file folder architecture and knowing their way around Plugins folder, such that they could properly install something without an installer?  Ideally, yes.  Unfortunately, that's not the always the case, especially with less experienced users, and I've seen this first hand.  Additionally, load order is quite sensitive and one can screw things up pretty badly if things aren't installed right, beyond just the infamous "brown boxes"--we're talking wide-scale conflicts potentially ending with corrupted cities, CTDs, etc. 

There's also the matter of established content as well . . . if you were to go through the just under 2500 files on the LEX, you'd find the vast majority of them do, in fact, use installers.  A substantial number on the STEX do as well.  Switching them over to non-installer forms would mean re-uploading all those files (a very lengthy and tedious process), and as the installers also have fairly optimized compression on them (the NSIS system we use on the NAM is actually slightly more efficient than even a .7z file, and the Clickteam system is also quite efficient), you'd likely end up substantially increasing bandwidth usage as well--by at least double. That last point is really not going to sit well with the site owners, who are paying to host these sites out of their own pockets, generally. 

I do see the potential in what you've described.  Perhaps if this were 8 years earlier, I think it'd stand a better chance.  I know you are well-intentioned, respect you for trying, and I don't mean to completely throw cold water on your ideas, but I am also having difficulty seeing it work out the way you intend.

-Alex

[joelyboy911]

I suspect what you're proposing may have been mentioned before  . For this sort of thing to happen, I think you will have to take the initiative to start working on it, as most are content to operate within the status quo.

I agree it is a very good idea, but it has to be very (very, very) easy to understand and use for it to have a significant uptake, and as I understand your post, all content ever made would have to be loaded onto a massive online repository. Unless the package manager program can find and download files from where they're found now, I don't think anybody would be interested in transferring all files onto one server, because:

a) There are probably at least tens (if not hundreds) of gigabytes of lots, BATs and mods that would have to be put into a unified format and then transferred.
b) There are certain (erm...) political issues that come into play with unifying these sorts of things.

With all due respect to all the amazing content creators in the Simcity community, the installer is the worst possible way to install a few files in the plugin directories!  I find myself clicking through hundreds of "Next" buttons and empty eula screens to install the couple dozen buildings I desire. Almost always, a readme pops up and redirects the computer focus to the browser requiring me to reactivate the installer screen just so I can hit "Exit". Then do that 19 more times.  Sound familiar?

I'm sorry to tell you, but you won't get a lot of support from this site on doing away with the installer. The BSC uses them as a matter of policy, and many others follow that lead. I use them as well just by personal choice, take it or leave it - I'm not too concerned.

I hate to be a negative commenter, I like to think of myself as a more can-do sort of person. Unfortunately my opinion is that this isn't a viable option, the community is hardly in it's boom phase, and I think the work required is too much. The concept is a good one, and it would make downloading a lot easier, but I don't think you'll find many people willing to change their ways, nor many willing to help you implement this systemic change.

Please don't take this as derogatory to you, I just think that a step change like this in the way the system is managed is too late to get people excited about doing it. Had this idea been implented seven, five or even three years ago, it might have been doable, but I think it's simply too late, and too much work to take off.

Hat's off to you for questioning the status quo, though.  

[deadwoods]

I'm a big fan of RPM, APT and Ubuntu's Update Manager. They are great tools and something like that would be brilliant for the custom content here and at the STEX. A mechanism like this would be the Holy Grail for all of the content. 

However, as those above have mentioned, this discussion has gone 'round the block many times and often becomes quite heated. It may be worth searching the various discussions, like the one Tarkus linked to above, and see the history of this.

This type of tool would certainly solve the "I've got a brown box" comments that often arise. It may need to force a particular plugin structure, which many users wouldn't like. It would require software to be installed on personal machines, again which many wouldn't like. It would need to cater for the various flavours of Windows and Mac (the installer doesn't work for Mac, a constant sore-point for the Mac users).

The biggest challenge, as highlighted above, would be the migration-to and support-of the various repositories. As mentioned Colyn is off somewhere else and he owns the LEX code (which is separate to SC4D), Dirk has just updated ST, and Peg may or may not play the game depending on the day of the week (sorry).

I know nothing of the APT-get mechanism, but I'm assuming it's some IP-level call to a socket with a service sitting behind it responding to some standard messages. All of this would need to be coded in the repositories. Plus the actual client would need to be coded (for different platforms) and distributed. Then someone would need to create a database of the existing shared content (prop/texture packs, mods etc.) and publish it for the tool to use. Then content creators would need to update their bats/lots/mods to use the new tool, many of whom are no longer around.

It's an admirable idea, but I can't see it flying.

[Andreas]

Well, I can't add much to what has been said already; the system that we established over the last few years is basically something that tries to reduce support issues as best as possible for non-professionals like us. If Maxis/EA would have given us better tools in the first place, things would have developed differently for sure. Just take a look at the official SC4 Exchange: It's using some ActiveX script (I think) that reads the LD file of a lot (which includes the filenames for prop dependencies) and includes those in the upload.

This is basically what you wanted: An automated system that makes sure all files are gathered for distribution. But very soon, custom modding tools were developed, and we created loads of stuff that wasn't ever forseen by the creators of the game, and as usual, each custom content creator developed an individual style. With the teams that have gathered, we were able to create some kind of standard, but there are still a lot of individual creators from all over the world that wouldn't get in touch with the maintainer of such a package manager (just think of all those Japanese BATs).

I can't speak for others, but as far as the SFBT goes, creating an installer takes less than a minute, and since there's an established folder structure that is exactly the same for each SFBT creation, I can use some templates to speed up packaging as much as possible. This is minimizing errors on my side, too, and I've rarely got any reports on broken installers and such so far, after uploading more than 150 files, although I don't do a whole lot of testing in that regard. Packaging an upload still takes about an hour or two, though, since I try to present it in a nice way, with custom splash screen, proper readme file and such - this is something every download deserve, after the countless hours a BATter or lot designer spent on the actual building or lot.

Anyway, to make a long story short, I'm rather content with the system that I have developed until now, and it allows me to work effectively. While I agree that a package manager for SC4 content would be awesome, I really can't see anyone actually picking up such a project, as it would be way too much work for everyone (programmer, site maintainer, custom content creators), and that doesn't even takes "political issues" and other obstacles, like language barriers, into consideration.

[Lowkee33]

Switching them over to non-installer forms would mean re-uploading all those files (a very lengthy and tedious process).

And if you're going to do that you might as well redo all of the Instances to make sure no custom building/prop/texture overrides each other.  At that point, you might as well make everything the same Group, so that you could make common cohorts.  Of course, that means re-making all of the building exemplars (which might as well be balanced), and re-making every lot...

My point is that, although it is a good idea, going through all installers ever made will take less time than doing what you propose correctly.

[7up]

Thanks for the thought out replies guys! 

I assumed that things like this have been discussed before; I didn't know where to look or if discussions were ongoing.  Of course all of the points made are very valid. I feel its a shame I wasn't around to contribute to these things.

With regards to re-uploading, modding lots, and things of that nature, of course that would all have to be done by hand and there is no real need or motivation for that king of effort. 

My thoughts were more aimed at repacking content in an automated fashion. The fact that certain installers are mandated (or just standard?) could be a good thing for automation purposes. Here the labor involved in naming and folder heirarches and readmes has been done already.

As you might be able to tell, I'm a fixer, sometime even when things don't need it!

This forum has been welcoming and a good place for discussion.

Cheers!

[7up]

The biggest challenge, as highlighted above, would be the migration-to and support-of the various repositories. As mentioned Colyn is off somewhere else and he owns the LEX code (which is separate to SC4D), Dirk has just updated ST, and Peg may or may not play the game depending on the day of the week (sorry).

You are dead-on. The gobs of legacy content and their maintainers are the main obstacle.

I know nothing of the APT-get mechanism, but I'm assuming it's some IP-level call to a socket with a service sitting behind it responding to some standard messages. All of this would need to be coded in the repositories. Plus the actual client would need to be coded (for different platforms) and distributed. Then someone would need to create a database of the existing shared content (prop/texture packs, mods etc.) and publish it for the tool to use. Then content creators would need to update their bats/lots/mods to use the new tool, many of whom are no longer around.

This is UNIX tech, very KISS!  apt repositories are plain http .

As for the conversion stuff, see my remarks on automation.