Terraformer and Save Game Bug

[Lowkee33]

I'd like to report a general problem with Terraformer and the way it makes new cities  (the same is probably true for SC4Mapper).  When you create a new region in Sim City, there are no city files created in the region folder.  This is different that Terraformer.  Terraformer has template cities that it uses to create new cities, found in the Terraformer\Datas folder.  When you're done making a region in Terraformer, the region folder is full of new cities.

There appears to be a certain amount of data that is saved in these cities, the one I noticed is Initial Demand (an RCI exemplar property).  I've been playing around with it, and was confused when I started my city that should only demand for farms, had the default maxis demands.

I was able to fix it though.  I made a "New Region" while playing the game, entered a city of each size and saved them before entering Mayor Mode.  I then copied over the templates in Terraformer\Data.  I wonder what other data was in there?

[whatevermind]

Interesting, I had never looked into just where exactly SC4TF got its cities from, but seeing as every region I play has passed through this program at some point, this caught my attention.

I wonder what other data was in there?

So, armed with Reader and a free afternoon, I dug into this one.  It gets interesting.  I did some basic checks on the SC4TF Large and Medium templates, but focused on the Small template.  So I then compared the Small template with a few small cities created directly in the game.

First, a few big differences that jumped out at me while I tried to create a file that most closely matched the SC4TF template.  Some of these might matter, most probably don't so much.

• Small, Medium, and Large cities don't all have their data in the same place.  For example, what might be at Number 109 in a Small city, might be Number 125 in a different size city's file.  TGIs, however, are consistent.
• The SC4TF template TGIs are generally better organized (ie easier to read, line breaks, etc.).  I don't know if this is just how Reader parses the files or any actual difference - at the end of the day it's all just a stream of bytes anyway.  Doesn't look like the TGIs actually contain extra line breaks coded into them.
• The SC4TF template files are in French (merci, Wou).  While this probably only impacts those parts of the savegame file that contain text strings, it does account for roughly half the differences between my test file and the SC4TF file.  It also means that the .dat Compare Tool flags a whole bunch of TGIs for having different sizes, because something "written in English" and "écrit en français" end up having different text string lengths.  I think the only way to control for this would be to have someone check a SC4TF file against a control file from a French version of the game.
• SC4TF template files are at a constant elevation of 252m, while the water based region I created was a constant elevation of 192m.  IIRC this can be controlled for, but that's what came up when I created the region.  This also accounts for some of the differences between the two files, however.  For example the terrain map (a9dd6ff4 e98f9525 00000001), and reconcile edges data (the other four Type a9dd6ff4 files).
• The savegame file changes between God Mode and Mayor Mode, as you might expect, adds some TGIs, changes others.  Going forward though, I used only files that were last saved in God Mode and hadn't been established yet - because the SC4TF created files always load in God Mode.
• Plugins matter.  See below.

So first, I created a new region, all land, and ran that file against the SC4TF template file, which is how I figured out that the SC4TF files are water based.  Although, the elevation in them is 252m, so they're just barely water.  I controlled for that by creating a new region, all water, to eliminate any differences that might pop up with one being land and the other being water.

This left four TGIs that still had a large difference - and by large I mean my test city had them at over 12kb and the SC4TF template had them at ~100 bytes.  All four of these were interestingly right around 12k in size and had similar Type IDs: A98F9D3B, A98F9D28, C98F9D4A, and C98F9D17, note that only the 1st, 7th, and 8th digit change.  Don't know if this is significant.  All four of these are considered unknown properties as far as I know.  On a whim, I figured these might be plugin related, so I removed my plugins, dropped a blank plugins folder in my game, and started a new, water based region, picked a small city, saved it in God Mode, and fired up Reader again.

This turned out to be the magic key - all four of those properties now matched the SC4TF template file in size and only differed in text.  Sizes were down around the SC4TF template file sizes, roughly 70-120 bytes.  Likewise, the .sc4 file sizes were much closer, mine at 115kb, the SC4TF template at 125kb (previously mine had been 188kb).  With this, I ran a difference check, and followed up on that by checking the two files to see what those differences were.

Roughly half the TGIs reported different sizes, however nearly all of these were files that contained text strings - Ordinances, Advisors, Taxes, Rewards (0a413481), City Data (8990c372 and ca027edb), etc. and I mark this up to the language difference.  Generally, size differences were small, within 50-100 bytes.  The .png files also differed in size, as did the directory file (same number of entries though, probably due to recording different file sizes).  The Terrain Array (2990c1bc) also differed in size, but not by much.  The contents of this also looked pretty similar, I figure some of the difference is due to the cities being at different elevations, but I didn't dig deeper into this one.

Then there were roughly a dozen TGIs that were identical.  Mostly these dealt with the png files or appeared to be blank filler, some are unknown properties.  The most interesting one was at Number 0 (499b23fe).  Which is not however at the beginning of the file, but somewhere 50kb deep.  Don't know what this TGI does, but it's about 600 bytes of actual data.

Now, the ones that differed only in the contents of the TGI, which is most likely where the differences you spotted are occurring.  Some of these were Advice (6a3848db, 6a9335de, 8a09f5f4 and aa371c32) files, others were My Sim (0bf606a5 and 4a1dbbbf) files. A huge chunk of these files fell into the category of TGIs that were never decoded as far as I know, and most of these are relatively tiny, often under 100 bytes, which tells me they are probably either flags of some sort, single value settings, etc. - what exactly though, I have no idea.  The terrain map and reconcile edges data also fell into this category, as expected.

The more interesting TGIs (with names at least) that differed in text only are:

• Number 51 - Historical Data/Graphs (89efa536)
• Number 57 - Water Pollution (8990c065)
• Number 59 - Pathfinding (6a0f82b2)
• Number 60 - Sims to Jobs Match (6990c1aa)
• Number 63 - Situation Manager (4be372cd)
• Number 82 - Power Simulator (0990c005)

This same chunk of locations also contained several TGIs that are undefined, pretty much Numbers 27 through 107 are more or less unknown TGIs that differed in text only.  Interesting notes I picked up along the way are that the Historical Data appears to only be a template, in cities that have been played that file gets much larger.  Also, in a small city (with 4096 squares) the water pollution file only appears to have 2048 entries.

That's all I have for now.  If anyone has decoded more of the TGIs than what were posted way back when, I'd love to know.  Especially since I'm working to get all that old technical info onto the Wiki.  It would of course also be immensely helpful here for figuring out what all those unknown TGIs do.


Edit:  Changed "Location" to "Number" and added Type IDs (full TGIs where necessary) to the names above.

[Lowkee33]

A quick response before bed.  You took more from today than I did.  I mostly tried to save the game as fast as I could to get multiple copies of a relatively similar condition.  Nothing I was looking for changed in any meaningful way, so I scratched my head a lot.

The save game files are largely unknown.  You will want to check out Wouanagaine's Save Game Explorer to get deeper into it.  Reader is a good tool also.

Most research that goes into the files is about prop pox.  All research sort of ends with the fact that we do not know the CRC Maxis used, and thus we can't effectively change anything in the save game.

You're calling a lot of TGIs by name (Terrain Map sticks out in my head).  As you said, the position/order of these files seems variable.  Could you perhaps list the actual TGIs instead?

Be careful about line breaks, Reader is displaying the hex code as ASCII characters.  A line break is probably/really just 0x0A or 0x0D.  No one hit "enter" or anything.  It is more interesting that there are places where there are "line breaks" at common intervals.  We can see that there is at least a certain pattern of data in those places.

[whatevermind]

Hehe, I was bored and this was a good excuse to poke around the savegame files a bit. Per request I added Type IDs for anything I mentioned above that hadn't already been included in Reader.  Some things, such as ordinances and advisors have names in Reader, so I didn't include the IDs for those since I wasn't talking about any specific ones anyway.  Also, changed Location to Number wherever I had used that incorrectly. If you're looking in Reader at a small city saved in God Mode, you can use the Number to find the entries.  Different size cities will not work, you'll need to use the Type ID.  As an aside, for savegames, all you generally need is the Type ID for locating something, very few files have the same Type.

I did look into the Savegame Explorer, it's not something I had used before.  Neat program, but you'd still need Reader to poke around the savegames.  Interesting to see that wouanagaine does appear to have advanced the decoding of at least the building, prop, and lot files.  I haven't yet looked at his flora plugin, and it looks like he never released the commute path data, but it looks like he decoded those as well.

I did some further research today, and it appears that a SC4TF generated file is largely rewritten to be similar to a game-generated file after the first save of the game in SC4.  A SC4TF generated file matched the template most closely after only being created and saved in SC4TF, but after opening that same city in game and saving it, it now much more closely matches a game-generated control file.  Still a lot of text differences, but nearly all the size differences disappear at this point.  Interestingly, the remaining size differences don't seem to disappear, at least not as you proceed through establishing and starting to run the city.

One of the Taxes files (aa313c9f), which appears to control transportation, reward, and utility funding, was among those that didn't match itself to the game-generated control file.  Because this file contains text and is more or less legible, I tracked this one through time, and even opened some cities I had sitting around that I knew to be old and fully developed, and also to have been created in SC4TF.  The result is that even in old and developed cities, some of the text was still in French.  However, the text in game has never displayed French.  So there's more going on here, but it was interesting to confirm that at least some fragments of the SC4TF template files do live on in the game files.

I agree that by far the easiest fix to this is exactly what you did.  I would love to see the savegame files fully decoded, and I'm sure if the data was available, someone would find a use for it, even if it was just as an analytical tool.  But, reality of course is that that knowledge would just be a novelty - we already have a powerful tool that can directly edit most of the data in these files, SC4 itself, and the rest of the data we can't change until someone figures out how to write to these files.