What causes Prop Pox (and how to avoid it)

[bap]

Index



1. Introduction

2. Hunting the cause of Prop Pox

3. Face to face with a cause of Prop Pox

4. How to avoid the problem

5. Testing Prop Pox by yourself



Introduction

What is Prop Pox?

A brief description is as follows. The last time you saved your city everything was fine. The next time you open it, you notice that props are missing from several lots. It includes props both in Maxis and in custom content lots, as well as street props. Parts of your city start looking strangely ... empty. Not all the city is affected, only some parts of it. The affected lots continue to work normally – you can query them as usual. If you destroy and replop the affected lots they will usually go back to normal, with all props appearing. But don´t foul yourself. If you save the city, logout and login again, you will notice that props are missing again, perhaps on different lots then at the previous time. It seems a waste of time trying to destroy/replop the affected lots and keep going, because the problem unfortunately spreads with time: every time you save the city, more and more lots are affected by this decease. This is way it was named "The Prop POX".

Prop Pox should not be confused with other bugs of similar graphical appearance such as the growing empty lots problem (see here) or when one runs one of Maxis updates (EP1 or BAT), which resets the graphics settings to low, making most of lots props to apparently disappear.


What do we know about Prop Pox?

Prop Pox was first reported about the same epoch by BruceAtkinson , Snorrelli , and Fledder200 back in April-May 2007. It is a rare decease. It seems only a dozen or so players have been affected among the thousands of SC4 users, most of them after early 2007.

Thanks to the patient collection of information over time and to several tests ( see here ) it was learned that:

1)   It is not a graphical bug. Prop pox is not solved by making the graphics settings to high, nor by selecting hardware or software rendering.

2)   It is not a computer memory issue or a conflict with other software. Desfragmenting and cleaning disc space as well as increasing virtual memory also doesn't solve the problem.

3)   It is not a hardware or operational system problem. It occurs on both Windows XP and Vista, on all sorts of processors and video cards.

4)   It occurs mostly on large (4 x 4 Km) city tiles, when most of the city surface is occupied/developed. It is not a matter of population, but of number of lots (and, or course, props).

5)   The problem is in the city game save file (city_name.sc4 in the corresponding region sub-folder at MyDocuments/SimCity 4/Plugins/Regions). The affected city sc4 file shrinks by 200-400 Kby when prop pox strikes for the first time and continues shrinking if the users keeps saving the city after it is affected ( see here ). The sc4 city file is made up of a large number of subfiles, identified by TGI (Type, Group, Instance) addresses. It was discovered ( see here ) that the props in the city (including street props) are stored in the subfile of Type 2977aa47, also called the 'network' subfile. This file grows in size as the city develops. When Prop Pox strikes this file shrinks in size by 200-400 Kby with respect to its size in the previous save game file, causing the sc4 city file to also shrink. The next time you open the city those 200-400 Kby of props will be missing. They were lost in the save game process.

6)   It seems a memory overflow somewhere is leading to the corruption of the network subfile and the city save file ( see here ). There was a suspicion that the EP1 update program could be the cause of Prop Pox. Another possible culprit would be a plugins folder infected by multiple definitions of a same prop.

[bap]

Hunting the cause of Prop Pox

Below I describe the series of tests I performed in the search for a cause of Prop Pox.

My game directories are organized as follows. Buildings, Lots and Mods go into MyDocuments/SimCity 4/Plugins (hereafter MP), while Props, Textures and Maxis plugins go into ProgramFiles/SimCity 4/Plugins (hereafter PP). All tests were performed on large city tiles grown on suburbia style (mostly low density RC lots).


A)   Initial frustration

The largest city in my region showed Prop Pox when its save file reached 15 Mby (network subfile of about 6 Mby). As a first test, I developed two comparison cities on a test region  to see if they developed the pox too. Prop Pox appeared on each city as soon as their network subfile went beyond 6 Mby. Prop Pox is not a random problem. It is a systematic bug and will appear in every city the affected user develops provided they grow up to the proper size.


B)   Two basic tests

I uninstalled and installed the game again from scratch, without performing the EP1 and BAT nightlightning upgrades. I also removed all lots from MP folder and started developing test City #1 only with Maxis lots. No custom lots. I saved a copy of the City #1 when it had 13 Mby for future tests. When the city reached 14 Mby (the network subfile reached the magic 6 Mby limit), Prop Pox made its appearance.  I was about to believe there was some silly limitation intrinsic to the game when I realized I grew the city with all custom dependency props at PP folder. Whatever caused Prop Pox it was still there, being read by and affecting the program. (it sounded an obvious after thought, that if one is looking for a problem affecting props, one should concentrate on props and not on building or lots.) I tried removing the custom props from PP folder and further developed the pre-Pox saved version of City #1. It got Prop Pox when it reached 14 Mby. Even if the cause of Prop Pox is no longer in your game folder, if your city has been saved while the pox "virus" was there, you are infected and will get Prop Pox.

I developed another test city, City #2, this time removing everything from PP, including Maxis plugins. So, no custom content at all. Exclusively Maxis vanilla stuff. To my relieve, this city got no Prop Pox (this was the first time I developed a city beyond the magic 15 Mby size without suffering from the pox – lots of beer to celebrate this). I saved versions of this city while I was building it for future tests. City #2.1 has 15 Mby file size with a 5.5 Mby network subfile. When it reached the 15 Mby limit it was still far from the 6 Mby network file barrier. With only Maxis contents, the city has several hundreds of props less to store. I used City #2.1 for additional tests. I tried further developing the city in my laptop (largely different hardware setup) and with different screen resolutions. I got no Prop Pox in any case. Without custom content, no Prop Pox appears. This was a promising start.

Interestingly, the lack of Prop Pox allowed me to see an important effect. When the network subfile goes beyond 6 Mby, it increases suddenly to 16 Mby and the city save file accordingly grows by 10 Mby. This has nothing to do with Prop Pox. It seems a change in the way Maxis handles the information about props in the city. That's why I haven't seen anybody reporting a city save file with 17-24 Mby. It is either 12-16 Mby or larger than 26 Mby. It seems if one gets Prop Pox after the jump in network file size and starts loosing track of the props in its city, one will probably get back to the initial save size and ones city file will jump back to 14-15 Mby size after being on the 26 Mby size for a while. This is consistent with what Snorrelli reported.


C)   Is the EP1 update responsible for the pox?

I installed the EP1 and the BAT nightlightening updates and further developed City #2.1 without anything in PP folder. The city goes up to 27 Mby in size with no sign of Prop Pox. So, the EP1 update does not cause Prop Pox (nor the BAT nightlightening update). I then installed only the custom props at PP and started developing City #2.1 again. Prop Pox strikes when the city reaches 16 Mby. I saved a version of this city prior to the pox appearance (City #2.2). I removed the now suspect custom props packages from PP folder and further developed City #2.2. Prop Pox strikes again at the same sc4 city file size. The cause of Prop Pox must be in one or more of the custom content prop packages. Once the city is saved with the infected file installed, it will get Prop Pox.


D)   Narrowing down the search

I ran SC4Tools (merge file option) on all custom prop packages in search for multiply defined props. It was able to find several conflicts and hundreds of multiple definitions of props. I eliminated all conflicting and doubly defined props from the prop packages, installed the corrected files back on PP, and ran another test with City#2.1. Prop Pox sets in when the city reaches 16 Mby (again, network subfile about 6 Mby). Multiply defined props per si are not the cause of Prop Pox.

The 'virus' was still there. In order to find it, I performed binary search (take half of the prop package files out, test with the other half; if you don't find Prop Pox, you can eliminate the tested half from the exercise; take half of the other half and proceed with the testing). After several such tests, I was able to narrow down the search to only one file: a blend of props from Pegasus CDK3 and OWW2 packages. Only when this package was in PP folder, did the city developed Prop Pox.

I tested developing the City #2.1 with only the texture packs and Maxis plugins in PP folder. No sign of Prop Pox. I then added all other prop packages except for the suspect one, and installed back all lots in MP (except for the lots which depend on the excluded prop package). Almost all my custom content back. No sign of Prop Pox. (another box of bier to celebrate.)

I broke my Pegasus mega prop into its CDK3 and OWW2 parts and performed a few more tests to find that I only got Prop Pox when the PEG-OWW2_BDK_RESOURCE.DAT file was installed. Removing everything except this file from the plugins, I got Prop Pox. Including everything in the plugins except this file (and the associated lots), I got no Prop Pox.

[bap]

Face to face with a cause of Prop Pox

I could have stopped at this point and refraining from using Pegasus' beach package. But I wanted to know specifically what causes Prop Pox and I also wanted to correct the problem, if possible, in order to continue being able to use that great package.

PEG-OWW2_BDK_RESOURCE's file has 19 props, of which 12 are time-dependent props. I did some binary search again. I removed everything from MP and PP leaving only the 7 non-time dependent props and developed City #2.1 again. No sign of Prop Pox. Getting closer.

Except for one, the time-dependent props in that file are modified versions of Maxis props (swimmers, picnic on blanket, beach chair & umbrella, etc) that Pegasus modded to appear only during beach time. One does this by creating another prop (i.e., using another instance number) with the modification from the original one. Unfortunately, four of the modified Maxis props (see below) were kept with their original TGI numbers. But because they were transformed into time-dependent props, their ResourceKeyType1 property (3 numbers to store) is now replaced by a ResourceKeyType4 property (16 numbers to store). This extra information requires additional 73 bytes for each prop.

Now, when SC4 game reads its vanilla files, it sets a buffer of Ni bytes to store the info of prop i in memory. When it later reads again the info of that prop in the custom prop file it find 73 extra bytes of information to store and it keeps writing beyond the expected final memory position of that prop. Thus, it overwrites other things and messes up the program buffer. This is memory overflow, an easy way of loosing track of how many props you have installed and where in memory the corresponding info starts/ends.

Prop Pox seems to be caused by the program reading again an already stored prop the exemplar file of which is of different size at the second read. This causes memory overflow and messes the information about props in the city. This becomes a problem (and leads to perceptible effects) when the number of props in the city is large enough for the network subfile to reach the 6 Mby limit where Maxis changes the way it keeps track of props (or at least the size of the buffer where it stores the props). For a medium size city (4 times less area), this limit should be about 1,5 Mby, For a small size city (16 times less area), this limit should be about 400 Kby. A player may not know he/she is infected by Prop Pox if his/her cities are not developed to the point where these limits are reached.

[bap]

How to avoid the problem

If you want to avoid the above described cause of Prop Pox, you have the following choices:

1-   Refraining from using Pegasus CDK3-OWW2 lots (the beach resource file contains a couple of SSH files used by the other OWW2 lots). Wait Pegasus to release a patch to his package correcting the problem props.

2-   Open the PEG_OWW2_BDK_RESOURCE.DAT file with the ilive Reader, delete the items 11, 12, 13 and 14 (exemplar names R1x1x2_BeachChair_29B2, R1x1x3_PatioChair_290D, R1x2x2_Recliner_2911, and R2x3x2_$$Beachumbrella_2900), and save the file again. The three first props will no longer be time-dependent (the umbrella will still be time-dependent, appearing only during day time). But the lots will work fine. With no Prop Pox.

3-   Open the PEG_OWW2_BDK_RESOURCE.DAT file with the ilive Reader, change the exemplar name (p.ex., add a PEG_ before each name) and the instance of the items 11, 12, 13 and 14 (mark each one and select "generate new instance"), and save the file. Load each of the beach lots into LotEditor and replace each occurrence of these props by the corresponding new PEG_* prop and save the lots. Replace the original lots by the modified ones. They will work exactly as designed, but will no longer lead to Prop Pox.


While the time dependent props of Pegasus' beach pack are a cause of Prop Pox, they may not be the only cause of it. Any other modding of Maxis prop which changes (=increases) the prop exemplar file size will equally lead to Prop Pox. For instance, Pegasus' beach pack was released on 2007 December 11, well after the original Prop Pox reports of BruceAtkinson, Snorrelli and Fiedler2000. Thus, their Prop Pox must be caused by another, possibly similar, memory corruption occurrence.

If you have Prop Pox and does not have Pegasus' beach pack, I suggest you scan your prop packages in search of modified original Maxis props. Changing the values of the properties of a Maxis prop is fine. Changing the number of properties (p.ex., adding a new property) opens a way to Prop Pox. You are also advised to search for multiply defined custom props. They may also lead to Prop Pox if the exemplar of the last read definition of the prop is larger than the first read one.

We found what causes Prop Pox, and we devised ways to avoid it or to find and eliminate the problem. However, at the time of writing there is no way to restore an infected city. I am afraid it would not be possible to restore poxed cities. It seems once the sc4 city file is corrupted, there is no way back. The best way to remedy this limitation is to make frequent backups of the region one is developing and to keep a record of when each custom content file was installed in ones plugins folders. If you find a custom file which leads to Prop Pox you can avoid the problem by eliminating (or correcting) the affected file(s) and by restoring a region backup prior to the installation date of the affected file(s).

It is worth noting that the file merge option of SC4Tools was not able to identify/warn the double definition of Maxis props in Pegasus prop file. It seems the program only searches for multiple definitions of custom-defined props. If this is right, being able to recognize redefinitions of Maxis props in custom files would be a great addition to this program on a future update.


I would like to thank all those batters and modders the work of whom led a great game to became even better. I would also like to thank ilive (Reader), Simrolle & Andreas Roth (SC4Tool). This work would not have been possible without their tools.

[RippleJet]

bap, this is the best analytical process I've ever seen in tracking down an "unsolvable" problem in SC4.
A well earned Karma point from me!

[callagrafx]

While the time dependent props of Pegasus' beach pack are a cause of Prop Pox, they may not be the only cause of it. Any other modding of Maxis prop which changes (=increases) the prop exemplar file size will equally lead to Prop Pox. For instance, Pegasus' beach pack was released on 2007 December 11, well after the original Prop Pox reports of BruceAtkinson, Snorrelli and Fiedler2000. Thus, their Prop Pox must be caused by another, possibly similar, memory corruption occurrence.

Pegasus has other releases out there that I'm sure employ the same modifications to Maxis material....Thank you for identifying the cause which should go a long, long way to helping people keep their hard work, as long as they have the knowledge to reverse-engineer Pegasus releases, until such time as he fixes the problem packages himself. 

A karma from me too, for the same reason as Ripplejet 

[Diggis]

And a 3rd from me.  I usually try to avoid giving multiple Karma for the same thing, but 3 posts, 3 points, a million thanks!

[JoeST]

That is Excellent A Grade information. Glad someone found a solution to it.

Joe

[Pat]

Ohh my wow a solution to the Prop Pox!!! WOW this is awesome Bap and a thank you over a million fold!!!! I wish I could give you a karma point for it...

[Ennedi]

It seems you have enough Karma points for this research yet  but big thanks from me too!
Your messages could be showed as a model of the research work

Adam

[wouanagaine]

awesome testing process 

Thicky thread

[CaptCity]

Bap, even though I have never been a victim of the Prop Pox, I have followed the discussion on other threads whenever I could, and I must admit to getting a chill as I read through your post. And I realized it wasn't because the problem seems to have been solved, but because there are people like you who go to such lengths to make this game enjoyable and playable for all.

While I am a relative nobody when it comes to the mechanics of the game, I do enjoy playing it. And not just because it is a great game, but also because of the SC4 Community as a whole. You (and those who helped you) are a perfect example of the best of that community, and I offer you a huge thanks and 

Wish it could be more...

Sorry if this seems a bit dramatic, but I just had to let it out. 

[BarbyW]

I have found an earlier instance of changing the file size of Maxis props without giving them new TGIs. It is in another Pegasus file - PEG Trail Parks III released in October 2004. This has three Maxis props in the PEG_TrailPark-Engine_305a.dat with the same TGIs but bigger filesize.
As bap seems to have the cities set up to test for the prop pox, would he care to test with this?

Edit: I also found many examples of modded Maxis props with the same TGI in PEG_CDK--IND_205.dat which was released in Dec 2004.

[BarbyW]

Much to my chagrin I also found that I could have contributed to this as well. I found a set of modded Maxis props that I did some time ago that have also been combined into BSC MEGA Props Misc Vol02

I have corrected these and uploaded the revised pack to the LEX. I am also checking other BSC packs to see if there are any other incidences in any other files.

[rooker1]

I would like to thank Bap for the outstanding analysis performed here.

And a big thanks to Barby for taking the initiative to look into the BSC prop packs to assure us all that the BSC work is always as close to perfection as possible.

Thanks to yuo both,
Robin 

[Diggis]

One thing that I am curios about... the existince of one of these files in your plugins, regardless of whether the lot is used or not, will still cause the prop pox?

And when you say it changes in size to the one in the memory... if the file was in the plugins from the start of the city will it still cause a proplem?  And if I get the props out of my plugins now will it mean I won't get the pox?  I have a city on the verge of getting it.. (ie got it, but I have a back up ) so want to avoid it.

[RippleJet]

if the file was in the plugins from the start of the city will it still cause a proplem?

Yes, if I understood bap correctly, you might have a "proplem" (is that the same as a "prop problem"? )

Even if the cause of Prop Pox is no longer in your game folder, if your city has been saved while the pox "virus" was there, you are infected and will get Prop Pox.

However, I'm not sure if that would be the case if the prop has only been in the folder, but not used in any lot in the saved city file...

[JoeST]

If it wasnt used, I doubt it would have been saved to the city file, so wouldnt be there to break. So fixing/removing it before its used should work

[Diggis]

If it wasnt used, I doubt it would have been saved to the city file, so wouldnt be there to break. So fixing/removing it before its used should work

Except from Baps post he indicates he got the prop pox with only Maxis lots used but all his prop dependencies installed, indicating the file is saved in some way... (d'oh, lightblub just came on) expect the prop that was modded was a maxis prop so could well have been used.

[RippleJet]

(d'oh, lightblub just came on) expect the prop that was modded was a maxis prop so could well have been used.

Right, I think all Maxis lots use Maxis props...