What causes Prop Pox (and how to avoid it)

[joshua43214]

A quick update,

I scanned through SimCity_1.dat and made lists of all props that use 9 of the properties that seem like they could cause a problem. Out of the 9, I settled on 5 that made sense. I then made a new dat that was nothing but the prop exemplars from a handful of each type. I then deleted the suspicious property, loaded the game.

So far all I can confirm is that deleting the Nighttime State Change and the Time of Day properties will cause the pox.
It is possible the ones that failed didn't have opportunity to cause the pox. For example, my first test on deleting Nighttime State change failed even though I have found this to be a cause previously. I remade the bat and went from two props in the exemplar to 15 (I made sure to leave out the umbrella on both tries as a control). The second attempt poxed the city.

I then made three grassy parks with one prop each, the recliner, the umbrella, and a bench. I made prop exemplar bats for each one as well. I then loaded each one up and added or deleted properties from the exemplars.
I could then scan the prop save file. a single prop creates a prop save file only 88 bytes long, so it is fairly easy to see changes even if I do not know what the change means.
Near as I can tell, adding or deleting the Nighttime Change State has no effect on the prop save. This property is not saved in what we believe the prop save file to be. It must be saved elsewhere, or there is something entirely different going on. There is a change in bytes 4 - 7, these bytes contain a flag for if the file is compressed or not, and its umcompressed size. I can use the same check for compression and algorithm for decompression that I use on regular files, so I didn't bother to write the code so those 4 bytes are readable. There might be a flag hidden in there though. But like other exemplar files, the reading frame seems to start at byte 9.
The Time of Day property does appear in the save file as a 2 byte flag, and a third 1 byte flag that toggles day/night.

Next step is to dig into the index and look for changes there.
Hopefully something will turn up.

I can post the savegame somewhere I suppose. I had not considered it since all you have to do is make an ugly grid of LD res with some ind and com mixed in.
I will see if I can find a file hosting site that is not about to vanish into the night like the others.

-Josh

[catty]

...I can post the savegame somewhere I suppose. I had not considered it since all you have to do is make an ugly grid of LD res with some ind and com mixed in.
I will see if I can find a file hosting site that is not about to vanish into the night like the others.

-Josh

As a CB member you have 50mb of storage space available in your CBEX briefcase, you just need to click the add button to access it, the only snag is its for registered users and/or visitors with a social login (facebook, google, twitter) will be able to access it

[HappyDays]

I can post the savegame somewhere I suppose. I had not considered it since all you have to do is make an ugly grid of LD res with some ind and com mixed in.
I will see if I can find a file hosting site that is not about to vanish into the night like the others.

We can make our own, true, but then we wouldn't be using the same microscope.

In addition to knowing the city and what to expect, everyone looking to test will have the same base to work from and there's a smaller number of steps involved as 90% of it's already done.

I recommend DropBox for your sending people stuff needs. It's free, no mess, you get a few gigs of space to start with, and it's unlikely to go anywhere.

Also, just leaving this here: https://xkcd.com/949/

[joshua43214]

Another long post.
tldr: you should really read it...(at least the end)

...I can post the savegame somewhere I suppose. I had not considered it since all you have to do is make an ugly grid of LD res with some ind and com mixed in.
I will see if I can find a file hosting site that is not about to vanish into the night like the others.

-Josh

As a CB member you have 50mb of storage space available in your CBEX briefcase, you just need to click the add button to access it, the only snag is its for registered users and/or visitors with a social login (facebook, google, twitter) will be able to access it

Excellent news!
I have been a member of CB for a while, it is a great resource. Your work on CB is in my mind, one of the most under-appreciated resources the community has.
Sadly, I got to working too quickly when I was too tired, and accidentally over-wrote my un-poxed master save.
I will recreate it and re-do the relevant experiments, then upload the save and maybe some dats for people to play with.

I have made a great deal of progress interpreting save files.
The thing that seems to be missing is a save index.
Does any one know if there is a save index? It would be an index of all the saves inside the '0x2977aa47' prop sub-file.
This is important because the game needs to know the location of each prop file inside the sub-file itself.
There are two ways to find each prop file: use an index (like the way a regular exemplar file full of props is), or use the size of the prop file to know when to start reading the next prop file.
The first piece of info in the prop file is its size. You can start reading the next prop file by simply offsetting by the accumulated sizes of the previous prop files. This is a really ugly way to do things because if the file size is wrong, all the following prop files are read incorrectly.

My script uses the file size to know when to start reading the next file.
I have only found two file sizes for props, 88bytes and 108bytes.
presumably there is more because the pseudo code that Ripplejet created would have other sizes.
see http://www.wiki.sc4devotion.com/index.php?title=Prop_Subfile
This page will need a big overhaul when I get a bit further, there are many discrepancies.

an example of a 88byte prop file. As before, all the clear text on the left is mine, only the hex text on the right is in the file.
This also contains information that is used for decoding such as "key_type" and "data_type."

Code Select Expand

size 88
CRC Фp
mem 0x28a2634
maj_v 0x6
min_v 0x4
zot 0x0
unk_1 0x0
ap_flag 0x0
something 0xa823821e
min_tract_x 0x40
min_tract_z 0x40
max_tract_x 0x40
max_tract_z 0x40
x_tract_size 0x2
z_tract_size 0x2
prop_count 0x0
name-val NA
name-val_1 NA
something_2 NA
data_type NA
key_type NA
word NA
prop_val NA
TID 0x6534284a
GID 0xc977c536
IID 0x1eda0000
IID_1 0x1eda0000
min_x 2.30816478127e-20
min_y 2.63198852539
min_z 270.0
max_x 43.9499969482
max_y 6.63198852539
max_z 273.0
orientation 0xcc
state 0xcc
start_time 0
stop_time 0
count 0x3f
A 0x42
B 0x2
C 0x0
D 0x0
E 0x64
lot_type 0x2
obj_id 0x8c356905L
cond_ap 0x0


This is an example of a 108byte prop file

Code Select Expand

size 108
CRC Фp
mem 0x28a2974
maj_v 0x6
min_v 0x4
zot 0x0
unk_1 0x0
ap_flag 0x0
something 0xa823821e
min_tract_x 0x40
min_tract_z 0x47
max_tract_x 0x40
max_tract_z 0x47
x_tract_size 0x2
z_tract_size 0x2
prop_count 0x1
name-val 0xcaa46595
name-val_1 0xcaa46595
something_2 0x0
data_type 0x00000003
key_type 0x00000000
word 0x0
prop_val 0x100
TID 0x6534284a
GID 0xc977c536
IID 0x29f10000
IID_1 0x29f10000
min_x 1.07025499574e-13
min_y 4.0
min_z 270.0
max_x 455.5
max_y 12.0
max_z 273.0
orientation 0x0
state 0x40
start_time 0
stop_time 0
count 0xe4
A 0x43
B 0x3
C 0x0
D 0x0
E 0x64
lot_type 0x1
obj_id 0xffffffffL
cond_ap 0x0


if you look closely, there are some obvious groupings that could be made.
This is a reformatted version that looks an awful lot like what we see in the Reader

Code Select Expand

size 108
0x28a2974
maj_v 0x6
min_v 0x4
zot 0x0
0x0
ap_flag 0x0
0xa823821e 0x40,0x47,0x40,0x47,0x2,0x2
prop_count 0x1
0xcaa46595 0xcaa46595,0x0,0x0,0x100
TGI 0x6534284a,0xc977c536,0x29f10000
0x29f10000 1.07025499574e-13,4.0,270.0,455.5,12.0,273.0,0x0,0x40,0,0,0xe4,0x43,0x3,0x0,0x0,0x64,0x1
0xffffffffL 0x0

There is a certain pattern that always occurs at a few points, importantly, notice that the IID is always given twice in a row

Code Select Expand

TID 0x6534284a
GID 0xc977c536
IID 0x29f10000
IID_1 0x29f10000
notice also the name value is given twice in a row

Code Select Expand

name-val 0xcaa46595
name-val_1 0xcaa46595
I started looking for corrupted files. Because my script offsets to the next prop based on the size of the previous prop, my script would crash when it hit a corrupted file length.
here is an excerpt from the last "readable" file in a poxed city.

Code Select Expand

size 1154551993
<snip>
zot 0x6400
<snip>
something 0x2a46ee
<snip>
prop_count 0x11569674
<snip>
prop_val corrupted
TID 0x4000000
GID 0x40006
IID 0xa823821e
IID_1 0x615e615e
according to this record, this prop file is about 1.1Gb. Notice also that the IID does not match the IID_1
The zot is not a valid zot type
the prop_count (property count, not "prop") is to big

If the game does not have an index for prop file locations, then all the following props will be lost not be read, and you will see a poxed city.
This corrupted file however is not the cause of the pox, it is a symptom

This is an excerpt from the previous prop file:

Code Select Expand

size 88
<snip>
zot 0x6
<snip>
prop_count 0x20002
name-val NA
name-val_1 NA
something_2 NA
data_type NA
key_type NA
word NA
prop_val NA
TID 0x89a1c16c
GID 0x1
IID 0x89a1c16c
IID_1 0x0
Again the IID does not match the IID_1
again the zot type is invalid.
more importantly, the property count is wrong. It says there are 131074 (0x20002) properties. In all the files I have looked at, the property count is either 0 or 1 (Ripplejet recorded values of 2 and 3 as well).
If the property count is 0, the file is 88bytes long. if the property count is not 0, the file is 88 + the length of the additional properties (in my case this is always 20).
The IID is 4 bytes long.
I modified the script to compare each 4 byte segment to the following 4byte segment. If I could find the offset where there was two segments that were identical, then I could find the proper offset for the rest of the file.
This condition did not exist, the data is corrupted.

I believe that this is supposed to be a prop record greater than 88bytes. If say this was supposed to 108Byte record, the following prop file would start 20 bytes later than it currently does.
offsetting the next (the last "readable" file) file by 20bytes yields this:

Code Select Expand

size 2770670
CRC yЫ
mem 0x101f2168
maj_v -0x698c
min_v 0x1156
zot 0x6
unk_1 0x4
ap_flag 0x4
something 0x4000000
min_tract_x 0x1e
min_tract_z 0x82
max_tract_x 0x23
max_tract_z 0xa8
x_tract_size 0x615e
z_tract_size 0x615e
prop_count 0x20002
name-val 0x1
name-val_1 0x89a1c16c
something_2 0x89a1c16c
data_type 0x00000000
key_type 0x00000000
word 0x0
prop_val Null
TID 0x36000000
GID 0x0
IID 0x4ac977c5
IID_1 0x653428
The property value is Null!
this appears to be another instance of the previous prop (there is a single record for each and every individual prop).
Notice the size is still wrong.
Notice that name_value_1 = something_2. name_value_1 is at offset 20. The bad data that precedes this actually belongs to the previous record. The key_type and data_type are both valid. The property count should be something valid.

Adding a loop right before the TGI value to bump the offset by 1byte until IID = IID_1 (turned out to be offset 6)
yielded this

Code Select Expand

TID 0x6534284a
GID 0xc977c536
IID 0x29000000
IID_1 0x29000000

Does anyone want to guess what prop has this TGI?
This is the TGI for the beach umbrella

Offsetting the next record by 112 to compensate for the corrupted previous file yields a perfectly good prop file (yet another umbrella). Offsetting again by 108 (the correct length) yields another umbrella (perfectly good record).

So this adds more support to my theory about a null value corrupting the file, and data reading from or into another file.
The upshot here is that in this case, the data is not being overwritten, just missed.
This means it might (might) be possible to actually cure prop pox

Next is to start trying to figure out what the values actually mean in relation to what kind of prop is being recorded

-Josh

[Simcoug]

Hats off to you Josh!  I am by no means a computer programmer, but your post made quite a bit of sense.  It seems like you are definitely on to something here 

[jdenm8]

I thought it was it over-writing the next entry. Turns out it's just the size being recorded wrong. Hats off to you for the in-depth analysis. If we had the CRC algorithm for the network subfile, someone could probably build a program that fixes it.

[catty]

...I have been a member of CB for a while, it is a great resource. Your work on CB is in my mind, one of the most under-appreciated resources the community has....

Thanks, as for the CBEX briefcases you can upload (in theory as no one has ever tried it) a file up to 25mb in size, if you need to upload something bigger then I should be able to tweak the CBEX settings to let you, the other thing to be aware of is you cannot delete any files you upload the reason being the CBEX uses group permissions so if I give registered users delete rights to the CBEX then they can delete anything, their stuff ... your stuff ... the official MAXIS files, etc so if you do upload something and you want to delete it then just send me a PM and I'll delete it, I hoping the company that makes the CBEX software will eventually change this from group permissions to individual permissions, but in the meantime its a pain.

If I was still on the SC4D staff I'd give you a karma point for the research you are doing re the prop pox   

That IID_1 column did ring bells with me as something I'd read about in the past, did a bit of search and found it was mentioned by a fellow RTMT team member in a post on traffic signals

http://sc4devotion.com/forums/index.php?topic=5136.msg265392#msg265392

[Turjan]

@joshua43214: I just have to say that this is an awesome piece of research. Thank you very much for this. I'm really looking forward to seeing where this goes. Let's hope there will be some more insight at the end of this, in addition to what you already found out.

Fascinating. Concerning, as well. There are many prop packs on this site alone that override props from other prop packs, and add/remove various things. Unless there are more specific limits on what causes prop pox than what you've described, many people could be infected.

One example of many:

BSC Mega Props - JES Vol01. Instance 10A83059 (jes_BN-Line-53_Container) conflicts with the one contained in BSC MEGA Props - Misc Vol02. Same basic prop, different properties.

BSC MEGA Props - Misc Vol02 is such a PITA. Unfortunately it was made a dependency for BSC Essentials. It really messes with existing plugins, like adding more props with bigger footprints to existing families in BSC Mega Props - JES Vol01, which makes many lots that use the latter dependency look pretty ugly. I had deleted it from my poxed city, too. (Disclaimer: No, that is no indication it is in any way responsible, as I deleted too much stuff to really pinpoint anything). I was thinking of deleting most of the questionable stuff out of this file for my next region.

Sorry for the tangent, as this is probably not related to prop pox, but this dependency is a bit of a sore spot with me.

[HappyDays]

Many of the older mega packs, the ones assembled from props made in the pre-SC4devotion days, are a bit messy. BSC MEGA Props - Misc Vol02 is the tip of a very large ice burg.

That said, I am impressed the BSC team did as well as they did sorting things out without the benefit of DataNode. We're talking tens of thousands of props of all levels of quality. That things work as well as they do, that's something to be commended.

My main issue with mega packs is that, generally, you're not using half of what the pack contains, with the rest adding to load times for no benefit. That's a rant for another time, though.

[Turjan]

That said, I am impressed the BSC team did as well as they did sorting things out without the benefit of DataNode.

Oh, yes, certainly. Just to make sure, I didn't want to denigrate anyone's work here. I'm very happy for all the wonderful content I got the opportunity to use, and I'm thankful for that. In this specific case, I'm just unhappy about these multiple interdependences, which cause stuff like this to happen, and it's a bit difficult to say "just don't use this" if we talk about a dependency for BSC Essentials.

Regarding Mega Packs, it's a trade-off. I guess they are the result of complaints about "too many dependencies".

[joshua43214]

Teaser...



WIP...

[Turjan]

I like the title of this .

[catty]

Looking good   

[joshua43214]

I have some updates.

I made a pair of new test cities. The first is a large tile with roads, some minor amenities, and fully zoned ready to run. The second is the same city just before the prop file decompresses. I have confirmed that it does not pox on it's own, but will pox when the BDK file is added. I need to double and triple check to make sure it is all kosher, then I will upload it to the CB.

Next, I have successfully removed the corruption from a poxed save game, generated a new save game file that the game loads and runs with no crash dump errors. The new save is "pox free" according to the SC4 Savegame Explorer, and it opens and looks fine in the Reader.
Sadly, it will re-pox after the game has run for a while.

Which leads me to the next part, I think I have tentatively found the actual cause of the pox. I was wrong about the null value messing up decompression, nulls seem to occur when the file corrupts and is an effect, not a cause. The issue seems to be faulty state flags either during a timed state change or during re-development.

Below is a truncated table of experimenting with plopping various props. I used the SC4 Lot Editor to add a bench, an umbrella, patio furniture, and a lotcarcluster (commonly found in commercial lots) to individual grassy parks. These are all un-modded Maxis props, and each has a single timed property different from the others. I would plop the lot, save, wait through a day/night cycle and save at various points during the cycle, and run my script over the save game to parse out the info. I would then delete the city, and create a new city for the next plop test.




   
   
   
   
   
   
   
   
   
   
   
   

bench

umbrella NSC

patio furniture PTOD

lotcarcluster ALS

day

night hidden

night visible

day 2 visible

day 2 late hidden

day

late day

night

day 2 late

day no power

day power

night

size

88

88

88

88

88

88

88

88

88

88

88

88

88

ap_flag

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

0x5

prop_count

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

0x0

name-val

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

name-val_1

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

prop_val

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

NA

state

0x0

0x0

0x1

0x0

0x0

0x1

0x0

0x1

0x1

0x0

0x1

0x0

0x1

start_time

0

90

90

90

90

90

120

120

120

120

70

70

70

stop_time

0

180

180

180

180

180

160

160

160

160

200

200

200

cond_ap

0x0

0xf

0xe

0xe

0xf

0xe

0xf

0xe

0xe

0xf

0x7

0xf

0xe

KEY:
NSC = Nighttime State Change, this is the property deleted from the BDK umbrella.
PTOD = Prop Time of Day, this is the property added to the BDK umbrella.
ALS = Active Lot State, just another timed timed prop for comparison.

According the work done by Ripplejet and published here:
http://www.wiki.sc4devotion.com/index.php?title=Prop_Subfile
a conditional appearance flag (cond_ap) of 0xe is a timed prop in state 0x1 (hidden/off). Note that the umbrella (NSC) can have a cond_ap flag of 0xe and yet have a state of 0x0 (visible/on). Both the patio furniture (PTOD) and the carcluster (ALS) follow the rule.

Either this confusion of timing and state flags is the issue, or there is a problem elsewhere in the savefile.

When a new grassy lot with a single prop is plopped, 5 new entries are added to the index (the prop, lot, building, type 10, and an unknown sub file). The unknown file is 91 bytes long. I spent some time trying to figure out what is in it, but nothing obvious came up. The TGI for this file is '0xc97f987c', '0x299b2d1b', '0x00000000' if any one has some ancient records floating around of what this might be. It does not seem to have either the IID or the instance IID for the prop.
Some of the other sub-files in the save game (there are 147 total with one prop) get bigger as well.

If the issue is with conflicting flags, each prop would have to be checked against a library of props and the flags individually corrected. This is not inconceivable, there are only 65 props with NSC and only 521 with PTOD in the vanilla game. The issue is that the other causes of prop pox besides the BDK are not found and eliminated. I suppose there is no reason the plugins folder could not be scanned and a list of all props and their properties could be created. My i7 with 16 Gb of RAM can handle this with no issue, but it could easily crash on a smaller system. I was considering making this script anyway to scan for this issue between modded props. As someone already stated, some of the BSC prop files do a lot of replacing other BSC props. I would really like to pin down the actual cause first.
At this point all we know for sure is that deleting the NSC and adding the PTOD properties to a prop causes the pox. Is the reverse true? The ALS files concern me to, parking lots are a popular item to mod.

-Josh

[catty]

...The TGI for this file is '0xc97f987c', '0x299b2d1b', '0x00000000' if any one has some ancient records floating around of what this might be....

Can't stop to read as I'm at work, but 0xc97f987c comes up in the Sims Wiki in the document they were recording the SC4 SAVEGAME details and which bits they had decoded

http://simswiki.info/wiki.php?title=SAVEGAMES

Its just above the "Tire Recycling Ordinance" that document was transferred to that site back in 2005 when the old wiki was shutdown so its really old, I don't know if anyone else has tried decoding any more of the SAVEGAME

[catty]

A few years ago I used SC4LotInfosGenerator and scanned my plugins, the LEX DVD and CD and the STEX Disks (2, 3 and 4) ending up with a huge number of duplicates records obviously and as I only wanted to know the name of the plugin and its file name I ended up with an Microsoft ACCESS database containing this information

Code Select Expand

GALLERY File Name
0xEE668F70 SG_Lots_Motels.dat CS$$4_3x3_SG_BrewPubFishTaleAle
0xEE668F70 CS$$3xSG_BrewPubFishTaleAle_ee668f70.SC4Lot CS$$3xSG_BrewPubFishTaleAle
0xEE66901B SG_Lots_Motels.dat CS$$2_2x5_FishTaleInn
0xEE66901B CS$$2xFishTaleInn_ee66901b.SC4Lot CS$$2xFishTaleInn

57215 lines in total, I've exported it out as a text file (its still 4mb in size) and put it "dino's" briefcase over at CB in case anyone wants it.

I've deleted this from my briefcase, did a much needed tidyup re getting rid of duplicates and have started putting it somewhere everybody can view, it will take a while as you can only have about 500 lines per post

http://city-builders.info/apps/canvas/25-discussions/group/17-simcity-4?customView=item&discussionId=47

[alexr19]

I'm currently experiencing Prop Pox. I haven't had a chance to read the entire thread, but does that 6MB filesize theory still hold? I ask because the city in which I'm experiencing PP has an overall size of exactly 10MB, but the filesize is 4,133,617 in ilives. Another city has an overall size of 44MB, and it's filesize is 14,134,667, but no PP.  The bigger city is about 800k and the other is around 450k.

[mgb204]

Sorry point of order first: are you sure you have prop pox, this must be the #1 misdiagnosis I hear. Please can you state exactly what is happening and when it started happening (if anything changed in your plugins for example). This information helps to make sure you are being given the correct advise.

As for file sizes, these are irrelevant, it's a specific part of the file that is being refereed to not the actual file size itself, such sizes are quite normal for developed tiles.

[dyoungyn]

All,

I post this again as this has fixed my Prop Pox issue.  I have studied this ever since the game came out and what I have discovered is the following:

I appears to all be with ones plugins.  When one downloads a plugin and builds the city.  When the creator makes an update change to the plugin, one re-downloads the same plugin and goes back into the city with the plugin, and wa-la, Prop Plox.  I have made the painful decision to NEVER re-download any more updates to ANY plugins and deal with what I have and ROUTINELY back up EVERYTHING even my Regions.  By me demolishing an effective map and starting all over again and fixed my Prop Plox issues. 

Now, I am no programmer or computer genius, just a die-hard SC4 player whom uses the excuse to demolish and modernize my maps with the latest NAM standards.  Now, what I have not concluded is that latest NAM is part of the Plugin Update fiasco or not; thus far, it appears not be a Prop Plox demons.

Just a little food for thought and what have been working for me.

dyoungyn   

[alexr19]

mgb204, yes I believe so. I did recently update the game from the xx38x to xx40x, but nothing else has been recently downloaded. However, the game has begun to crash at exit. I can ctrl-s save all day, but when I attempt to save and exit to region it will actually save but crash to desktop before I get back to region view. If I simply decide to exit it wont exit smoothly, but instead will kind of crash to the desktop. That's a recent change.

EDIT: I did recently update the Mipro Essentials file and the METMW Prop Pack from 1.2 to 1.3 as a dependency for a new building. I don't think it would have been the mipro because I played with it for some days, but the METMW is within the past few days.