• Welcome to SC4 Devotion Forum Archives.

What causes Prop Pox (and how to avoid it)

Started by bap, February 24, 2009, 08:37:13 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

metarvo

Oh, man!  It's a shame you're going through this with all the good work you're doing, CT14.  Unfortunately, reading the part about the 1x2 residentials scares me since I tend to have a lot of them.  If I ever do have this happen to a large tile (which I tend to build on a lot), the plan is to rebuild that area on medium tiles.
Find my power line BAT thread here.
Check out the Noro Cooperative.  What are you waiting for?  It even has electricity.
Want more? Try here.  For even more electrical goodies, look here.
Here are some rural power lines.

twalsh102

CT14 and all others:

Despite the efforts of mgb204 and other long-time players to dispel the myths surrounding "Prop Pox", these myths still continue.

To try to be clear once again (and this is all spelled out in the initial post on this topic):
1.  Prop Pox occurs under very narrowly defined circumstances:
     a)  A prop of a certain type (in the only KNOWN cause of Prop Pox - an untimed Maxis prop) is modded to be a different type of prop (in the only KNOWN cause of Prop Pox - a timed prop), and keeps the same IID as the original prop.
     b)  The group ID has nothing to do with the problem.  The relative sizes of the actual prop exemplar files is not the direct cause of the problem.  The problem is when the savegame file is created, a certain amount of space is reserved for different types of exemplars.  Because the Maxis exemplar is saved first, x amount of space is reserved.  When the modded exemplar with the same IID, but with additional information is saved, the savegame file tries to overwrite the original information.  The additional information causes corruption because of trying to write x + .5 amount of information in a space reserved for x amount of information.
     c)  This has been found to occur in only one specific file created by Pegasus, and in only a specific subset of the prop exemplars included in that file.  No other Pegasus content has ever been proven to cause problems (and there has been a plethora of testing accomplished over the years).  No other content by other authors has ever been proven to cause the Prop Pox.  Note that the existence of the Prop Pox does predate the known problem Pegasus file.  So there WERE other files that likely caused the Pox available at one time.  But none of these potential trouble makers has ever been identified, and are not likely to be still available anywhere.
     d)  Even if you have never used any of the affected Pegasus props, if you have ever started a large city (I will not go into the esoterica of why only a large city - again, this is spelled out in the original post(s)) with a non-modified copy of the affected Pegasus file in your plugins folder (even if you later removed it), then that city is already essentially "infected", and its only a matter of time if you continue to develop that city.
     e)  Modifying a Maxis prop will not in and of itself cause Prop Pox.  Even doing so and keeping the same IID will not in and of itself cause Prop Pox, even though this is a bad practice.

So, in summary:
1.  There is no list of files known to cause Prop Pox because there is only the one known file spelled out in the original post.
2.  There are no other files by Pegasus or any other author that are known to cause Prop Pox currently available.
3.  The Prop Pox will not occur outside of a narrowly defined set of circumstances (spelled out in the original posts).
4.  You will need to address why the only file known to cause Prop Pox has not been locked, with the Administrators at the only known location it can currently be downloaded (STEX).
5.  This conversation could be ended if people would just go back and read the original posts, and base any questions off of those posts.  None of the other 36-37 pages of posts that have occurred since then have brought to light any new information about any new PROVEN problem files.

Wiimeiser

#742
So every city I've started since circa 2010 is corrupt from frame 0, for having a file I rarely use... In that case it genuinely surprises me that the file still hasn't been locked yet. I can't find it on the STEX at all, so something might've been done about it regardless.

Sidenote, I now have evidence that the game save props on an individual basis; I was missing the prop file for that bridge, installed it, reloaded the city and replopped the top lot.

EDIT: This is the offending download, correct?
Pink horse, pink horse, she rides across the nation...

twalsh102

So every city that you've created since 2010 has the potential to develop the Prop Pox.  It doesn't matter whether you have ever used anything from the problem file.  If you have ever saved the game with the problem file in your Plugins folder, it overwrote the Maxis exemplar having the same IID in the savegame file, hence providing the "seed" for the Prop Pox.  However, the fact that you have the potential to develop the Prop Pox, doesn't mean you will ever see it.  A city still needs to meet a limited set of circumstances before Prop Pox is more or less guaranteed.  For example, I don't recall anyone reporting a verified case of Prop Pox in a city that is not a large city square.

If you've got cities that you've been developing for 10 years and haven't yet experienced Prop Pox, that means they don't meet the limited set of circumstance for Prop Pox to actually occur.  If you don't continue to develop them (assuming the large city square size factor), the Prop Pox is not likely to occur.

Please go back and read (or re-read) the first 3 posts where bap explains in detail what causes Prop Pox and what conditions must exist in a city before it will appear.

Check reply #36 to this topic.  Ripplejet provides a quick tutorial on how to check if a particular city meets the circumstance necessary for Prop Pox to appear.  Also read any replies in this topic that reference SC4Savegame Explorer (created by wouanagaine), which is a tool that will scan your savegame files and tell you whether the files have been corrupted by Prop Pox.  Also check the topic for this tool under Third-Party Tool Discussion.

Sorry, the link in your edit didn't come through properly.
The problem file can be found in this download:  http://community.simtropolis.com/files/file/19111-peg-oww2-bdk-beach-development-kit/

Wiimeiser

It's not as huge a deal as it is for most people, since I have a habit of just abandoning my regions and starting over every so often. I've been gone for a while, and a fresh start on regions after some practice might do it.

I've removed the exemplars in question, but I could potentially still get it through missing props, like the Venetian canals, at least in theory, if the way props are saved and my screenshot of the two bridges may suggest.

Is it odd that the modified file is 399kb on my internal hard drive but 400kb on my external one?
Pink horse, pink horse, she rides across the nation...

mgb204

Quote from: twalsh102 on May 06, 2017, 10:35:43 PM
     e)  Modifying a Maxis prop will not in and of itself cause Prop Pox.  Even doing so and keeping the same IID will not in and of itself cause Prop Pox, even though this is a bad practice.

For the most part you have summed up the problem very nicely. Although I must disagree that overriding Maxis props or other items with the same Id is bad practice. It's the exact opposite, a very useful and frequently used modding technique. As you point out, it's only a problem when you use an override and change the properties to a different RTK type. Although the latest information points to the problem being a result of the night-time state change property.

If prop pox concerns you - DON'T install the Beach Development Kit (BDK) by Pegasus. Or, fix the files as outlined in the first page of this thread.

If you don't have the BDK file in your plugins folder and never did have it at any time during the creation of your region, you should not fall victim to this issue. There is a caveat though, people seem to be having the problem anyway. My conclusion is there is more to the problem than explained in this thread. I think many people incorrectly identify Prop Pox as the source of their problems. But it also seems probable that either some other files exist that cause it, or that such file corruption is possible through other means we don't yet have proof of.

Yes the original files (BDK) are still on the STEX. They can not be patched without the explicit consent of Pegasus (I tried), so that's pretty much not going to happen IMO. At the same time, would we prefer these wonderful items were simply locked and never made available ever again? I know that's not what I'd want to see either. So I'm glad they are available on the STEX. The frank reality is that even with these files, most users are unlikely to ever see problems. But it depends upon how you build your cities as much as having the file installed.

CT14

#746
@twalsh102 Indeed I do have the dependencies for that beach lot installed, so there's probably nothing more to it as the city seems to meet all conditions for the classic pox case. I'll check with the Savegame Explorer.

I appreciate your summary of the pages I didn't read.

Edit: here's the Savegame Explorer view... I will remove the beach lot's dependencies, and do some "split testing" to make sure the next city with a prop file size past the threshold does not trigger pox.



Edit 2: Unfortunately, in those first few posts, bap was not precise about which files he removed to cure his pox. He says it was the BDK resource file and "associated lots", so presumably means the other BDK file "PEG-OWW2_BDK_Beaches_110.dat".

Here's the thing - I don't have the BDK installed and never have on this city...but I do have its listed dependencies installed. If bap actually removed those files too to cure his pox, things would make more sense. I do see that he's referring to specific exemplars in the BDK resource file.

I finished the SC4Reader run, checking Plugins TGIs against SimCity_1.dat's group ID c977c536 exemplars, and it found a number of shared TGIs, but of course they will need to be examined closely for the types of changes in properties that might cause the buffer overflow problem. And I also need to check against *all* group IDs... From the sound of it and all the testing that has been done in the past, most will be false positives. I also need to re run it with a few DAT packs unpacked, to narrow down some hits.

Edit 3: temporarily installing the beach mod, then running the SC4Reader scanner for the exemplar mentioned by OP shows just these four overlapping instance IDs in the beach resource file, which is less than expected: 29b20000, 290d0000, 29000000, 29110000. There are others in related files, so I suppose split testing should start with remaining TGI overlaps from the same source.

And: side by side examination of the properties differing from Maxis in the confirmed pox source (for example Beach Umbrella prop 29000000), with the properties differing in other exemplars from that group (for example Effect Small Fountain 2a7ccda1 from the Seaport Village Resources dat), shows the exact same characteristics which are said to lead to the buffer overflow: a change from Nighttime State Change property to Prop Time of Day property, and corresponding property type change from Uint8 to Float32.

Now that I've done that sanity check, I'll split test and narrow the source down.

Kitsune

I got one tile that has prop pox - the content in it is in every other tile in the region at the moment. The tiles that SC4save can open have 0 disabled props, however there are a couple where it runs out of memory (and the files themself are around 55mb in size). The tile in question is 63.9mb at the moment., and was doing fine until I built a road connection. Is it possible for file size itself to cause the issue?
~ NAM Team Member

Wiimeiser

A few questions might help:

Can we get a few screenshots of the tile?

Can we also get the size of your plugins folder?

Is this lot in that particular tile?

Does starting new cities pox them too?

From what I've heard, if a plugin causes the pox it will cause it by simply being installed, so if only the one city is affected, then yes, there is a size limit.
Pink horse, pink horse, she rides across the nation...

mgb204

Quote from: CT14 on May 07, 2017, 10:01:57 AM
Edit: here's the Savegame Explorer view... I will remove the beach lot's dependencies, and do some "split testing" to make sure the next city with a prop file size past the threshold does not trigger pox.

You misunderstand how this works. If you have ever had the file (and ONLY this file), PEG-OWW2_BDK_RESOURCE.dat installed, then your save files are already affected.

  • Even if you do not see any symptoms, once a city has been saved with this file in your plugins, it is sufficient to trigger a series of events which CAN lead to Prop Pox.
  • Prop Pox will only ever occur IF the network subfile is converted, as it reaches the limits outlined in this thread.

In short, so long as those other cities do not have the Network Subfile convert to the second format, you should not see Prop Pox in them. But, the underlying trigger will exist in the save files and any older backups of them, once there, it's always there.

QuoteEdit 2: Unfortunately, in those first few posts, bap was not precise about which files he removed to cure his pox. He says it was the BDK resource file and "associated lots", so presumably means the other BDK file "PEG-OWW2_BDK_Beaches_110.dat".

No. ONLY the main resource file (RED) listed can cause problems. The other related items simply use these props on the included lots, which is very different. In the event the BDK Resource file isn't there, they will use the Maxis default versions of these props. However, if the BDK resource was removed entirely, then you'd probably see brown boxes and missing items on these lots too. So in reality, why keep the beaches if you are deleting the dependency for them? Or put another way, if you do decide to delete the BDK resource, you should also delete/remove any content that lists that file as a dependency for it.

QuoteHere's the thing - I don't have the BDK installed and never have on this city...but I do have its listed dependencies installed.

Either you have the BDK resource file or you don't, that's the only question here? Sorry but this statement seems to contradict itself. Do you mean you never used the content, but the files were in your plugins folder?

QuoteAnd: side by side examination of the properties differing from Maxis in the confirmed pox source (for example Beach Umbrella prop 29000000), with the properties differing in other exemplars from that group (for example Effect Small Fountain 2a7ccda1 from the Seaport Village Resources dat), shows the exact same characteristics which are said to lead to the buffer overflow: a change from Nighttime State Change property to Prop Time of Day property, and corresponding property type change from Uint8 to Float32.

Not really, there has never been anything more than the suggestion that changing the Nighttime State Change property might be the real cause. Not to mention, that was by one user only and never really got verified. The problem being that all these years on, everyone just accepts the answer outlined in the first post of this thread as gospel, when the evidence points to a wider problem. However, since no definitive data exists to give us a clearer understanding of the problem, it's very hard to protect yourself against this issue. Given the community activity levels and general lack of assistance/user participation for testing anything, the odds of progressing on this are very low indeed.

The elephant in the room is, we don't really know much about the problem.

As mentioned previously to others, unless you are an expert with a completely thorough understanding of the ins and outs of SC4/modding, realistically these are your options:


  • Remove the file PEG-OWW2_BDK_RESOURCE.dat and never use it again.
  • Remove or Fix the props as mentioned in this post, from the file PEG-OWW2_BDK_RESOURCE.dat.

If you don't understand the instructions in point 2, you are wasting your time going any further with this. You might as well remove the file (step 1) and be done with it. If you can't properly interpret the data you see, how can you reasonably expect to find something no one else has in the last 10+ years? Of course if you want to spend your time checking into it, knock yourself out, I won't stop you. Just trying to inject a dose of reality into the picture. Which is that the odds of you finding something others haven't is very small, rather than worrying about it, accept the worst has happened, protect yourself as best you can and move on with a new region. That's my (albeit typically dislikeable) advice.

Quote from: Kitsune on December 14, 2017, 06:21:20 PM
I got one tile that has prop pox - the content in it is in every other tile in the region at the moment. The tiles that SC4save can open have 0 disabled props, however there are a couple where it runs out of memory (and the files themself are around 55mb in size). The tile in question is 63.9mb at the moment., and was doing fine until I built a road connection. Is it possible for file size itself to cause the issue?

Quote from: Wiimeiser on December 15, 2017, 03:13:08 AM
From what I've heard, if a plugin causes the pox it will cause it by simply being installed, so if only the one city is affected, then yes, there is a size limit.

That's hardly proof now is it?

We know that save files can get quite large, in fact that's exactly why the Network Subfile needs to be converted to a different format when a city gets beyond a certain size. So it can continue to save the data. So is there a size limit?, yes absolutely, it's only logical that a given data format will have an absolute limit. However, it's also logical that it would be pretty much impossible to ever hit that limit.

So what is most likely going on here? The application SC4 Save is simply unable to read the file, possibly a RAM issue, or an impatience one. You'd be surprised how often Windows will tell you an application has crashed when it's just busy in the background. Either way, if SC4 itself can load the file, you've not reached a limit the game or file system can't handle. Never in the history of this thread has any link been made between a large city/save file and problems of this nature. Bereft of some actual evidence to the contrary, I'm not convinced this is a sound or rational theory.

Kitsune

Thats the thing: those props are no where in my folder. The only peg lots I have are a couple seaport stuff and the avenue median. Interestingly, I found another tile that is not poxed but has disabled props - and I found ground zero for it as the lot was missing its props: something from another creater that I simply shrunk the lot in lot editor and deleted everything except the prop and then added a couple more. The new lot itself was prolly destroyed and replopped a couple times to properly place the props. The lot does contain an animated prop that I moved ever so carefully. At somepoint it got flagged and disabled, and the nature does suggest some modern digging may lead to a DLL that can change the parameter of whatever check its causing it to be disabled. Also - part of rl job is to triage programs that have issues that I may not have ever heard of (hey the joy of ITIL). Sometimes you go up the wrong tree, sometimes you notice things that have gone unnoticed in the past and sometimes you hit the golden nugget.
~ NAM Team Member

Kitsune

I did a ton of reading of the subject this week - is it correct that Large tiles with a network file that exceeds a unknown size gets poxed? And that smaller tiles (and possibly Large tiles with a large water content) are immune as the network file never exceeds that unknown size?

Also - here is a file with prop pox...



A file with disabled props that is slowly climbing and while not flag as poxed - it is in proto-pox and a ticking time bomb:



And a free of everything city:



We can see when something gets poxed - they do not only get disabled but the entire line is corrupted. Cause is unknown on this one.

On the tile that has disabled props - we can already see the beginning of Pox. Although not flagged as poxed, you can see multiple line (and I can confirm thousands) have lost there TID, GID, IID and its only a matter of time before corruption occurs somewhere. I will note these props are still visible at this point as confirmed by sc4save and the game itself. For the disabled 7 props - I do wonder what would happen if we reflagged them to visible. While sc4save does not have modify capability on .sc4 city file due to a checksum issue according to that apps development thread.... I tested and confirmed iLives reader can modify the .sc4 city file. So it does seem if somebody took some coding from ilives reader we could get an app to reset the flag to try and stave off full blown pox. The cause in this case is the usage of two different plugin folders and merging the two together.

I included the last tile as an example of a normal happy tile with multiple visibility types.
~ NAM Team Member

mgb204

#752
Quote from: Kitsune on December 20, 2017, 10:35:19 AM
is it correct that Large tiles with a network file that exceeds a unknown size gets poxed? And that smaller tiles (and possibly Large tiles with a large water content) are immune as the network file never exceeds that unknown size?

The theory outlined here is based on the use of predefined data blocks as part of the save file. Think of this a bit like a card file system, one card or unit of data, can only contain a specific number of characters. The limitation in such a scenario is not actually based on the file size, technically it's based on the array size, data types and input length combined. When all the data blocks are full though, it's very likely that any given set of data at this point would be roughly the same size. Which is where we get the 6MB size before the Network Subfile needs to be converted, in order to store more data in an altered state.

Theoretically, assuming you could fit enough props into a small or medium tile to hit the same ceiling, the problem could exist for them too. In reality though, it does seem pretty unlikely when you consider that you have 4 times the space in a large city over a medium one. It's the same with a medium over a small tile, so for small tiles the odds are very low. Given this, unless your goal was to fill a tile with props, it's simply not very likely under normal use for this to occur.

QuoteI do wonder what would happen if we reflagged them to visible. While sc4save does not have modify capability on .sc4 city file due to a checksum issue according to that apps development thread.... I tested and confirmed iLives reader can modify the .sc4 city file. So it does seem if somebody took some coding from ilives reader we could get an app to reset the flag to try and stave off full blown pox.

When you loaded the game, it would probably delete your save file. Because that's what the Checksum is there to do, prevent the save files from being modified in any way. That's why SC4Save doesn't have the facility to edit the files, because whilst it could save the data, just like iLives reader can, it can't set a valid Checksum as part of the process.

QuoteA file with disabled props that is slowly climbing and while not flag as poxed - it is in proto-pox and a ticking time bomb:

You may be confusing cause with effect here. Prop Pox disables props in your save file, but having disabled props is not the cause of prop pox. There are other reasons why these may be disabled, although there isn't a whole load of solid facts surrounding such things. Ultimately, this doesn't tell us anything about what is happening.

But the real question here, how are you getting Prop Pox in these city tiles? That is the area you would benefit most from looking into. Because if Prop Pox is as simple as overriding RKT1 props with RKT4 props, (or similar where the data is larger), then something amongst your files must be the culprit, in the form of an override. If you don't have the Pegasus BDK file, then it's probably going to take some serious looking to find the potential problem file/exemplar. Datanode will help you to find all overrides, then it's just a case of trawling through it's output, to see if there is either an RTK or NitetimeState change amongst the results. But, assuming no such modifications are to be found, that rather supports my theory that there is more to this than initially thought.

The problem is, as of now, no one has ever taken the time to do such a thorough check. Given that all that time staring at exemplars won't bring your cities back, at best it may prevent problems in future. Add to that the general lack of technical ability of most players to interpret the data, it's really not so surprising this was never done. In short, we simply have no hard facts/data to go on. Frankly, the prospect of rummaging through a hideous list of data like that, doesn't exactly motivate me to the task. But in the first instance, if you saved the Override data from your entire plugins folder, maybe it's worth attaching here in case anyone else fancies a peek. That is assuming you aren't motivated to do this your own self?

But lets say someone did trawl through it all, what then? What if there were no instances that connected the dots based on our current understanding? Could we be 100% certain that your plugins folder now, has everything in it, that you ever had installed, since the inception of your affected region? Because if not, right away that data set is simply invalid, because the data is unreliable. The point being, without some sort of absolute reference data, there are simply too many if's, coupled with too little benefits to be had, to justify the amount of work involved. Being me, the only data-set I could trust was one I personally knew the history of. Ironically perhaps, if I found myself in this scenario, I couldn't trust my own set of plugins either, I simply change things too often to keep track of it all. Most of the really knowledgable players probably have the same problem, because learning the ins and outs is intrinsically linked to the sort of player that's modding things a lot.

Perhaps then a better solution is to look at the very first posts from Bap in this thread. Look at how he did his testing to locate the problem file. Replicate those tests and theoretically you should find the file causing your problems. Once we've a valid suspect file to look at, that may provide some meaningful data that moves our understanding forward. But bear in mind again, that any changes, especially removed files from your plugins folder, could prevent you from reaping any results here too.

Kitsune

Large datasets dont scare me so long as there is a CSV output or something I can have excel translate so I can have excel deal with it. However it does seem while Datanode outputs a csv file - it doesnt state the RTK or nightstatevalue. I will give the programs it due - it did handle my 6gb plugin folder like a champ. Also, the only thing that has been removed from my plugins folder are lots, not props. Infact its partial flaw I consider with the way I handle my plugin folder.
~ NAM Team Member

Tarkus

#754
For those who haven't heard yet, simmaster07 of SC4Fix.dll fame cured the Prop Pox yesterday.  All the details (including where to download the fix) are in the link. 

I don't think it's possible to thank him enough for solving not just one but two seemingly unsolvable showstoppers with the game--and 15 years after the game's release, no less.

Edit: Here is the link to the official release of the new, Prop Pox-curing version of SC4Fix.

-Alex

Andreas

Wow, this is just amazing! Fortunately, I never suffered from the prop pox myself, but it has been one of the biggest obstacles in the history of SC4 alright. Surely one of the best birthday presents ever!  :thumbsup: &apls
Andreas

AsimPika3172

I loves Sim City forever!

vortext

Wow wow wow indeed! Almost nine year since the start of this thread, and after 38 pages of explanation & confusion (in equals amounts  :D) it seems we can finally close the prop pox chapter once and for all. What a momentous day!  &apls &apls
time flies like a bird
fruit flies like a banana

fantozzi

Many years I've been playing always in mind it's possible to loose all your work and having to build everything again. Funny thing - I have to learn, these now are silly fears. The fear is still there like a bad habbit (saving every five minutes, making security copies all the time) but the reason has gone.

xxdita

Quote from: vortext on January 21, 2018, 04:58:26 AM
Wow wow wow indeed! Almost nine year since the start of this thread, and after 38 pages of explanation & confusion (in equals amounts  :D) it seems we can finally close the prop pox chapter once and for all. What a momentous day!  &apls &apls

I have good news to add.

The current STEX Admins have taken the bold steps to update the 3 PEG files proven within this thread to be the only known causes of Prop Pox. New players just coming into the world of custom content will never have to worry about seeing the props in their cities totally decimated.

If you have already installed PEG's original files, whether you update them or not is up to you. But either way, I would still strongly suggest continuing to use SC4Fix.