What causes Prop Pox (and how to avoid it)

[bap]

Hello everybody,

First, thanks a lot for the several positive and encouraging messages. This certainly makes it worth the large amount of time spent in doing careful but boring city developments in seach for the cause of Prop Pox. I underestimated the impact this thread would have in the SC4 community. I was not prepared to drop by and answer questions at the pace needed to handle the rate at which posts are appearing here. Thank you to all of you who have taken this responsability. Thank you RippleJet for the illustrative tutorial on how to check the size of one´s 2977aa47 props subfile (yes, I agree with Barbyw that this file should be better renamed as the props subfile).

CaptCity: I share your feelings about the SC4 community. I appreciate being part of it, and benefit from the dedication & hard work that hundreds of people put towards making this nice game an even better one to play. SC4 would certainly not be so fun if it was not by the great custom content and modds people developed over the years, and by the fan sites where you can see how other players build their cities and where you can share your experience and problems with other players. The nuisance of being not treated respectfully and of seeing your work subject to prejudice and rough irony over one specific site is fortunately not able to change that feeling.

Snorrelli: I am glad to see your here again. I was afraid you had quited playing the game. As you see, your efforts to understand what causes Prop Pox was not in vain. I share the relief with & credits for the findings presented here with you and all those who posted at the original Prop Pox thread.

Barbyw: I appreciate your fast response to the results presented here. It is fine to give new instances to the affected props, as this allows lot makers the chance to edit their lots and replace Maxis prop by the modded one with new instance. But it would also be fine to simply drop the modded props from the dependency file and let the lots that use them appear with the original, unmodded Maxis props. Loosing some time dependent prop behaviour in a few lots seems a fair price to pay to avoid Prop Pox. I am performing other tests concerning Prop Pox and I will be glad to test the other suspected props you already found as soon as I can.

Diggis: As RoppleJet said, if you have an affected prop in your plugins folder but it is not used in any of the lots installed in your city, then you are on the safe side. The problem with those specific props that cause Prop Pox is that they are part of two prop families (A000005E for the beach & patio chair and the recliner, A0000001 for the umbrella) that Maxis used to fill the backyard of almost every low density residential lot. One swallow alone does not make summer, but hundreds of them do. A couple of props with extra bytes floating in the save game file apparently do no harm; but it is a different story when there are hundreds of them. Pegasus beach lots themselves, do no harm. But the infected Maxis low density residential lots do if there are several of them in the city.

Ennedi: there is no such thing as a stupid question, although there may be stupid answers. Thanks for sharing the results of your experiments. It will be great to hear about your new findings.

In his reply to Pegasus´ arguments Callagrafx made an important point. Maxis only gave us the LotEditor, BAT and Plugin Manager. When they developed the game, they expected us to create new lots with existing buildings and props, and later to add our own buildings & props to the game. Thus, it is reasonable to imagine they were careful in the way the information about lots and buildings are stored in the game. This must be flexible & robust to account for things the users would do outside of the game creators control. However, they didn´t created the Reader or SC4Tools. They did not expect us to modd the game, to change their stuff. In particular, there is nothing in LotEditor or Plugin Manager that allow one to change values or add new properties to Maxis props. Thus, it would not be a surprise if the way props are stored in the game are not as protected or flexible as the way lots and buildings might be. Writting props as a fixed-length array in memory is only foolish and poor programming practice if the programmer expects those variables to change in size somehow. More specifically, if they could imagine the same prop (same TGI address) would be read several times while loading the game, and that the prop will have different sizes at different time it is read. Could they imagine we would do that back in 2003 when the program was written? Maxis programmers were not foolish. They simply did not expected us to change their game the way we do. We don´t know what is in the EXE file. We don´t even know how the information is stored in the sc4 game save file (at least not yet). We have been doing modding at our own risk for more than 5 years. It shall not be a surprise that we "touch the wrong button" here or there. Prop Pox seems to be just one of such cases.

Cheers,
Bap

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

[Meastro444]

I am not sure what to do really, one says it's there, someone else says it doesn't. Confusing and disturbing, to me anyways.

I am curious, though not sure if it will work; Is deleting the props subfile, while running the game (and having loaded the city) a 'cure', so to speak? If I interpreted all these posts correctly, the 'pox' is triggered, so once infected it stays. But if there is no file to be infected, could it be cured?

It's an idea, though I am not sure if this will be deleted.

[Andreas]

No, you can't delete the "props" (or "network") subfile, as this would result in a complete loss of all props in your city. PEG claimed that he recognizes the file structure, so if he actually has that knowledge, maybe you can ask him how to proceed with such a task.

[BarbyW]

It is good to see you, bap. I hope you realise that here we are trying to help and find a solution.

The thread at SimPeg is now locked but I am posting here a copy of his final post:



Unfortunately he makes one incorrect assumption as several of his earlier files contained modded Maxis props with original TGIs.
CDK_IND_205.dat - dated Dec 2004
Trail_Park_Engine_305a - dated Oct 2004
These are two that I found with dates from 2004. His Snake Petting Zoo dated Dec 2003 also has modded Maxis props with the original TGIs but that has only been available on his CD for some time now.
I have no idea if these files may cause the Prop Pox but am not happy that the person responsible for irresponsible modding for a number of years is not willing to either come and post his views here or to accept that he may, just may, be wrong.

[snorrelli]

It is good to see you, bap. I hope you realise that here we are trying to help and find a solution.

The thread at SimPeg is now locked but I am posting here a copy of his final post:



Unfortunately he makes one incorrect assumption as several of his earlier files contained modded Maxis props with original TGIs.
CDK_IND_205.dat - dated Dec 2004
Trail_Park_Engine_305a - dated Oct 2004
These are two that I found with dates from 2004. His Snake Petting Zoo dated Dec 2003 also has modded Maxis props with the original TGIs but that has only been available on his CD for some time now.
I have no idea if these files may cause the Prop Pox but am not happy that the person responsible for irresponsible modding for a number of years is not willing to either come and post his views here or to accept that he may, just may, be wrong.

I have them both... This is without mentioning that there may be other, non-PEG, modified Maxis props among custom content available from the STEX at that time.

[RippleJet]

Is deleting the props subfile, while running the game (and having loaded the city) a 'cure', so to speak?

That would actually instantly trigger the prop pox, as Wouanagaine showed when he removed the prop subfile (2977aa47) after it was discovered that the source of the pox was in there:

I removed the 0x2977aa47 of one of my savegame and I've got a lot of prop disapperead ( not networks props but lots ). I still have some props, my first guess is that I still have those that are 'timed' props, which may be saved in an other subfile

as you see, a typical Prop pox pic :

The trees are here because they are flora, and you can spot parked cars that are timed props

btw the subfile 0x2977aa49, seems to be the one where effect props are saved

I wonder if the fact that Wouanagaine still had those timed cars in his city would indicate that they are saved in another subfile, and that they thus would be in the wrong subfile if you modded static props into timed ones...

[Ennedi]

@bap, you made your experiments very correctly from the scientific point of view   and you made an excellent preparation to the next research work. But we must remember that we are in the beginning stage of researching now. We are far from final conclusions.As for your theory about fixed-length arrays in the memory is very probable but not proven yet.

But I'm sure we go in the good direction. We not only want to find an ultimate answer ie. get to know the mechanism which acts here, but we also want to know what to do in the meantime. Looking at everything what was written here I think we should use the method of thinking similar to Pascal's proof of the existence of God  : "We can't prove if He exists or not, but it will be better for us to act as He would exist". And we just do it in this case. Assigning an unique Instance ID number to every game object modified by us will not cause any harm in any case. So it will be safe to take it as a principle in future and to improve existing game objects according to this principle.

I think it can be also a polite and objective answer to PEG http://sc4devotion.com/forums/index.php?topic=7066.0's arguments. I don't say what is true or not, I only say we should make more experiments but we know what is the best for us now.

(...)
"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

Very true!

Bap, one of good methods of testing the theory is trying to falsify it. So don't be surprised if I will try to dio it, it is simply a testing method as I said  . I also suggest others to take another possibilities into account, it can only increase our knowledge, maybe we will get to know another important things too.

Tage and Barby, I suggest the next experiment:
- Put the unmodded toilet into Plugins and build it in your city (that's a really good choice of this prop!  ) as you did earlier, save Hex data;
- Put the modded one into Plugins and build it in your city too (timed toilet - that's really a good joke  ), but don't remove the unmodded one;
- Now open the 2977aa47 subfile, identify/separate data for both props; compare Hex data for unmodded prop for both cases: when you had only this prop and now when hou have them both.

Adam

Edit: At this stage we can't eliminate the possibility that other Prop Pox sources can be collected in other subfiles too. I will try to find another subfiles where particular data are collected.

[bap]

I am computer stupid and know nothing about modding, but I am interested in what's happening here, because I have one infected city, and one I think will get the pox soon. Both are large city tiles with populations over a million. The city that is poxed has a .sc4 file size of 50.8mb, and a 2977aa47 file size of 18.5mb. There is one other large file in that city - 6a0f82b2, which is 24mb. (I have no idea what that means.) The city that I think will fall victim soon has a .sc4 file size of 59.9mb and a 2977aa47 file size of 19.4mb. The 6a0f82b2 file in that one is 30.6mb. There are no other files over 5mb in either city. I hope that info can help. I use downloads from both Simpeg & SC4D.

owlsinger, the 6a0f82b2 subfile stores the underground network. I presume your million hab city has a well developed metro system. This files grows quite fast. In one of the tests I made with a city without getting Prop Pox, I created two simple metro lines running north-south and east-west and the sc4 city save file grew from 27 Mby to 42 Mby after I build my metro lines. All the extra bytes were in the 6a0f82b2 file (16 Mby). It didn't exist before the lines were installed.

By the way, to whom it may concern, the 49c05b9f subfile stores the water pipes in the city.

I have another question - in poxed cities, there is a phenomenon of peeps showing up in a crowd in the upper left corner of the tile. How might this be related? I went into my poxed city last night & observed that they are time dependent - a huge crowd shows up at 8am, and there are only a few in the corner at night. And, well, this really sounds nuts, but if you look at the bottom edge of that corner of the tile, you can see peeps appearing, and some sort of weird shadow. I have pics, if anyone is interested. It sounds crazy, but that's what I saw.

This has also been reported by another player affected by the Prop Pox (I can't remember exactly who).
The upper left corner of a city is that one with x=0, z=0 coordinates. If the infected props cause memory overflow and overwrite the information about the location of the following prop, one may have a prop appearing at the wrong place in game, p.ex., at the tile with x=0,z=0 coordinates. Alternatively (see the last post of RippleJet), the infected prop my destroy the information of its own position (zeroing its coordinates) while evoking a completely different prop (does anybody knows if the prop with TGI 0x6534284A, 0x47BDDF12, 0x05040000 has anything to do with peeps and/or a crowd of people?).

Thanks for posting and reporting your findings. And don't worry. Nobody here will dismiss your report simply because you are not a famous (or "serious") game player - whatever that might be.

And sorry to know you have Prop Pox. I know exactly how disgusting it might be.

Bap

edit: Ok, there is no 0x05040000 prop, and 0x47bddf12 group is for Maxis buildings (thanks RippleJet!)

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

[RippleJet]

1. Which prop is described by this second exemplar with changed Group ID (0x47BDDF12)?

That is actually not even a prop, it's a building exemplar.
GID 0x47BDDF12 comprises all Maxis building exemplars.

Tage and Barby, I suggest the next experiment:
- Put the unmodded toilet into Plugins and build it in your city (that's a really good choice of this prop!  ) as you did earlier, save Hex data;

We actually tested by plopping two lots with the in-game unmodded prop, and actually got the same result as above.

Based on yet another test with a third custom prop having a unique number, I can pretty safely say that if you have two or more instances of the same prop in the city, the Instance ID seems to be split up, so that the highest byte gets an offset position that's increased by two (underlined with red in the picture below)... doesn't really make sense to me though...




We've also tested the city with two timed props and saved the city at different times of the day.
Every time the bottom part of the prop file (after offset 0x50) has contained completely different numbers.

I'm starting to wonder if the prop file contains a quick lookup of all props with their current states (on/off, brightness, shadow, etc.) in order to speed up the rendering process...

- Put the modded one into Plugins and build it in your city too (timed toilet - that's really a good joke  ), but don't remove the unmodded one;
- Now open the 2977aa47 subfile, identify/separate data for both props; compare Hex data for unmodded prop for both cases: when you had only this prop and now when hou have them both.

Back to the testing facilities...

This has also been reported by another player affected by the Prop Pox (I can't remember exactly who).

By Silentbreaker...

And something weird that somehow has to do with the bug: SC4 spawns tons of pedestrians in the northwest corner after the bug appeared:

[Ennedi]

Thanks Tage for the report!

Maybe it will help:
- When you build a new kind of object in your city, two subfiles appears - not only 1 (I saw it in every case I checked till this moment). When you build the first building, not only 2977aa47 appears. You can see also the second new subfile - 2977aa49. It is much smaller, but it also stores some data and grows with placing new buildings.

Next report:
- I built 10 tiles of street in an empty city. Two new subfiles appeared: 896e714a (290B) and c9c05c6e (703B).

Adam

Edit: Something beyond my knowledge, but maybe it should be taken into account if you read Hex data: if the subfile is compressed, data will look different and it can be misleading - am I right?

[jeronij]

This is one of the best brain storming I have seen in a while  

Whatever it comes from it will be an improvement about the game mechanics, and consequently about how we can properly modify and enjoy it. It is a pleasure to me to see this happening at this site, because this was its main purpose when it was created. Thanks to all for your participation  

[Pat]

why in the world is there a constant link showing up when ever someone mentions about Pegauses but uses just the three worded version only? yes I even intentionaly misspelled the name as well to keep the link from showing up to this thread...  sorry just a tad bit confussed when trying read the post and the link keeps poping up it distracts me..

[RippleJet]

- Put the unmodded toilet into Plugins and build it in your city (that's a really good choice of this prop!  ) as you did earlier, save Hex data;
- Put the modded one into Plugins and build it in your city too (timed toilet - that's really a good joke  ), but don't remove the unmodded one;
- Now open the 2977aa47 subfile, identify/separate data for both props; compare Hex data for unmodded prop for both cases: when you had only this prop and now when hou have them both.

In this case, when there is only one instance of both the original prop and the modded prop (with a unique TGI), both records (separated by a yellow line below) in the prop subfile are exactly 88 bytes, and look very much the same (in other words, no size difference between static and timed props). It also appears as if the Instance ID is always doubled if there's only one instance of a prop in the city:

Maybe it will help:
- When you build a new kind of object in your city, two subfiles appears - not only 1 (I saw it in every case I checked till this moment). When you build the first building, not only 2977aa47 appears. You can see also the second new subfile - 2977aa49. It is much smaller, but it also stores some data and grows with placing new buildings.

That's the subfiles containing effects (see the post above by Wouanagaine that I quoted).
It hasn't appeared in our test city as we have no effects around the portable toilets...

[Ennedi]

1. Which effects can be connected with a street (I built only a straight stretch 10 tiles long, no sidewalks, buildings around etc.)?
2. No effects around a toilet? 
Btw I'd like to ask if this timed toilet appears at particular time of the day or randomly? (I'm asking not as the Prop Pox hunter but as a potential toilet user  )

[BarbyW]

It hasn't appeared in our test city as we have no effects around the portable toilets...

Yet

Adam, the toilet appears from 08:00 until 20:00 each day.

[metarvo]

It is pleasing to see the abundant knowledge that has been brought to a potentially chaotic situation.  I haven't had the Prop Pox yet (knock on wood ), but I must applaud the efforts that you have all undertaken to put a stop to it, with a big thanks to bap for coming forward with this.

[bap]

Honestly, I'm starting to worry for my cities now, and this has begun to scare me just a little.  It's one thing to have a temporary glitch like the Water Bug affect a city; at least that is not permanent.  It's another to have something like Prop Pox occur that could permanently ruin a good city.  If the props disappear as a result, are they gone for good?  At least my cities are small enough in file size (despite being large quads) that they won't be affected by this crazy epidemic.  Whatever it is, it sounds positively frightening.  Thankfully, I have enough advance warning to start monitoring my file size before anything bad happens, but I do have and use some of the mentioned content.  Does using it at all automatically set me up for Prop Pox, or should I just not use too much of it?

If you destroy the affected lot and rebuild it, all props are back. Some of the people suffering from the pox tried to do this and attempted to keep going with their infected cities. However this seems pointless. Once the city is infected, more props will disappear everytime you save the city. The next time you enter the city, different and more lots with have missing props. It's an unfair battle. At some point you will loose your entusiasm in replacing a progressively larger number of lots whenever you enter the city. Having the affected props in your plugins is enough to make you "infected" because they are used in most low density residential Maxis lots (those that grow all over the place). It makes no difference if you plop or not the lots of the OWW2-BDK package. They are not the real problem.

Please remember that having the affected props is a necessary although not a sufficient condition in order to develop Prop Pox. Your cities must grow large, the prop/network subfile must go beyond 6Mby size. This is quite hard to achieve (this is why Prop Pox is so rare). If geographical reasons (mountains, water, etc) prevent you from fully occupying a city tile, you will probably be far from the critical 6Mby file size. If you build your city mostly with high density lots, you may also end up in the safe side. What one should avoid is having too many props installed in the city. If you build your city mostly with low density residential zoning, a typical 4x4 tiles block may have, say, eight 1x2 lots. At an average of 6 props per lot, you have about 50 props per city block. Now, if you switch to high density zoning, your block may instead contain a single 4x4 residential building filling most of the space. The block population grows by a factor 10, the numbers of props per block drop by a factor of ten. If this policy is applied to the whole city, you may have it grow until it fills the whole surface while keeping the 2977aa47 props subfile under the 6 Mby limit. You may also leave lots of free space (like natural parks) around your city. Just remember to fill those spaces with God mod or Mayor mod flora and not by lots with trees as those also contribute to increase the number of props in the city.

Bap

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

[RippleJet]

The last test for tonight is a city with four lots;
two having the in-game static portable toilet, TGI 0x6534284A, 0xC977C536, 0x03040000,
and two of them having the modded, timed toilet, TGI 0x6534284A, 0xC977C536, 0x42468D06.

With this setup the prop subfile contains only one record, sized 183 (0xB7) bytes.
Obviously the game saves space by merging the prop records, and omitting at least the Type ID and Group ID for those props that have them in common. In this case only the IID is recorded for the custom, timed prop:



I've checked a few larger cities of mine and Barby's, and they all seem to have one single, large record in the prop subfile.

What one should avoid is having too many props installed in the city.

This is something I've actually been thinking about... the city where I tested CAM did not have many custom lots other than the skyscrapers (which have fewer props), whereas Dublin, the smaller city which had a much larger prop subfile, has mostly low density residential and industrial zones, and with a lot higher percentage of custom content.

[bap]

One thing that I am unsure of, is if the file was in place from the moment the city was started will it be OK?

Nope. One of the first experiments was developing a city from scratch only with Maxis lots, but will all custom content props (including the infected ones) in the plugins folder. It got Prop Pox when 2977aa47 subfile reached 6 Mby. I went back to a version of this same city prior to showing Prop Pox signs, removed every custom content (props files) from the plugins (including the affected ones) and further developed the city. It got Prop Pox again at about the same stage.
One may conclude the problem is not changing the definition of a prop that is used in a city after the city started (this would only affect the new occurrences of that prop), but changing the properties/exemplar size of a given prop while reading files at the start of the game (this occurs every time you start the game with Maxis modded props).

Bap

[RippleJet]

Edit: Something beyond my knowledge, but maybe it should be taken into account if you read Hex data: if the subfile is compressed, data will look different and it can be misleading - am I right?

No, Reader automatically always shows the decompressed content of all compressed files when you're viewing them.