• Welcome to SC4 Devotion Forum Archives.

Questions regarding DatPacking

Started by Kolossi, February 17, 2019, 03:36:52 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Kolossi

Been a while since I last played SimCity 4, largely due to CTD and freezing issues that I believe are caused by my badly cluttered Plugins folder. It worked fine, albeit slowly, for a while until I started a downloading spree of buildings, props, textures etc. Then it began crashing and freezing, so I got disgruntled and left it. Today, I decided to I'll at least try to mend the game a bit through datpacking, but I'm afraid I'm somewhat clueless on the matter, so I've come here with a few questions and seeking general advice.

First off, what exactly can and cannot be datpacked? I did a few trial runs, made backups naturally, on a couple of folders in my plugins and I noticed that the Datpacker called models of buildings "useless". That raises an addition to the question: can models be datpacked, and if not, do I keep the models and the "insert-name-here.dat" in the same folder or...?

Second, what do I keep in plugins folder after datpacking? Just the datpacked file or the non-datpacked files, or do I move them elsewhere as a backup, should there be issues.

Third, I read once a while ago that DLL files should be kept in the main game directory plugins folder, instead of regular plugins folder over at Documents. Just want a confirmation to this.

These are the main questions I have right now, I'll add more if any rise up. Thank you in advance.

mgb204

Indeed .dll files should be placed in the Plugins folder of the Installation DIR. This is all to do with Load Order, since there they will load before other content. This is important, since sometimes if the .dll files are loaded in the middle of other mods, they don't initialise properly. I.e., you won't see their effects in-game, a common enough issue with Buggi's ExtraCheats for example.

Similarly, for the average user, everything else should be kept inside the User Plugins DIR, i.e. the one inside Documents\SimCity 4\. Of course if you understand load order well, this is not mandatory, but it will be much simpler for most.

DAT Packing is very similar to the concept of making .zip files, but that analogy doesn't hold completely true. See when you zip a bunch of files, the .zip file contains an index of all the files that were packed together, allowing you to unzip or restore the original files. DATPacking can not do this, it's a one-way process, you can turn many small files into 1 (or more) big files, but once packed, you can't split them back to their component parts.

Part of the reason for this is where the analogy between zip's and dat's falls down. All SC4 files contain data in a very specific format. DATPacking looks at all such files, but does more than simply pack them into less files. It checks the IDs of every item, if it finds duplicates, it will only keep those that load last. It also removes a lot of needless files that are totally useless, but otherwise gum up the works.

So what can be DATPacked?
Well, basically everything, DATPacking doesn't alter how things will work. SC4 will find the same content, no matter how it's stored.
But there are some caveats. If the load order of your Unpacked files are causing undesirable behaviour, that will persist after you pack them. But, so long as you get everything working before you pack it together, that need not be a problem.
It is also generally advised not to DATPack the NAM mod, since this will make updates/changing installed features much harder.

What can not be DATPacked?
Put simply, if a file can't be DATPacked, the application simply will ignore it. Any non-SC4 file, along with .dll files can not be packed. But otherwise, there is no restriction for SC4 mods/downloads. But this is nothing to do with the file types, rather the data format of their contents.

Coming back to the inability to undo the DATPacking operation, the best practise for using DATPacked files is as follows:


  • DAT Pack your desired folders.
  • Move all those folders used above, somewhere outside of the Plugins folder for safe keeping.
  • If you want to modify your plugins, delete the DATPacked files from Plugins. Then move the original unpacked files back to your Plugins folder. Modify/add to them as desired, then return to step 1.

Quote from: Kolossi on February 17, 2019, 03:36:52 AMBeen a while since I last played SimCity 4, largely due to CTD and freezing issues that I believe are caused by my badly cluttered Plugins folder. It worked fine, albeit slowly, for a while until I started a downloading spree of buildings, props, textures etc. Then it began crashing and freezing, so I got disgruntled and left it. Today, I decided to I'll at least try to mend the game a bit through datpacking.

Note that DATPacking is not a magical solution to problems. If there were badly modded files causing instability before packing, those will remain afterwards too.

Some major causes of CTDs, which are mostly avoidable:


  • Failure to limit SC4 to a single core
    Fix this by adding the command:
    -CPUCount:1
    to the end of the Target field of SC4's shortcut properties (or under Launch Options for Steam/Origin users)
  • Placing a puzzle/starter piece over a TE lot. Fixed by using SC4Fix.dll
  • Having non-SC4 files cluttering your plugins folder. So text files, readme files, images, .zip/.exe files and other junk. All of this should be moved elsewhere and not be in Plugins
  • Faulty/Conflicting Mods are a possibility, but harder to diagnose

In any case, it's advisable to sort these issues before you DATPack anything.

QuoteI noticed that the Datpacker called models of buildings "useless". That raises an addition to the question: can models be datpacked, and if not, do I keep the models and the "insert-name-here.dat" in the same folder or...?

Likely it's because the contents of these files are duplicated in other files. For example, you might have a copy of the contents of an .SC4Model file in a MegaPack .DAT file. In such cases, DATPacker will ignore the duplicate, since it's not needed. In short, DATPacking is done on a folder by folder basis. Whatever is placed into the final DAT/DATs it creates, is all you really need. Anything missing, broken or incorrectly overriden, would be identically problematic whether SC4 loaded the unpacked folder or DATPacked file.

Kolossi

Thank you for the reply mgb204, that does clarify a lot. Based on the factors relating to the CTD/freezing issues, I've come to the conlusion that some building is the most likely cause for them, as I have input the CPU Count to 1, I have the SCFix (now in the Installation directory), and I've made sure I've removed all non-plugin stuff from the folder. So a faulty lot/model/desc is the most likely culprit, as apart from NAM, CAM, the SC4Fix and Columbus Tree Controller, I dont use any other mods; the rest are lots, models, props, textures and desc-files, and of course, dat files.

I'm afraid load order is a bit of an unknown to me. I have a hunch on how it's suppsoed to go, but I'm not sure if it's correct. The hunch is that files, without folders, directly behind Plugins are loaded first, after that, it'll go one folder down, load them, and so on until all are loaded. That's how I've understood it, though clarification/confirmation would be appreciated.

About the models, I assume it's safe to remove the said models if DATPacker indicates them as "useless"? I've heard duplication isn't a good thing in regards to game performance.

Finally, I need to pinpoint that lot/model/desc before I can enjoy the benefits of DATPacking to their fullest. Got any tips for hunting down the culprit?


mgb204

Quote from: Kolossi on February 17, 2019, 05:35:26 AM
Thank you for the reply mgb204, that does clarify a lot. Based on the factors relating to the CTD/freezing issues, I've come to the conlusion that some building is the most likely cause for them, as I have input the CPU Count to 1, I have the SCFix (now in the Installation directory), and I've made sure I've removed all non-plugin stuff from the folder. So a faulty lot/model/desc is the most likely culprit, as apart from NAM, CAM, the SC4Fix and Columbus Tree Controller, I dont use any other mods; the rest are lots, models, props, textures and desc-files, and of course, dat files.

Whilst instability could come from something poorly modded, usually those are easier to pinpoint, because the crash is triggered when using/selecting/hovering over the content. If it was so broken a CTD was inevitable, that would happen either whilst or just after loading a city with the content.

Wherever such a problem is coming from, there is a reasonably easy way to find the culprit. Remove roughly half of your plugins folder, then load the game. Has the instability gone away? If no, remove half of what is left and keep repeating this process until everything works once more. Note if using existing cities for these tests, DO NOT save during this process. Not a bad idea to backup your Region before you begin or use a blank test region.

Once you get to the point where the last removed set of files solved the issue, you know where to start hunting for the culprit. Put the working files aside for the meantime, then repeat this process of removing half of what remains.

This might sound like it would take a lot of time, but actually due to the order of magnitude of such things, you should end up with a very small set of suspects without too many steps.

QuoteI'm afraid load order is a bit of an unknown to me. I have a hunch on how it's suppsoed to go, but I'm not sure if it's correct. The hunch is that files, without folders, directly behind Plugins are loaded first, after that, it'll go one folder down, load them, and so on until all are loaded. That's how I've understood it, though clarification/confirmation would be appreciated.

The key thing to remember is that Load Order isn't usually important. You only need worry about it when two mods try to alter the same thing. Regular buildings/lots don't need any special setup, unless they are overriding something that already exists. The easy way to know is to pay attention whilst installing everything, unravelling that later requires either knowledge or lots of time.

QuoteAbout the models, I assume it's safe to remove the said models if DATPacker indicates them as "useless"? I've heard duplication isn't a good thing in regards to game performance.

The beauty of DATPacking is that it will take care of all this for you. Once you switch out the Unpacked originals for the DATPacked files, everything will already be optimised. In that sense, if you aren't completely sure, I'd leave such files well alone. Since other than taking up extra HDD space, if you aren't using the Unpacked files, they are harmless. Of course, if you aren't using the DATPacked ones, it's a different story. DATPacker isn't giving errors here, just a log of things it sees.

Also, SC4 indexes your plugins on startup, if duplicates are found, it takes longer to index the extra data, slowing the loading time. But, crucially, once the game has loaded, it won't make one iota of difference, because if two or more objects have identical IDs, SC4 will always ignore all but the last one.

QuoteFinally, I need to pinpoint that lot/model/desc before I can enjoy the benefits of DATPacking to their fullest. Got any tips for hunting down the culprit?

Datanode can help to find duplicate files, but you'll have to trawl through a lot of data and have a good understanding of SC4 files. Once more, if you DATPack your files, this is all just handled for you.

One tip I would offer, always pay attention in future to any "Cleanitol" files that may come with downloads. Many people don't really understand what this application does, but it's really very simple. Drag the cleanitol file onto the cleanitol shortcut on the desktop and it will run that file. It can either produce a list of missing dependencies, with links to download them. Or, it can remove any files made redundant by the one you are installing. Note, whilst it will move files automatically, they are never deleted, so you can restore them in the event of problems.

Find more info on Cleanitol here, and the application itself here. Not all downloads make use of it, but a lot of the MegaPacks included them to help clear out the old files they replaced.

Kolossi

Again, thank you for the reply. I finished Datpacking and backed-up the un-Datpacked files so all now left is to return to the game proper.

I did notice something strange yesterday when I was Datpacking one particularly large folder, could also be relevant to the crashing issue. The folder contained close to a hundred hidden desktop.ini files. By hidden I mean that when I went to the folders specified by DATPacker, the files weren't there. Furthering the issue was that, before, desktop.ini files were visible, in all previous folders I found them in. I'm assuming the files in question are unneeded clutter and ought to be removed. Im also assuming that, after Datpacking the folder, the desktop.ini files won't be an issue anymore, as the folder now only contains the Datpacked file.

I also discovered, while DatPacking, a folder of a building that would result in errors with Datpacker, a building called Keaton Plaza. These are the errors I got:

********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Users\User\Documents\SimCity 4\Plugins\KP\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x00000010:{"Exemplar Type"}=Uint32:0:{Buildings}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Users\User\Documents\SimCity 4\Plugins\KP\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x27812832:{"Wealth"}=Uint8:0:{High Wealth}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Users\User\Documents\SimCity 4\Plugins\KP\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x27812833:{"Purpose"}=Uint8:0:{Office}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Users\User\Documents\SimCity 4\Plugins\KP\Keaton Plaza\DT_Keaton Plaza Growable.dat
0xAA1DD396:{"OccupantGroups"}=Uint32:6:{Building: Commercial,Building: CO$$$,Style: Chicago,Style: New York,Style: Houston,Style: Euro}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Users\User\Documents\SimCity 4\Plugins\KP\Keaton Plaza\DT_Keaton Plaza Growable.dat
0xAA83558F:{"Crane Hints"}=Uint8:0:{Outer Then Inner Crane}
*************************************************

Any idea what this means? I only ran to this issue with this specific building.

Lastly, I recalled last night that the crashing only occurred in one large city. The game would chug slowly when playing the city and very prone to freezing, the usual triggers being moving the camera quickly and attempting to save, the latter which triggered a freeze 80% of the time, at least. The game could freeze also just by doing regular city things. Guess the city-save file became corrupted at some point, in one form or another. On couple of occasions, the game froze before I even managed to load the city.

Adding to that, the crashes were almost exclusively freezes, CTDs were very rare, happening only once or twice.

Regardless, Im planning on starting a new region altogether. Hopefully I can avoid making another corrupted city.  Thanks for the help, and I hope assistance will find me again, should I ever seek it.

mgb204

Quote from: Kolossi on February 18, 2019, 01:16:51 AM
I did notice something strange yesterday when I was Datpacking one particularly large folder, could also be relevant to the crashing issue. The folder contained close to a hundred hidden desktop.ini files. By hidden I mean that when I went to the folders specified by DATPacker, the files weren't there. Furthering the issue was that, before, desktop.ini files were visible, in all previous folders I found them in. I'm assuming the files in question are unneeded clutter and ought to be removed. Im also assuming that, after Datpacking the folder, the desktop.ini files won't be an issue anymore, as the folder now only contains the Datpacked file.

A Desktop.ini file is a file that determines the way a folder is displayed by Windows, i.e. a system file. Delete them all you will, Windows will just recreate them again. They are not harmful, but of course when you DATPack, such folders will no longer be scanned by SC4 on startup.

QuoteI also discovered, while DatPacking, a folder of a building that would result in errors with Datpacker, a building called Keaton Plaza. These are the errors I got...

A quick check with DataNode and indeed some of the properties/modding are not quite correct. SC4 should compensate by defaulting to pre-set values in these cases. Whether of not that's likely be a serious problem is hard to say for certain. If you removed this particular folder, you could see if that changes anything. I'll see what I can do about fixing those things up for you.

QuoteLastly, I recalled last night that the crashing only occurred in one large city. The game would chug slowly when playing the city and very prone to freezing, the usual triggers being moving the camera quickly and attempting to save, the latter which triggered a freeze 80% of the time, at least. The game could freeze also just by doing regular city things. Guess the city-save file became corrupted at some point, in one form or another. On couple of occasions, the game froze before I even managed to load the city.

Adding to that, the crashes were almost exclusively freezes, CTDs were very rare, happening only once or twice.

Moving the camera/zooming is very typical of the multi-core issue. It's worth checking that is actually working properly. With the game running, open Task Manager (CTRL+SHIFT+ESC), find the SimCity 4.exe Process, then right click and select "Set Affinity". It should show a new Window where only one of the listed CPU # options is selected. If <All Processors> or multiple CPUs are ticked, then something is not working. It's certainly worth double-checking, since it's easy to get the syntax wrong.

Another possibility, if your system is running low on RAM, that can make things very unstable. It could just be the save file, but in my experience you can have a large city full of content, it will be slow, but it doesn't have to lead to instability. Again whilst Task Manager is open with SC4 running, take note of the Physical Memory percentage, shown on the bottom of the window. If this is anything close to 100%, that may explain your problems.

Kolossi

I removed Keaton Plaza for now, but I still have a back up of it, so I can get it back at any time.

I'll check the CPU/RAM stuff when I boot up the troublesome city. I just find the RAM issue hard to believe, since I have 16 Gigs of it, though since the game is close to 20 years old, I assume it cannot fully use all of it. For CPU, I have the CPU Count:1 on Steam, though I wouldnt be surprised if Windows has somehow overridden this and gone with all four cores, but I will check and see if anything's out of place.

I just want to thank you for all the help, SimCity 4 is one of my favorite games and seeing it not working right is really disheartening. Hopefully I can get it fixed this way, would love to return to it full time. Cities: Skylines is great, but it's not SimCity 4. ;D

mgb204

#7
SC4 can use a maximum of 3GB RAM, but yes if you have 16GB in the system, that makes this scenario very unlikely. Note though, in order to allow SC4 brething space, a "4GB Patch" needs to be applied. In essence it's just a flag that tells Windows it can utilise more RAM, but without it performance can be an issue. The NAM installer should patch SC4 for you, but of course if you've re-installed SC4 since, it can get reverted.

I've attached two fixed lots for Keaton Plaza. They replace the original "Grow" and "Plop" DAT files, you just need my two and the original SC4Model file now. Stats are slightly different, but they are in line with Stage 8 CO$$$ stats as calculated by PIM-X. That will resolve any potential issues with the originals. Given it's a 2004 download, we knew a lot less back then and didn't have the tools we have today. So it's always possible such older lots may have issues.

Quote from: Kolossi on February 18, 2019, 05:42:23 AM
I just want to thank you for all the help, SimCity 4 is one of my favorite games and seeing it not working right is really disheartening. Hopefully I can get it fixed this way, would love to return to it full time. Cities: Skylines is great, but it's not SimCity 4. ;D

Couldn't agree more.

Kolossi

I swapped the Keaton Plaza building into the updated one you provided. Unfortunately, it was not the cause of the issues, as, in the troublesome city, freezing and CTDs still occur, either by moving the camera, cursor or attempting to save. I tried to remove a couple of ordnance mods I thought could be the culprit, but their removal did nothing.

So, I believe the save file is somehow damaged, how it happened I do not know. There's no specific building that sets it off that I hover over, though Im not sure if simply having a broken building in the city will cause these issues.

Most likely I'll have to start on the path of splitting my plugins folder and going over them, however, I have noticed that smaller, less developed cities do not cause a crash, just the two large tile cities I have, the troublesome city more so than the other, often enough that progress is impossible; in the other I can still do things, but the crash could occur at any time.

I'll try to find the time to start the hunt for the possible broken mod. I'll post results here when/if I find it, if it isn't simply a broken save file. I assume it's safe to remove textures, props and buildings from the plugins folder and then play the city and it wont cause more instability issues, so long as I don't save?