Menu

LEX File Exchange
EA Support Files
SC4 Wikipedia
Network Addon Mod
Dependencies
Chat

Author Topic: Large datpacked file torment.  (Read 7764 times)

0 Members and 1 Guest are viewing this topic.

Offline HappyDays

Large datpacked file torment.
« on: October 03, 2012, 05:22:54 PM »
I have 3.2 gigs of plugins. 1.4 gigs of those are various BSC prop packs. Ideally, I compress them all together and get rid of overrides and, simply, to get rid of large numbers of files in favor of a few big ones.

Except SC4 loads significantly slower when I do this. Normally, it takes 55~ seconds to load to the region screen. When I datpack the props, it takes 70~ seconds to load to the region screen.

Why? Does the game have problems loading very large files? I've noticed this problem occurring with files that are "merely" 50 megs or so in size. Inevitably, the larger the file gets the longer it takes for SC4 to load it.

This is really strange, too, when you consider how quickly vanilla SC4 loads: About 5 seconds on my machine. Stranger when you consider that the SimCity_x.dat files are 100+ megs each. Is it possible that the game fundamentally treats its core files and the plugins folder differently?

I'm thinking, in the end, the cause of slower loading is something I just haven't thought about yet. With luck, someone will offer that thought to me and it'll all make sense.

How can I split up large datpacked volumes without manually splitting them up using Reader? Is it possible, via commandlines, to have Datpacker spit out smaller files?
« Last Edit: October 03, 2012, 06:14:52 PM by HappyDays »

Offline dragonshardz

  • 3RR Team
  • Forums Parliamentarian
  • *
  • Posts: 1385
  • Total likes: 1
  • Reputation: 1
  • Feel free to call me Patrick.
Re: Large datpacked file torment.
« Reply #1 on: October 03, 2012, 07:31:36 PM »
If you want to make your datpacked files smaller, what you can do is split the massive folder up into smaller folders.

Offline HappyDays

Re: Large datpacked file torment.
« Reply #2 on: October 03, 2012, 08:04:03 PM »
Well, yes, I can split the 400 meg file remaining after datpacking into smaller files. But, currently, the only way I know how to do this is manually via Reader. Is there an automated way to do this, or a way to configure DatPacker to create files of a specific size?

Offline dragonshardz

  • 3RR Team
  • Forums Parliamentarian
  • *
  • Posts: 1385
  • Total likes: 1
  • Reputation: 1
  • Feel free to call me Patrick.
Re: Large datpacked file torment.
« Reply #3 on: October 04, 2012, 03:36:32 PM »
No, that's not what I meant.

Datpacker works by packing folders into individual unifed files. If you have a folder that is 400 MB of stuff, you can instead have 4 folders each with 100 MB of stuff in them. Then Datpacker will make 4 100 MB files instead of 1 400 MB file.

Offline HappyDays

Re: Large datpacked file torment.
« Reply #4 on: October 05, 2012, 02:17:42 AM »
Ah, right. I get what you're saying.

Unfortunately, I already have the 400 meg file. Or a 50 meg file. Or a file composed of two x size files that result in an x size file. Is there a way to split them apart besides doing it manually via Reader?

Offline SC4BOY

Re: Large datpacked file torment.
« Reply #5 on: October 05, 2012, 02:37:43 AM »
I will tell you right now, you may as well go back and learn to "leave a trail" for DatPacker. Not only is it not an "undoer", it is also not an archiver. Plus there are many functions (simple maintenance items) that you cannot do within a DatPacked file.. the program is intended to work from an archive base location TOWARD the SC4 working directory. You need to rethink what you're doing and figure out how your going to establish a scenerio whereby you can do that.

There are many suggestions and discussions regarding this matter in the support discussion for DatPacker.. I suggest you scan it and decide on a strategy that suits your style and that you go back and recreate your archive file.

You are currently on a path that will only cause you unhappiness. There are many a tale of woe that have gone before you of others who've tried to use the same path.

Offline HappyDays

Re: Large datpacked file torment.
« Reply #6 on: October 05, 2012, 03:16:13 AM »
&mmm

This emoticon seems appropriate. What is Datpacker's purpose if not to take file x and file y and smash them together to make file z, eliminating overrides and theoretically making the game load things faster?

As I recall, the reason Mega Prop packs exist in the first place is because it was found the game loads large packed files better than many small files (Up to a point).

Quote
Plus there are many functions (simple maintenance items) that you cannot do within a DatPacked file..

Such as?

Quote
the program is intended to work from an archive base location TOWARD the SC4 working directory.

I don't understand. Could you clarify?

Offline catty

Re: Large datpacked file torment.
« Reply #7 on: October 05, 2012, 04:52:35 AM »
Quote
the program is intended to work from an archive base location TOWARD the SC4 working directory.

I don't understand. Could you clarify?

I can only give my own setup as an example of how I use the datpacker, I have all the zipped plugins I have ever downloaded in its own folder (6gb in size), I have two folders in my "My Documents" area

SimCity 4 = holds my undatpacked plugins folders and I can move the contents of the folders or add or delete the plugins as I like
SimCity 4 Compressed = is my datpacked plugins area

When you use the Datpacker program you select in the "destination directory" area were its to datpack the plugins in my case that's the "SimCity 4 Compressed" folder.

The shortcut on my desktop is setup to use the "SimCity 4 Compressed" folder when starting the game

"D:\Program Files\Maxis\SimCity 4 Deluxe\Apps\SimCity 4.exe" -UserDir:"E:\SimCity 4 Compressed\"

Using that setup I can do whatever I like in my uncompressed master plugins setup, re datpack to my compressed folder and play the game.

-catty
I meant," said Ipslore bitterly, "what is there in this world that truly makes living worthwhile?" DEATH thought about it. "CATS," he said eventually, "CATS ARE NICE.

Offline vortext

  • Administrator
  • Forums Guru
  • *
  • Posts: 3364
  • Total likes: 575
  • Reputation: 23
  • CL:
    Street Smarts
Re: Large datpacked file torment.
« Reply #8 on: October 05, 2012, 06:00:52 AM »
Quote
Plus there are many functions (simple maintenance items) that you cannot do within a DatPacked file..

Such as?

Well, it's not like it's impossible but it sure is inconvenient when, for example, a proppack gets updated and you need to pull the old proppack from the large datpacked file. Or maybe you don't use that one parkset as often as you initially thought and want to remove it from the plugins. Or maybe you want to reorganize the itemorder in the mayor menu. Or maybe there's this one lot that grows like weed and needs rebalancing. Or. . or. . or. .

Anyway, I've datpacked a few things that won't change very often, like a customized terrain mod, MMPs, and a few dependencies. The latter are mostly BAT sets which consisted of single SC4Models. Other than that I would advise against datpacking your entire plugins folder and if you do, keep an unpacked version at hand like Catty does so you can easily make changes when necessary.
time flies like a bird
fruit flies like a banana

Offline HappyDays

Re: Large datpacked file torment.
« Reply #9 on: October 05, 2012, 04:41:21 PM »
My primary query at this point: Why does SC4 load all 819 or so megs of base game in about 5 seconds, while the other 3.2 gigs takes an additional 50?


Using that setup I can do whatever I like in my uncompressed master plugins setup, re datpack to my compressed folder and play the game.

-catty

I understand now. Yeah, that'd make it easier to backpedal in the event something goes wrong.


Anyway, I've datpacked a few things that won't change very often, like a customized terrain mod, MMPs, and a few dependencies. The latter are mostly BAT sets which consisted of single SC4Models. Other than that I would advise against datpacking your entire plugins folder and if you do, keep an unpacked version at hand like Catty does so you can easily make changes when necessary.

The only things that end up datpacked are files with tons of overwrites, such as ones with the same FSH copied repeatedly. Files unlikely to be updated, such as most prop packs, are also likely to get datpacked at some point.
« Last Edit: October 05, 2012, 04:44:47 PM by HappyDays »

Offline HappyDays

Re: Large datpacked file torment.
« Reply #10 on: August 22, 2014, 01:20:07 AM »
Almost 2 years later it finally makes sense: Instances. While I have not done an extensive experiment to prove it's true, my datpacking of only model files results in titanic sized files and quicker load times versus before where datpacking everything resulted in slower load times. Model files mostly contain just that, files for models and perhaps a prop based upon the model. Altogether they don't count for many instances.

Prop files, however, can run in the neighborhood of tens of thousands of instances BEFORE they're squished together. Having a huge file filled with a hundred thousand or more instances seems to have been the root cause of my issues with loading times.