Question for a new packing device

[FrankU]

Dear friends,

A.   As probably all of you, I am very happy with the DATpacker by Wouanagaine.
B.   As probably most of you, I have a very large list of dependency files. But from most of these megapacks
      only parts are used by lots.

Now if I combine these points I come to a question:

Is it possible to make a kind of DAT packer that :
1. Reads all lots,
2. Combines these into large files,
3. Then searches for all needed dependencies (props, buidings, models, dats, textures, etc.) and
4. Copies these dependencies into a new file.

Thus leaving us with a packed folder with only the needed data. So all the models, textures or props that are containes in certain megapacks, but that are not needed by the lots I installed, are left out of the resulting files.
This would really pack my folders and especially the dependecy files!
Probably this will also put an end to all negative comments on dependencies etc.

I do realize that there are some very unclear details concerning loading order etc.... But hey, they are probably  solvable.

[M4346]

I do believe that the latest version of Datpacker has significant changes in terms of correcting the loading order, off topic though.

[JoeST]

you mean dependencies that you dont have? it isnt possible to figure out what an unknown dependency is.

Joe

[Andreas]

I think what FrankU is looking for is a "dependency pruner" that collects all files that are installed, makes a list of the props and textures that have been used, and then deletes (or separates) the props and textures that are not used by the installed lots. This might shrink down the amount of data quite a bit, since nobody will ever use all content that is packaged into the various dependency packs.

[JoeST]

yeah that might be good, except that there will be problems when you get a new plugin that has dependencies that you deleted. a general solution to all these problems would be a central tool that does stuff like the LEX dl'er and the SUM and DatPacker all in one kinda thing, and would compile the super-efficient pack every time but kept the larger, fuller packs for later use. and stuff

JOe

[wouanagaine]

It can be possible to prune off all unused contents ( at least prune props & models ) not used. However that means you can't pack one folder anymore, datpacker will need to pack all your plugins ( which can be now possible, as my 'DVD' subfolder is 3.2gb ) and also need to be aware of all entries in 'simcity_x.dat files' as well as the Maxis\plugins folder

And of course the time to build such mega pack will be one or two of order of mgnitude bigger as it will have to analyse the all examplars

[FrankU]

What Andreas thinks is exactly what I meant.

JoeST: Of course you should keep a "master" directory with all installed plugins. Let's call it "plugins_full". If there is something new to install, you will put it into this folder.
The packing device then would use the content of this folder to extract all necessary data and put these into the folder that is called "plugins" and that we all know as the folder being used by the game.

Wouanagaine: well, I myself don't really care wether it turns out to be one large file or several smaller ones. And I myself think taht packing everything will be the most efficient way. I see no real problems, but of course the player should keep in mind waht she or he is doing.... I just care about gameplay... The absence of care is because I don't know about what problems arise from my proposal... No technical knowledge available on this front.

But the time the program needs to build the files is not very important, in my opinion. It is an action that is not necessary every week or even month. So if it would take 15 minutes... It's the ideal moment to make a cup of coffee.

[wouanagaine]

But the time the program needs to build the files is not very important, in my opinion. It is an action that is not necessary every week or even month. So if it would take 15 minutes... It's the ideal moment to make a cup of coffee.

You need to do that everytime you'll add a plugin

[Andreas]

Well, personally, I always keep a "master" plugins folder that has individual files, and only this will be used for adding new content and such, since it is a lot easier to do updates and repair files. So what the "dependency pruner" would have to do is scanning your entire source folder and make a list of all IDs that have been used on your lots, find all textures, models etc. within the files inside your plugins folder, build the new DATs based on that list and finally remove the unnecessary stuff from the packed DATs. The latter one might be a bit complicated (or at least very time-consuming), since basically every exemplar file and every S3D has to be scanned for the related IDs.

[FrankU]

Yes, indeed,
So, do you think it is possible and useful?

[wouanagaine]

It is possible
It is useful

Now comes the 'who/how/when'  question

[JoeST]

wou: do you have an algorithm for (de-)compression that I could have the (pseudo)code for? I just couldnt work out how to do it from the sims2wiki pages
also if your willing to release the pseudoalgorithms for datpacker i might take a crack at this
Joe

[wouanagaine]

The decompressing/compression code is from Denis Auroux ( you might google for his name )
here is a wrapper for python I wrote
http://sc4terraformer.svn.sourceforge.net/viewvc/sc4terraformer/trunk/modules/qfs.c?view=markup
( just a side note, the one that comes with the new DP fixed a memory allocation error that was in the SC4TF one ), you only need the compress_data/uncompresss_data method

as for DP pseudo code =>

for each selected subfolders sf
  sortedFiles = sortSC4LoadOrder( sf, "*.*" )  ( alphabetical order, files in folder 1st, then subfolder, *.dat are put at the end )
  allEntries = [] ( array or vector )
  for each file f of sortedFiles:
    fentries = get all entries of f without duplicates ( only keep the first one in case of duplicates )
    add all entries of fentries ( remove any same TGI that already exists in allEntries but keep it in the same original place ! )
  create a new file in plugins_compressed with the name of the folder
  save allentries to the new file

This is really the brute force approach, the new version try to be smarter

[JoeST]

thanks, will take a look when i get home

Joe

[FrankU]

Now comes the 'who/how/when'  question

Well, it looks as if you are answering the question yourself.
No need to hurry though...