Menu

LEX File Exchange
EA Support Files
SC4 Wikipedia
Network Addon Mod
Dependencies
Chat

Author Topic: Announcement : SC4DatPacker 2008  (Read 13606 times)

0 Members and 1 Guest are viewing this topic.

Offline wouanagaine

  • NHP Team
  • Forums Guru
  • *
  • Posts: 2854
  • Total likes: 7
  • Reputation: 43
  • CL: :computer: SC4 CodeMaster
Announcement : SC4DatPacker 2008
« on: November 25, 2008, 01:56:38 AM »
Hi folks

A new version of SC4DP has been uploaded to a LEX near you http://www.sc4devotion.com/csxlex/lex_filedesc.php?lotGET=26


What's new
  • This version is up to date with the latest finding in SC4 Loading order, faster, and provide some better compression which turns to speed up your loading time even more.
  • You can now choose the folders you want to pack by selecting them in the mainscreen
    Folder bigger than 392 Mb will be automatically split by SC4Datpacker, and will produce more than one mega pack, so no more need for manual folders splitting :)
  • A new dependancy is needed, so make sure you read the readme.html to get the link

Some results
  • 1.18 Gb of plugins packed in 2 min, loading time to region 35 sec, loading time to a large developped city 1min20
  • 3.46 Gb of plugins packed in 5 min, loading time to region 55 sec, loading time to a blank city 1min15
  • Rough estimation for a 3.6 Gb of plugins, without SC4Datpacker, 5 min loading time, with SC4Datpacker 2 min

This version has been tested intensively ( as you may notice given the letter of the build in the zip ) but of course I can't for sure tell it is 100% bug free, so please if you have some problem running it ( and after you check the readme for dependancies ) please post your problem here and if applicable post as well as the file that SC4Datpacker can't pack


Take care and enjoy fast loading time
« Last Edit: November 25, 2008, 02:15:43 AM by wouanagaine »
New Horizons Productions
Berethor ♦ beskhu3epnm ♦ blade2k5 ♦ dmscopio ♦ dedgren ♦ emilin ♦ Ennedi ♦ Heblem ♦ jplumbley
M4346 ♦ moganite ♦ Papab2000 ♦ Shadow Assassin ♦ Tarkus ♦ wouanagaine
Divide wouanagaine by zero and you will in fact get one...one bad-ass that is - Alek King of SC4

Offline JoeST

  • Forums Guru
  • *
  • Posts: 2649
  • Total likes: 14
  • Reputation: 10
  • fbstj
  • CL: Not your average...
Re: Announcement : SC4DatPacker 2008
« Reply #1 on: November 25, 2008, 03:10:17 AM »
ACE!!! another version. *installs windows just to test it out*  $%Grinno$%

Joe
Copperminds and Cuddleswarms

Offline toxicpiano

Re: Announcement : SC4DatPacker 2008
« Reply #2 on: November 27, 2008, 12:56:43 PM »
is this on the lex disk or do I have to download this?
I find that alcohol, taken in sufficient quantities, can bring about all the effects of drunkenness.

Offline Andreas

  • Administrator
  • Forums Legend
  • *
  • Posts: 4747
  • Total likes: 943
  • Reputation: 33
    • SimCityPlaza
  • CL: ()bobby() Transit Controller
Re: Announcement : SC4DatPacker 2008
« Reply #3 on: November 27, 2008, 12:58:52 PM »
It is available at the LEX (the old download has been updated).
Andreas

Offline toxicpiano

Re: Announcement : SC4DatPacker 2008
« Reply #4 on: November 27, 2008, 01:33:19 PM »
It is available at the LEX (the old download has been updated).

so the lex disc version is outdated i take it?
I find that alcohol, taken in sufficient quantities, can bring about all the effects of drunkenness.

Offline cogeo

  • NAM Team
  • Forums Parliamentarian
  • *
  • Posts: 1163
  • Total likes: 33
  • Reputation: 18
  • CL:
    SC4 Station Master
Re: Announcement : SC4DatPacker 2008
« Reply #5 on: November 27, 2008, 01:34:34 PM »
So there is a new version! Great!

Any thought of implementing the feature I had proposed earlier? I mean making it possible to exclude some folders from DatPacking by putting a marker file in the folder, eg DO_NOT_DATPACK_ME.txt (no contents)? Yes, the user can specify which folder to DatPack and which not, but it would be good to let the developer decide this too, not just the user. I'm talking about RTMT, which has optional plugins, and these may be changed during the lifetime of the RTMT installation. I think it would be impractical to expect the players to remember which plugins should not be DatPacked.

Please consider implementing this feature in the next release or update (I think it's quite reasonable to reckon that this isn't much work).

Offline wouanagaine

  • NHP Team
  • Forums Guru
  • *
  • Posts: 2854
  • Total likes: 7
  • Reputation: 43
  • CL: :computer: SC4 CodeMaster
Re: Announcement : SC4DatPacker 2008
« Reply #6 on: November 27, 2008, 05:10:08 PM »
Cogeo, I think you'll have a hard time explaining me why you don't want a user to pack a folder

I've just played with a 3.6gb all compressed, but the blgprop*.dat, even my working files are compressed, as well as your public RTMT version, without any problems.

If a player doesn't pack all its stuff, then he will have to think about where/what is loaded first, if it is needed to load stuff A in folder X before stuff B in folder Y which was packed in compressed plugins etc...
whereas using everything in the packed form, he just has to push the button and run the game, because if it works with all unpacked files, it will works with all packed files. That doesn't mean that if it works with all unpacked files, it will works with some packed and some unpacked files.

If anything change, like I useally change the pathfinding plugin in NAM depending of the kind of tile I'm playing, I just move in/out the correct file, and just run SC4DP and hit the 'start button. SC4DP only rebuild then NAM folder, that's it, nothing more, nothing to select or care about .



New Horizons Productions
Berethor ♦ beskhu3epnm ♦ blade2k5 ♦ dmscopio ♦ dedgren ♦ emilin ♦ Ennedi ♦ Heblem ♦ jplumbley
M4346 ♦ moganite ♦ Papab2000 ♦ Shadow Assassin ♦ Tarkus ♦ wouanagaine
Divide wouanagaine by zero and you will in fact get one...one bad-ass that is - Alek King of SC4

Offline Pat

  • when life tosses you lemons, make lemonade!
  • 3RR Team
  • Forums Legend
  • *
  • Posts: 8812
  • Total likes: 6
  • Reputation: 45
  • SC4D Podcast, You Know You Wanna Hear IT!!!!
  • CL: ()pathfire() gettin' it right!
Re: Announcement : SC4DatPacker 2008
« Reply #7 on: November 27, 2008, 07:15:26 PM »
Great news for 2008 and also a new update for the anniversary of SC4D.... (Moved original post to the proper place....)
« Last Edit: November 27, 2008, 08:18:17 PM by Pat »

Don't forget the SC4D Podcast is back and live on Saturdays @ 12 noon CST!! -- The Podcast soon to Return Here Linkie

Offline High5Tower

Re: Announcement : SC4DatPacker 2008
« Reply #8 on: November 27, 2008, 08:31:37 PM »
Here is my report about the new version...Its great!
My specs:
Plugins unpacked      3.80GB
Packed                   3.62
Maxis plugins (NAM)    .91.6MB
z folder & root files     .54.6 
Total plugins used     5.082GB
Comuter used: Dell 8400  Pentium 4 3GHz  2GB RAM
All files in both plugins folders are either datapacked or dat files, no model,lot,desc. Except for one
On first run I had 37.7MB of new stuff to pack in my various 21 folders that make up my original plugins.
It took around 5 minutes to do this and gave me a report with about 30 items I needed to remove and did so.
Ran again in about 1.5 minutes.
Settings used : Force repacking...NO: Optimize...Yes:  Include TGI...Yes. Datapacker reported 2.68GB
My average load up time has dropped by 1 minute.
I played for around 2 hours in many cities (18sec between cities) everything seems to work will. :thumbsup:
Thank you for your great contribution to this game. &apls

Offline bob56

Re: Announcement : SC4DatPacker 2008
« Reply #9 on: November 27, 2008, 09:07:21 PM »
This is probably a stupid question, but what is the DatPacker? What does it do?
You can call me Grif

--Currently out of the office, will resume SC4 7/19

Offline High5Tower

Re: Announcement : SC4DatPacker 2008
« Reply #10 on: November 27, 2008, 10:29:10 PM »
To put in simply it converts your basic building file that is usually made up of  a mode lfile, desc file, lot file into a dat file. And computers can read dat files a lot faster. As proof my game runs better now with 5GB of data packed plugins than it did when I only had 550MB of unpacked plugins. Hopefully there will be someone along that can give you a really technical answer.

Offline wouanagaine

  • NHP Team
  • Forums Guru
  • *
  • Posts: 2854
  • Total likes: 7
  • Reputation: 43
  • CL: :computer: SC4 CodeMaster
Re: Announcement : SC4DatPacker 2008
« Reply #11 on: November 28, 2008, 02:32:43 AM »
Good non technical answer, High5Tower  :thumbsup:

Technical answer

Every file that SC4 is reading is made up of 'entries' ( like afile in a zip file ). The job of DatPacker is just to read all your SC4 files, extract all those entries and group them in less files ( originaly one per subfolder in your plugins folder ). So instead of having 10000 files in 25 folders, you'll now have only 25 files.

As each entry is uniquely indentified, DatPacker takes care of getting the correct one when you have duplicates ( this can happen when a file is updated, and you didn't read the readme of that file asking you to remove older stuff for example, or when 2 mods modified the same entry ) SC4 only use one. So instead of SC4 making the correct choice at loading time, it is left to DatPacker, this will of course make a win when loading SC4, because SC4 will only read an entry once

In OS low level world, opening 10000 files is a very long operation, so now SC4 will only open 25 files, this is a big win
Chances are high that your 10000 files will be scattered all over your HD, whereas 25 big files are more likely to be in the same place ( especially if you run a Defragmenter ) so your HD will locate and read them much more quickly

Moreover, it has been proved by users that the zoom bug CTD ( ie zooming in/out very rapidly can leads to a crash to desktop ) is related to the number of files SC4 is trying to read to get the correct graphics, so with 25 files you don't have this bug anymore

New Horizons Productions
Berethor ♦ beskhu3epnm ♦ blade2k5 ♦ dmscopio ♦ dedgren ♦ emilin ♦ Ennedi ♦ Heblem ♦ jplumbley
M4346 ♦ moganite ♦ Papab2000 ♦ Shadow Assassin ♦ Tarkus ♦ wouanagaine
Divide wouanagaine by zero and you will in fact get one...one bad-ass that is - Alek King of SC4

Offline bob56

Re: Announcement : SC4DatPacker 2008
« Reply #12 on: November 28, 2008, 11:22:39 AM »
thats amazing! I'm getting that right now   &apls
You can call me Grif

--Currently out of the office, will resume SC4 7/19

Offline shardis

Re: Announcement : SC4DatPacker 2008
« Reply #13 on: November 28, 2008, 12:41:50 PM »
Hi, how can download this program? the link on top opens a window without possibility to download (1 photo, many comment, nothing more :S )
thx

Offline Swamper77

  • BSC Team
  • Forums Parliamentarian
  • *
  • Posts: 1284
  • Total likes: 2
  • Reputation: 19
  • BSC Grease Monkey
    • Swamper's Twitter
  • CL: ()swamper77() BSC Mechanic
Re: Announcement : SC4DatPacker 2008
« Reply #14 on: November 28, 2008, 12:46:14 PM »
You need to be registered to the BSC LEX before you can download. It is a separate registration from the forums.

-Swamper
You can call me Jan, if you want to.
Pagan and Proud!

Offline cogeo

  • NAM Team
  • Forums Parliamentarian
  • *
  • Posts: 1163
  • Total likes: 33
  • Reputation: 18
  • CL:
    SC4 Station Master
Re: Announcement : SC4DatPacker 2008
« Reply #15 on: December 06, 2008, 01:28:49 PM »
Wouanagaine, why is it so hard to believe that some folders should better be not datpacked?

In many cases the configuration of a pack is determined by the presence or not of certain plugins. And if the configuration can change during the lifetime of the installation, datpacking a folder (and having a single dat file) has two obvious disadvantages: a) the player is not able to tell if a certain plugin is installed or not, and b) removing a plugin that is merged into a common datfile is not possible (the player should delete the plugin and install it anew). In such cases having the option to skip a folder (and its subfolders) would be preferable.

And if you wonder if such plugins exist, NAM is the first that comes to mind. Surely, a DatPacked NAM will work, but then it's impossible to tell wheteher a plugin is installed or not, and it's also impossible to remove, say, turning lanes for roads. Therefore, to my opinion NAM should not be datpacked.

Many of my plugins can also be configured dynamically:
- RTMT props can be installed/uninstalled at will at any time allowing the props to be selected for each individual plop (eg design #1 along the main avenue, #2 at the suburbs and #3 in the industrial area); this is impossible if RTMT is datpacked - I can post pics if you prefer (I will have to reinstall SC4 first) but I think it's not needed.
- Streetlight colours can be changed at any time.
- Fenced GLR pack is actually based on this, the player can choose the black fences for a part of the city, then change the props and choose the red ones.
- Other packs using this feature is the Ped-Mall transit pack and Suburban GLR Stations, not to mention all packs with MML lots, where the player can install/uninstall the MML plugin; using an MML-controlled set of lots is not very practical, so some players may want to remove the MML plugin temporarily, make say, a waterfront using a set of walls, and then reinstall the MML to reduce the length of the menu.

So there are cases where DatPacking could cause inconvenience, and in such cases DatPacking should better be avoided. In some cases I warn in the Description and Readme: "DO NOT DATPACK THIS PLUGIN", but it's questinable how many players will read this, and remember it when they decide to datpack their plugins. It would be much better if this could be specified by the developer (using a text file, as I proposed in my previous post).

I hope I have convinced you.

Offline nerdly_dood

Re: Announcement : SC4DatPacker 2008
« Reply #16 on: December 06, 2008, 03:22:02 PM »
Here is a list of things that I figure shouldn't be datpacked:
- RTMT
- All NAM Team products
- Terrain mods, rock mods and water mods
- Jeronij's and SFBT Street Side Mods
- Anything else that's modular, with interchangeable DAT files corresponding to aspects of the mod

What I think should be datpacked:
- Dependency packs
- BATs and lots that don't have interchangeable DAT files
- Mods that are in a single DAT file
- Mods that you install once and never intend to change

Also, from Wouanagaine's screenshot I get the feeling that this version is Vista-compatible... I'll have to try this!
« Last Edit: December 06, 2008, 03:24:42 PM by nerdly_dood »
My days here are numbered. It's been great and I've had a lot of fun, but I've moved on to bigger and better things.
   EGO  VOBIS  VADELICO  
Glory be unto the modder and unto the fun and unto the city game!

Offline Gaston

  • 3RR Team
  • Forums Senator
  • *
  • Posts: 912
  • Total likes: 1
  • Reputation: 7
  • Cry Havoc! ...
  • CL: Solid Citizen ()monocule()
Re: Announcement : SC4DatPacker 2008
« Reply #17 on: December 06, 2008, 09:19:25 PM »
This is really great.   Thanks again Wou.   

Okay, now for a question.
     There are two "plugin" files.
     My document/ simcity/ plugins      and      Program files/maxis/simcity4/plugins    My question is,  What should go in the Program files/maxis/simcity4/plugins and not in the other?   I have a bunch of stuff in there but I can't remember how or when it got there.    ()what()


---Gaston
白龍

They say that the memory is the second thing that goes....
...dang , I wish I could remember the first.
WooHoo made Councilman - 05 FEB 07 Yipee made Mayor - 13 MAR 07 Hip Hip Hooray made Governor - 04 AUG 07 Rock On made Senator - 15 MAR 09

Offline Andreas

  • Administrator
  • Forums Legend
  • *
  • Posts: 4747
  • Total likes: 943
  • Reputation: 33
    • SimCityPlaza
  • CL: ()bobby() Transit Controller
Re: Announcement : SC4DatPacker 2008
« Reply #18 on: December 07, 2008, 07:13:59 AM »
Most, if not all custom content developers set their installers in a way that their work is installed into "My Documents". The Maxis landmarks are installed in "Program Files". For the game, it doesn't matter where the content is installed, since it reads from both folders - the one in "Program files" first and the one in "My Documents" afterwards. Using both folders can be interesting if there is more than one user account on the computer. Each account has a separate "My Documents" folder, so you can use the "Program files" folder for items that all users need for their regions, and things that are only used by a specific user can go into "My Documents". If you're the only user, it might be easier to use only one of the folders, since this is easier to manage in most cases.
Andreas

Offline Gaston

  • 3RR Team
  • Forums Senator
  • *
  • Posts: 912
  • Total likes: 1
  • Reputation: 7
  • Cry Havoc! ...
  • CL: Solid Citizen ()monocule()
Re: Announcement : SC4DatPacker 2008
« Reply #19 on: December 07, 2008, 09:35:20 AM »
Thanks Andreas.    I have always wondered what the "progam file" plugins were for.    I seem to remember YEARS ago when we all were just starting this custom content stuff that some folks would have you put files in one or the other.   I was looking closer at it and notice most of the stuff is PEG stuff.    Hmmmm?   Thanks Again.   Sorry for the side track in the topic.   But it was sorta related.     ;)


---Gaston
白龍

They say that the memory is the second thing that goes....
...dang , I wish I could remember the first.
WooHoo made Councilman - 05 FEB 07 Yipee made Mayor - 13 MAR 07 Hip Hip Hooray made Governor - 04 AUG 07 Rock On made Senator - 15 MAR 09