DAMN-NAM v.5 9/09/13

[droric]

See there's this problem.  When importing the menu's with no images attached 4-digit 5-digit make no difference, except the piece is unploppable.

Today I started importing RHW images.  I ran into an issue where importing a RUL file with images results in only a portion of the actual entries due to entries with a 5-digit rul have a 0x00000000 network entry and won't import since there isn't an image associated with them.  Actually I just solved part of my problem   I can drop a placeholder 00000000.png dummy file.  (imports without a dummy file were problematic since its missing 5-digit pieces).

Well anyways it's good to hear that 4-digit RUL's are a possibility for the next NAM.  As long as the ordering of pieces doesn't change much in the RUL file I can account for the eventual renaming by keeping the raw numerically formatted entries.  I am using a script to rename images from the split up RUL files based on the order they currently appear.  New pieces can always be easily added individually down the line as they come in with new revisions.


Exhibit A

[droric]

Actually, I think it is quite easy to automate, if you want to stick with the name of the LText files. Though, if you are planning to change the description text here and there, it is easier to do it by hand.

Anyway, what you have so far looks very promising. I've once attempted it myself, but gave up before the part of creating the icons. I'm definitively looking forward to this. My only concern is that the HIDs in the RUL0 file are not set in stone and might change a bit from release to release. Fortunately though, all the HIDs have been rearranged prior to the NAM 31.1 release, so there won't be any big changes for the time being.

How would one go about doing something like that?  Running into alot of cosmetic pieces that are not named in an easily readable way from the entry in the rul file.

[memo]

Actually, I think it is quite easy to automate, if you want to stick with the name of the LText files. Though, if you are planning to change the description text here and there, it is easier to do it by hand.

Anyway, what you have so far looks very promising. I've once attempted it myself, but gave up before the part of creating the icons. I'm definitively looking forward to this. My only concern is that the HIDs in the RUL0 file are not set in stone and might change a bit from release to release. Fortunately though, all the HIDs have been rearranged prior to the NAM 31.1 release, so there won't be any big changes for the time being.

How would one go about doing something like that?  Running into alot of cosmetic pieces that are not named in an easily readable way from the entry in the rul file.

Well, there wouldn't be any way around scanning the RUL0 file. In each HID section, you would look for the PlaceQueryID. There are several ways to obtain the LText-text from the IID. For one, you could use the Reader to export all the LTexts from the NAM Locale file to your file system and read those files to update your RUL0. Alternatively, you could use CasperVG's [SC4LTEXTool] to export the texts to XML, if it is easier to process.

If you are fond of Java, you could directly load the dat files using one of the Java libraries [one] or [two]. Though, I am biased. For instance, using the latter, you could initialize a DBPF file like this

Code Select Expand


        File inputFile = new File("the path goes here");
        DBPFFile dbpfFile = DBPFFile.Reader.read(inputFile);


and get the text from the placeQueryID, if it is contained in the file:

Code Select Expand


    private final long queryGID = 0x2a592fd1L;
   
    String getText(long queryIID) {
        DBPFTGI tgi = new DBPFTGI(DBPFTGI.LTEXT.getType(), queryGID, queryIID);
        DirectDBPFEntry entry = dbpfFile.getEntry(tgi);
        if (entry != null) {
            DBPFLText ltext = entry.createLTEXT();
            return ltext == null ? null : ltext.getString();
        } else {
            return null;
        }
    }


[droric]

Actually, I think it is quite easy to automate, if you want to stick with the name of the LText files. Though, if you are planning to change the description text here and there, it is easier to do it by hand.

Anyway, what you have so far looks very promising. I've once attempted it myself, but gave up before the part of creating the icons. I'm definitively looking forward to this. My only concern is that the HIDs in the RUL0 file are not set in stone and might change a bit from release to release. Fortunately though, all the HIDs have been rearranged prior to the NAM 31.1 release, so there won't be any big changes for the time being.

How would one go about doing something like that?  Running into alot of cosmetic pieces that are not named in an easily readable way from the entry in the rul file.

Well, there wouldn't be any way around scanning the RUL0 file. In each HID section, you would look for the PlaceQueryID. There are several ways to obtain the LText-text from the IID. For one, you could use the Reader to export all the LTexts from the NAM Locale file to your file system and read those files to update your RUL0. Alternatively, you could use CasperVG's [SC4LTEXTool] to export the texts to XML, if it is easier to process.

If you are fond of Java, you could directly load the dat files using one of the Java libraries [one] or [two]. Though, I am biased. For instance, using the latter, you could initialize a DBPF file like this

Code Select Expand


        File inputFile = new File("the path goes here");
        DBPFFile dbpfFile = DBPFFile.Reader.read(inputFile);


and get the text from the placeQueryID, if it is contained in the file:

Code Select Expand


    private final long queryGID = 0x2a592fd1L;
   
    String getText(long queryIID) {
        DBPFTGI tgi = new DBPFTGI(DBPFTGI.LTEXT.getType(), queryGID, queryIID);
        DirectDBPFEntry entry = dbpfFile.getEntry(tgi);
        if (entry != null) {
            DBPFLText ltext = entry.createLTEXT();
            return ltext == null ? null : ltext.getString();
        } else {
            return null;
        }
    }


Unfortunately I do not know java or really any 'real' programming languages.  I do all my file and text automation using powershell and regex.  Now this isn't really an issue as I have identified a method to query the RUL and find the PlaceQueryID.  The issue is that when using reader to export the ltext files they are in one of two formats.

file00000001.lng - Exported using save selected files (these are in binary and not decoded therefore not usable by a regex query)
file_dec00006327.lng - Exported using save decoded files (these files do not contain the instance ID as far as I can see.)

Any idea how I can identify the exported files?  I may be exporting them incorrectly.

Actually there are LNG.TGI files that are exported as well that contain the necessary information so if there isn't an easy other way to export them I suppose I can rename the LNG file using the TGI files.  One other question is there a reason for the invalid characters that display at the beginning of the LTEXT file (when saving a decoded file).  Is this needed for the description to appear properly or should I just filter and only capture a-z characters?

[memo]

One other question is there a reason for the invalid characters that display at the beginning of the LTEXT file (when saving a decoded file).  Is this needed for the description to appear properly or should I just filter and only capture a-z characters?

The reason for this is that the Reader exports the LText raw byte data. The LText format defines a 4 byte header, the first two bytes of which determine the length of the following text. By chance, the length might represent a valid character, so instead of filtering for a-z, you would just skip the first 4 bytes.

[droric]

Each RUL0 entry should have a line called "PlaceQueryID" referencing the IID of an LTEXT file which is the description displayed by the game. Otherwise, the PlaceQueryID is copied from another HID via the "CopyFrom". The LTEXT file is usually contained in the NAM locale file.

Is this an instance of a "CopyFrom" scenario?  This and a few others are coming up as not found when running my scripts.  I don't see this IID in the NAM_Locale_english.dat file.

[HighwayIntersectionInfo_0x00005FE4]
;Created by MandelSoft on 2013/05/21
;RHW-10S Ramp Type C3 - Exit Ramp
Piece = 0.0, 0.0, 0, 0, 0x5E192605
PreviewEffect = preview_rhw10s_ramp_c3_01

CellLayout =...........
CellLayout =.....AC....
CellLayout =.....XX...<
CellLayout =.....XX....
CellLayout =.....XX....
CellLayout =......X....
CellLayout =......X....
CellLayout =......D....
CellLayout =.....^.....

CheckType = A - dirtroad: 0x02000200 OneWayRoad: 0x03000003, 0xffffffff optional
CheckType = C - dirtroad: 0x02000200 Road: 0x01000001, 0xffffffff optional
CheckType = D - dirtroad: 0x02000200 Road: 0x01030003, 0xffffffff optional
CheckType = X - dirtroad: 0x02000200

ConsLayout =...........
ConsLayout =...........
ConsLayout =.....||...<
ConsLayout =.....||....
ConsLayout =.....||....
ConsLayout =......|....
ConsLayout =......|....
ConsLayout =...........
ConsLayout =.....^.....

AutoTileBase = 0x5E192600
PlaceQueryID = 0x5E192600
Costs = 400


Other than that I have all the relative stuff done, took me like 3 hours but im still learning scripting.  I am hoping to complete the RHW section of the DAMN-NAM by the end of this week/weekend which will mark a large step forward in the project since most all automation aspects are now in place.  For who might be interested here is the powershell code used to relate the file dumped from reader to the RUL entries.  It relies on a HIDS.txt which is the 4-6 digit HID entry from the RUL file (I pulled these manually).


To harvest the names from the ltext files from reader.

Code Select Expand


$IIDs = (Get-Content *.lng.TGI)
$files = (Get-childitem *.lng)
$IIDfile = (Get-childitem *.lng.tgi)
$a = 4
$b = 0


$files | foreach-object {
ren $_ (($iidfile[$b] | get-content)[4] + ".txt")
del $IIDfile[$b]
$a += 5
$b += 1
}


$b = 0
$files = (dir *.txt)
$files | foreach {
$z = ((get-content $_ -raw -encoding unicode)).remove('0','2')
$z | out-file $_
$b += 1
}

Sleep 10




And to relate the HID entries to the names from the previous scripts output.

Code Select Expand



$file = (get-content hids.txt)
$rul = [io.file]::ReadAllText("D:\Desktop\SC4 Stuffs\NAM DAMN Menu Proj\V.1\SplitRUL.rul")
$file | foreach { $_.padleft('8','0') | foreach {

$rul | Select-String "(?smi)\[HighwayIntersectionInfo_0x$_\](.*?)\[HighwayIntersectionInfo_0x\w\w\w\w\w\w\w\w\]" -AllMatches | Foreach {$_.Matches} | Foreach {$_.Value}
}
} | Out-File matcher.txt


$file = (get-content matcher.txt)
$HIDS = ($file | select-string 'PlaceQueryID \= 0x\w\w\w\w\w\w\w\w').matches.value.replace('PlaceQueryID = 0x','')

$HIDS | Foreach {
(Get-Content "D:\Desktop\SC4 Stuffs\NAM DAMN Menu Proj\V.4\IIDs\$_.txt")
} | Out-File Names.txt
Sleep 10


[memo]

Is this an instance of a "CopyFrom" scenario?  This and a few others are coming up as not found when running my scripts.  I don't see this IID in the NAM_Locale_english.dat file.

No, it is not. The PlaceQueryID is 0x5e192600. For whatever reason, it is contained in the "RealHighway_FractionalAngleNetworking.dat" and says "RHW-10S Exit Ramp Style C3". Those LTexts should be moved to the locale files, actually. The simplest thing to get around this for you is to datpack the entire NAM, then sort by GID and IID and copy the English LText files.

[droric]

Is this an instance of a "CopyFrom" scenario?  This and a few others are coming up as not found when running my scripts.  I don't see this IID in the NAM_Locale_english.dat file.

No, it is not. The PlaceQueryID is 0x5e192600. For whatever reason, it is contained in the "RealHighway_FractionalAngleNetworking.dat" and says "RHW-10S Exit Ramp Style C3". Those LTexts should be moved to the locale files, actually. The simplest thing to get around this for you is to datpack the entire NAM, then sort by GID and IID and copy the English LText files.

Okay that part is solved I now have a proper names list for each entry.  I did notice a few entries that show up in game as '## Invalid Intersection Placement ##' or such and I am accounting for them as "Description Missing".  Here is one example.

[HighwayIntersectionInfo_0x00005FC0]
;Added by SA 02/18/2011
;RHW-6S Ramp Type E - Exit Ramp
Piece = 0.0, 0.0, 0, 0, 0x5e151005
PreviewEffect = preview_rhw6_ramptype_e_001


CellLayout =...........
CellLayout =.....Y.....
CellLayout =....+A....<
CellLayout =....+A.....
CellLayout =....+A.....
CellLayout =.....A.....
CellLayout =.....Y.....
CellLayout =.....^.....

CheckType = Y - dirtroad: 0x02000200 Road: 0x01030003, 0xffffffff optional
CheckType = A - dirtroad: 0x02000200

ConsLayout =...........
ConsLayout =...........
ConsLayout =....+|....<
ConsLayout =....+|.....
ConsLayout =....+|.....
ConsLayout =.....|.....
ConsLayout =...........
ConsLayout =.....^.....

AutoTileBase = 0x5e151000
ReplacementIntersection = 0, 0
PlaceQueryID = 0x5e151000
Costs       = 410

Again much thanks to the NAM team for helping me understand all of the intricacies of the NAM structure!




Bonus question - Can anyone thing of a way to fill in the size information for puzzle pieces?


Also noticed that 0x00085066 appears twice in the RUL0 e-series RHD controller.  Is this on purpose?

[droric]

Oh yea... and the RHW section is now 100% finished.   

http://community.simtropolis.com/files/file/28943-damn-nam/

[Yild]

Hello

I was gonna try to implement adding NAM pieces in my DAMN Manager (without any user info about network IIDs and such) but parsing RUL file is more tedious than I anticipated :/

There is no real pattern by which automation can be done without human touch (ATM I didn't find one) :/
There are many inconsistencies like:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; CANAL PUZZLE PIECES ;
; CANAL PUZZLE PIECES ;
; CANAL PUZZLE PIECES ;
; 0x01##
RotationRing=

and there are some entries like:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAIL ADDON MOD (RAM) BUTTON SECTION

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAM Set1 - Single Track Railroad (STR) button
RotationRing=


and now try to parse this

[droric]

Hello

I was gonna try to implement adding NAM pieces in my DAMN Manager (without any user info about network IIDs and such) but parsing RUL file is more tedious than I anticipated :/

There is no real pattern by which automation can be done without human touch (ATM I didn't find one) :/
There are many inconsistencies like:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; CANAL PUZZLE PIECES ;
; CANAL PUZZLE PIECES ;
; CANAL PUZZLE PIECES ;
; 0x01##
RotationRing=

and there are some entries like:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAIL ADDON MOD (RAM) BUTTON SECTION

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAIL ADDON MOD (RAM) BUTTON SECTION
;RAM Set1 - Single Track Railroad (STR) button
RotationRing=


and now try to parse this

I used powershell and regular expressions to parse the RUL files.  I ended up adding section start and end flags to split the rul file up into sections to make it easier to categorize.

[Yild]

ok, I managed to parse this ***, gather ltext entries for corresponding puzzle pieces, an I have found something odd...

why there is no info for entries like 0x0504 -0x0516 (advenced tuleps) and many more (GLR-IN-ROAD bridge puzzle pieces, RAM -FARR-2-3 CROSSINGS BUTTON SECTION
and so on) ?

I searched for any instance of exemplar 0x6a47, AutoTileBase or by Piece value - it seams that those pieces ware 'abandoned'?

any info would be appreciated

[droric]

The descriptions on the lines in the RUL file do not necessarily properly describe the piece.  I used another script to pull all of the LTEXT data for the pieces and import them into my split RUL files to add descriptions.  If the entry in the RUL file starts with a semicolon it means it is commented out and should be ignored.

Are you saying your RUL entry doesn't contain entries for 0x0504-0x0516?  If your using the NAM 32 you may need to reinstall and ensure you install all of the components since with NAM 32 brought a compiled RUL file customized to the users needs.


This is what my Advanced Tuleps section looks like

Code Select Expand


AddTypes = 0517, 10517, 20517, 30517, 40517, 50517, 60517, 70517, 80517, 90517, A0517, B0517, C0517, D0517, E0517, F0517 ;Road Type 210/210 Blank Lane
AddTypes = 0518, 10518, 20518, 30518, 40518, 50518, 60518, 70518, 80518, 90518, A0518, B0518, C0518, D0518, E0518, F0518 ;Road Type 210 Transitions
AddTypes = 0519, 10519, 20519, 30519, 40519, 50519, 60519, 70519, 80519, 90519, A0519, B0519, C0519, D0519, E0519, F0519 ;Road Type 210/Avenue Type 120 +
AddTypes = 0515, 10515, 20515, 30515, 40515, 50515, 60515, 70515, 80515, 90515, A0515, B0515, C0515, D0515, E0515, F0515 ;OWR Base Intersection 1 (4 OWR +)
AddTypes = 0516, 10516, 20516, 30516, 40516, 50516, 60516, 70516, 80516, 90516, A0516, B0516, C0516, D0516, E0516, F0516 ;OWR Base Intersection 1 (2 OWR/2 Road +)


[memo]

ok, I managed to parse this ***, gather ltext entries for corresponding puzzle pieces, an I have found something odd...

why there is no info for entries like 0x0504 -0x0516 (advenced tuleps) and many more (GLR-IN-ROAD bridge puzzle pieces, RAM -FARR-2-3 CROSSINGS BUTTON SECTION
and so on) ?

That's because the file has a long history. There are a few button sections in there that are not currently in use. Some of them have been used before and others have never been publicly released. Nothing to worry about; just ignore the buttons that you've never seen in the game.

[Yild]

sorry to say that but you are explaining basics - my nam is installed with full available options (nam 32), rul0 file is original


after retrieving info for 0x0500 i have (debug info):

Code Select Expand


0500   2     Road Type A-Transition                                Road TuLEP-Type A/Base Network Transition         Road TuLEP-Type A/Base Network Transition (Rotate for T-End)  |  NO_DESC_ITEM_DESCRIPTION_KEY
0501   2  0  Road Type AX                                          Road TuLEP-Type A1                                Road TuLEP-Type A1 (Rotate for A2)  |  NO_DESC_ITEM_DESCRIPTION_KEY
0502   2  1  Road Type AT                                          Road TuLEP-Type A2                                Road TuLEP-Type A2 (Rotate for A1)  |  NO_DESC_ITEM_DESCRIPTION_KEY
0503   2  2  Road Type B-Transition                                Road TuLEP-Type A0                                Road TuLEP-Type A-Blank (Rotate for Dashed)  |  NO_DESC_ITEM_DESCRIPTION_KEY
0504   2  3  Road Type BX                                                                                              | 
...
0516   2 24  OWR Base Intersection 1 (2 OWR/2 Road +)                                                                  | 


second and third columns are my loop iterations counters, fourth - comment from rul0 file, fifth is exemplar name (from piece exemplar entry), sixth and seventh info from ltext files

in this example from 0504 to 0516 info are missing through not finding exemplars for those entries as mentioned in previous post, after checking your DAMN files there is no entries for 0500 branch either (or I couldn't find them in TULEPs folders)

[droric]

I pulled the advanced tuleps from the DAMN-NAM release but can't seem to remember the exact reason why.  I will try to remember to take a look at this later today.

[droric]

Unfortunately I do not have a list of all the sections I pulled as documentation has never been a strong point.  I pulled out certain entries if they were either redundant or did not work.  This was all a manual process where I would complete a menu section then go in game to test everything.  What do you look to implement in the DAMN Manager?  I am not sure if you saw but I have released the DAMN-NAM V1.0 over at simtropolis which contains all the menu items with images and root menus except for Project Symphony, Hole Diggers, Maxis Highways and Canals.

[Yild]

eh...

I was missing hole diggers/risers in Your menus so i was gonna try to pull those info by myself and in addition all available items, atm I can make icons without ingame screenshots... just point me to 3DM file (that code was implemented in my unfinished project but it works just fine)

Your project is manual work ( respect   ) - if anything change in puzzles you will have to modify this manually...
But in other hand if there's a mess like the one that I was pointing at... i don't have any chance to do all your work by code (there's to many unknown/nuances to code and be 100% sure the output is correct).

[droric]

I plan on updating the whole thing once the HIDS have been changed to use all 4 digit identifiers since the 5 digit items will not work the DAMN.  I burned myself out a bit when I did this so I am taking a break on this project until we can get the HIDS values taken care of.  All of the 'placeholder' images you see I already have images for but due to the technical limitations currently with the HIDS I can't use them or even select that piece.  I will probably add in all the missing pieces I excluded from the release when I do update it.

I probably wouldn't have been able to get all of this done without your app so im glad you are getting some use out of the menu's I made.  The SSP Tool just isn't as reliable as the DAMN Manager is and it had issues handling so many pieces.