SC4D Homepage
BSC File Exchange
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

 

News:

All SC4D members can freely PM any Staff member regarding issuing Karma Points to a deserving member on their behalf.


Author Topic: SOLVED: Linux Plugin Load Order  (Read 9226 times)

0 Members and 1 Guest are viewing this topic.

Offline Alafel

SOLVED: Linux Plugin Load Order
« on: April 21, 2014, 04:50:39 PM »
Weirdest. Bug. Ever.

What's wrong with this picture? More succinctly, what isn't wrong with this picture?



Let's take a look at this, shall we?
  • The Sidewalks. Okay, I had Jeronij's sidewalk mod installed at one point, but decided to switch to first the sandstone one and then to all white concrete. Jeronij's is still in my plugins folder but should be overridden. Suddenly it's not. I tried removing it and it still shows up, so either it's hidden away somewhere else in there, or my computer's possessed.
  • The NAM Textures. Firstly, notice the missing texture on the Shinkansen wide curve. Secondly, notice the distinct lack of the Canadian RHW textures, which I definitely have installed. Thirdly, notice how the game is displaying both the default and modified El-rail and Monorail models, but neither one completely.
  • Somehow, everything seems to be functionally intact. I inspected the massive RHW interchange I just built on the other side of the map and it isn't broken, apart from having the wrong textures.

Has anybody seen a bug like this before? It showed up very suddenly: I installed some new plugins, (just some buildings, no mods or anything) and restarted the game so they would show up. This was the result.

Momentary background info about my plugins setup: All miscellaneous plugins (such as lots) go into a folder called "To Be Packed". This is because I run SC4 in Linux, and the game freezes on loading if there are too many plugins, so I have to pack them up. The "To Be Packed" folder is where I put the aforementioned new buildings.

I removed the "To Be Packed" folder entirely to see if something in there was causing the problem... no luck. Next I'm going to try searching for recently modified files and folders to see if something managed to land in the wrong place. After that, reinstall the NAM. Figured I'd post this in the meantime, though, to see if anyone has any suggestions.
« Last Edit: May 04, 2014, 09:58:18 AM by Alafel »
You can call me Alex.

Come and explore Calidore: A Steampunk Experiment with me!

Elsewhere on the internet I am known as The Trippmeister.

Sim City 4 Devotion Forums

SOLVED: Linux Plugin Load Order
« on: April 21, 2014, 04:50:39 PM »

Re: Plugins... Half-loading!?

Offline Alafel

Re: Plugins... Half-loading!?
« Reply #1 on: April 21, 2014, 09:05:27 PM »
Update, in case anyone cares:

Sidewalk bug was, as I suspected, a deeply DatPacked file. So that's fixed. But the odd thing is, it shouldn't have been showing up regardless, because it should be overridden by other things farther down in the load order. It's like the game is ignoring load order all of a sudden.

And it didn't fix the NAM texture problems. Those are still broken.
You can call me Alex.

Come and explore Calidore: A Steampunk Experiment with me!

Elsewhere on the internet I am known as The Trippmeister.

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #1 on: April 21, 2014, 09:05:27 PM »

Re: Plugins... Half-loading!?

Online Tarkus

  • *
  • *
  • *
  • *
  • *
  • *
  • Posts: 11229
  • Country: us
  • Total likes: 3876
  • Reputation: 69
  • Gender: Male
  • NAM Team Tankadillo
    • NAM HQ
  • CL: Dr. PuzzlePiece
Re: Plugins... Half-loading!?
« Reply #2 on: April 21, 2014, 09:19:41 PM »
It might be a load order situation.  Linux doesn't use the same system of sorting as Windows, upon which many SC4 plugins are designed to operate.

-Alex

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #2 on: April 21, 2014, 09:19:41 PM »

Re: Plugins... Half-loading!?

Offline memo

  • *
  • *
  • *
  • *
  • Posts: 1765
  • Country: de
  • Total likes: 16
  • Reputation: 34
  • Gender: Male
  • CL: RULin' RULin' RULin'
Re: Plugins... Half-loading!?
« Reply #3 on: April 22, 2014, 12:15:24 AM »
In fact, there isn't any order at all. Your best bet would be to use a different file system. This post by lostwizard contains more information about this problem:

It turns out you are, in fact, mistaken when it comes to there being a "predetermined" order the game uses. The game itself doesn't sort anything at all when it is loading the files in the plugins folder. Datpacker might sort the files before making its dat files, but I'm not using it because I have no need of it and it just adds more complexity when messing with plugins. (Besides, it's recommended not to use it with the NAM.)

It turns out that the file name ordering behaviour being exploited in this case is a feature of the NTFS file system on Windows rather than the SC4D exe itself. Because I'm using Wine on a Linux machine instead of Windows, the underlying file system is not NTFS and, thus, does not have the file name ordering behaviour. Note that I'm referring to the low level operating system stuff here, not the explorer window or what have you. If you look at the low level system calls that are used to read a directory, the order you receive the file names may be any random order, and may not even be related to the order you created the files in a particular directory.

I have been able to conclusively demonstrate that the exe itself does not sort the files. The cul-de-sac textures were missing when I just left the files alone on the file system, just as they stored when they were installed. Then, when I installed a shim file system that sorts directories before returning the list of files to the application, the textures started appearing correctly. That means with absolute certainty that the exe file itself is not sorting the file names. Thus, SC4D itself has no predetermined order in which it loads the files. Instead, the fact that the file naming convention for controlling such loading works at all is an accident of the way Windows implements NTFS. (Obviously, if you've datpacked the files, the order they appear in the dat file would matter instead. Odds are datpacker itself relies on the same feature of NTFS and Windows rather than doing any sorting on its own but since I haven't studied its implementation, I don't know.)

The upshot of all this is that if one is going to use Wine to run SC4D with the NAM or any other order-sensitive addon, one needs to use a file system that sorts the file names in a directory before passing the list to the application.

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #3 on: April 22, 2014, 12:15:24 AM »

Re: Plugins... Half-loading!?

Offline JoeST

  • *
  • ****
  • *
  • Posts: 2646
  • Country: gb
  • Total likes: 9
  • Reputation: 10
  • Gender: Male
  • fbstj
  • CL: Not your average...
Re: Plugins... Half-loading!?
« Reply #4 on: April 22, 2014, 01:29:10 AM »
that is super interesting!
Copperminds and Cuddleswarms

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #4 on: April 22, 2014, 01:29:10 AM »

Re: Plugins... Half-loading!?

Offline Alafel

Re: Plugins... Half-loading!?
« Reply #5 on: April 22, 2014, 08:15:36 AM »
That is really weird, because (a) I've never had this problem before, it began very suddenly; and (b) My SC4 files are on an NTFS drive. I guess Linux ignores file names when loading from NTFS as well?

I'm going to go around touching all my files on the off chance that modification date has anything to do with it (note: in case anyone reading this doesn't speak Linux, I promise that's not creepy).
« Last Edit: April 22, 2014, 08:44:55 AM by Alafel »
You can call me Alex.

Come and explore Calidore: A Steampunk Experiment with me!

Elsewhere on the internet I am known as The Trippmeister.

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #5 on: April 22, 2014, 08:15:36 AM »

Re: Plugins... Half-loading!?

Offline lostwizard

  • *
  • Posts: 9
  • Country: ca
  • Total likes: 0
  • Reputation: 1
  • Gender: Male
  • Theory and practice often differ
Re: Plugins... Half-loading!?
« Reply #6 on: May 03, 2014, 11:40:39 AM »
That is really weird, because (a) I've never had this problem before, it began very suddenly; and (b) My SC4 files are on an NTFS drive. I guess Linux ignores file names when loading from NTFS as well?

I'm going to go around touching all my files on the off chance that modification date has anything to do with it (note: in case anyone reading this doesn't speak Linux, I promise that's not creepy).

It could be an update to Wine or the NTFS driver that started it going.

The underlying operating system calls that return directory entries on Linux do not sort the directory entries. They never did. It is, however, possible that one version or another of the NTFS driver did sort the directories at some point and an update removed that feature. It could also be that the files were accidentally loading in a "close enough" order previously but adding another file or something similar juggled the order. "ls -U" or "ls --sort=none" should show you what order the operating system calls return the files in.

I have a shim file system created using FUSE that sorts directories before returning the file list. I'd be willing to share that with anyone that wants it but it's not in any shape to be released to the general public though it does work. PM me if you're interested.

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #6 on: May 03, 2014, 11:40:39 AM »

Re: Plugins... Half-loading!?

Offline Alafel

Re: Plugins... Half-loading!?
« Reply #7 on: May 04, 2014, 09:57:43 AM »
Well, I've found a solution. Creative Googling turned up the following page:
http://www.linuxforums.org/forum/newbie/111044-change-order-files-directory.html#post636911

This script automatically resorts the files in a directory to match alphanumeric order. Since I run SC4 from my own launch script anyway, I've just copied and pasted it into my script so it'll run every time I start the game.

However, warning:
The script does not sort into the same order returned by ls when there are underscores in the file names. For example, ls will sort the following two file in this order:
Code: [Select]
z___NAM
zzz_load-last
Whereas the script will sort them in this order:
Code: [Select]
zzz_load-last
z___NAM

Keep this in mind when naming files, if you plan on using this script!


It could be an update to Wine or the NTFS driver that started it going.


I don't think that was the case, because the problem started in the space of a few seconds: I'd added some new plugins while the game was running, and when I restarted the game so it would notice them, something was rotten in the state of SimCity! I think it must have been, as you said, a "close enough" order just by luck, and something in the new plugins jiggled it loose.

Thanks for all the help, everyone!
You can call me Alex.

Come and explore Calidore: A Steampunk Experiment with me!

Elsewhere on the internet I am known as The Trippmeister.

Sim City 4 Devotion Forums

Re: Plugins... Half-loading!?
« Reply #7 on: May 04, 2014, 09:57:43 AM »

Re: SOLVED: Linux Plugin Load Order

Offline memo

  • *
  • *
  • *
  • *
  • Posts: 1765
  • Country: de
  • Total likes: 16
  • Reputation: 34
  • Gender: Male
  • CL: RULin' RULin' RULin'
Re: SOLVED: Linux Plugin Load Order
« Reply #8 on: November 13, 2014, 01:58:52 AM »
Frolius666 described another workaround in this topic:

To keep everyone informed, to warn about a potential pitfall, and most of all to shake the tree and see if an occasional apple would fall... here are my findings till now on the "NAM on Linux" issue posted above.
Should such apple happen to fall, this story might be better of in a Linux thread, as this is not really something caused by the NAM.

FYI: I'm running SC4 DE on the current Linux Mint with the current Wine. My plugins folder sits (or sat) on a Ext4 formatted HDD.

I went through the thread Memo gave me, it describes a similar situation with a link to a Linux script at the bottom.
This script however doesn't work on ext4 drives, which is normal because ext4 doesn't keep an alphabetical file list.
I checked with "ls -f" command (which does indeed list files in loading order)
A few hours on the internet thought me that, if something similar to this script exists for ext4; it is very very hard to find.

Alafel told me the script worked on his NTFS drive... soooo
Sometimes you have to make those decisions...  &mmm
Backup... reformat to NTFS... configure...
Put back my plugins... ls -f... still random.
Ran the script and bingo... almost...
Plugins are in alphabetical order, but starting from H to Z, then A to G. Very strange...

Did several tests on that, could get it to start at M too, don't know why...
Then I got the idea of testing with a "maiden" folder structure, born on NTFS.
Made folders A to Z and tried the same as before.
Always perfectly alphabetical A-Z...
So I started renaming those new folders and dragged my plugins in. (I kept the subfolders).
This went OK until I got to M, which all of a sudden wanted to be on top again, script or no script (why not B...)
I accidentally dropped an original "J" in which I corrected but could have triggered it.

At this point I started testing with the very usefull Red-Blue test files from Jondor found here: http://sc4devotion.com/forums/index.php?topic=10542.msg414566#msg414566
This to confirm ls -f really did list in loading order and I wasn't chasing ghosts... confirmed... unfortunately... the red "M" loaded before the blue "A" resulting in a blue square.
The best explanation for this strange sorting behaviour I could think of is that there might be some residual ext4 data in the folders which is recognized by Linux that causes this.

Finally I started back from the maiden A-Z folder structure but kept this and dragged my plugins in the subsequent folder.
This time the folder structure stayed as it was, game loads and runs as it should, roundabouts and all...
Requires a bit more maintenance when using installers, but it's manageable.
Until above mentioned apple appears this is the best working solution I could come up with.

Current conclusion:
I you run the game on Linux, keep your plugins on a NTFS disk and preferably start out this way.

Big thanks to everyone who helped me find the pieces of this puzzle... I hope the above is useful for somebody...

(Someone please sticky this topic.)

Sim City 4 Devotion Forums

Re: SOLVED: Linux Plugin Load Order
« Reply #8 on: November 13, 2014, 01:58:52 AM »

 


If you experience visual glitches in our site, we strongly suggest you to use Firefox to browse the forums. Also a 1024*768 or higher resolution is highly recommended to view this site

Latest News

All SC4D members can freely PM any Staff member regarding issuing Karma Points to a deserving member on their behalf.

SMF 2.0.13 | SMF © 2016, Simple Machines
TinyPortal © 2005-2011
SMFAds for Free Forums

Page created in 0.445 seconds with 72 queries.