TerraFormer, Mapper, an i950 chip and some interesting info

[dobdriver]

G'Day Devotee's and Wouanagaine,

Well after having this new computer for about 4 months I have noticed several really good things have occurred. It was upgraded from no slouch either, a quad core 9550.

First up the increase in speed is a given. Although again the old machine was was quite pacy over the previous to that, a 3.4 pentium E.

I have always liked what terraformer and mapper could achieve but by the same token I have always immensely disliked their frailty while performing these tasks. The crashes for no particular reason while importing maps left one staring at the monitor with baited breath and fingers, toes and whatever else crossed wondering if it was going to make through this time. The recognition of them being not particularly adept at multitasking certainly lowered the chances of a crash somewhat but was no sure cure.

Enter the i950, in February I purchased one, a motherboard, 12gig of ram and a copy of W7 64bit with the megalomaniacal intention of busting the 80km barrier for 16bit maps. (I have no idea who else would have been able to use them) But unfortunately while both happily run on the 64bit system, neither take advantage of the extra ram on offer, and so the old limits still apply.

The one real good thing that has eventuated though is, both very rarely crash anymore, I could count them on one hand but the number actually would be only two or maybe three times. Crashing now has become the exception rather than the rule it was previous, I still let them do their thing in peace as I am very much used to that anyhow. Besides the speed increase means a map say in the 50km class takes around 35 seconds to import against the 90-100 seconds for the older quad barring any re-starting, re-loading and re-importing that may have also taken place.

Wouanagaine, perhaps I have wrongly accused your software of being less than exemplarary in the robustness stakes, perhaps it was a hardware problem and only now are we seeing hardware that is truly capable of running your applications as they should. Thankyou.

The one sad thing is having lost the source for these applications, they will never be truly 64bit aware and able to advantage of the extra ram it provides. The last frontier of 16bit mapping, the 160km class may never be seen in SimCity 4. We can only wish. 

Thanks and have a great day.

Cheers
dobdriver

[wouanagaine]

Hey, I never claim it is a monument of stability
It is quite hard to make it works on my own machine, and I have only one, I can't test on multiple configuraiton, nor change my own configuration, hence why people experience problems
I do my best at solving them, but when the problem doesn't show on my machine, I can only guess what is happening, and quite often I can't guess anything

Anyway you've got a good monster PC

Take care
Wouanagaine

[ScottFTL]

That's great news.

I always thought that the video card and driver were more problematic than the CPU, although I do know some folks have had fits with multi-core processors.  I'm curious what video card you are using in that monster rig.  I'm thinking about upgrading soon, but I'm hesitant because SC4 and all the tools are running like a dream now.

[dobdriver]

G'Day Devotees,

Well the meglomaniac inside me got out again and in chasing the elusive monster size map I have discovered a few more things.

1. While both mapper and terraformer are virtues of stability on the i950, their ultimate size is smaller than the preceeding LGA775 chips and no advantage was to be gained as a 64bit machine (in fact the opposite, only smaller maps as either a 32bit or 64bit machine). To make the 80km class maps seen me running off with virus stick in hand doing different bits of the maps on different machines I have set up. This has something to do with the triple channel memory controller of the LGA1366 platform. It has a legacy dual channel mode but it made no difference to mapper or terraformer. They would arrive at the same memory address then quit with the usual note "Do you want to blurt this to Microsoft". Well no thanks.

2. The only solution, build a 64bit machine with native dual channel memory. Out came the trusty old quad to do service once again with 8gigs of memory onboard I made a couple of maps at 88km², but instability reared it's head again and importing was fraught with fingers and toes crossed.

3. With the new LGA1155 platform I decided to build a new puter, ordered the bits along with 16gigs of memory. Yes once and for all I was going to find out exactly how big they could be made.

With terraformer I was a little disappointed as its' size only increased from 68km² on a 32bit machine with 4gigs of memory to 72km² on a 64bit machine and 16gigs of memory. Not much to be had here.

So onto mapper, The improvements while greater than terraformer's was a wee bit less than what I was expecting. (perhaps I was expecting too much). The top size mapper could cope with was 96km². The thing to remember is it is not really 96km but 24 large tiles and about 98.3km. Mapper can import them into and reload them from the region sub-directory ok, but cannot produce a .SC4M file at this size. I haven't yet figured what that size is yet, but rest assured I'll be looking. I may just pop a few 80km+ maps out in the future, these quite obviously would only be suitable for 64bit machines and one could probably get away with only 8gigs of memory.


Some numbers:

The system idles using about 1.5gig of memory, when mapper fires up it jumps to about 2.8gig and when importing uses about 3.2gig. So the size limit does not appear to be a memory issue per se as there would seem to be plenty of spare gigs in the tank. Maybe other issues in the chip architecture cause problems, eg. no more frontside bus or northbridge, onboard memory controller, etc and perhaps mapper and terraformer do not quite know how to deal with this. For now though I think this is pretty well the limit in size and maybe forever.

A 96km map contains 576 large tiles and will take about 3.5 hours to render manually.


Cheers and have a great day.
dobs

[dobdriver]

G'Day Devotees,

While this is an old thread now the info I am adding pertains very much to it and so does not really warrant a new thread.

Okay so for about 10 months now I have been running a 980x with 24GB of ram and find I can create identical sized maps as the 2600k with 16GB of ram. ie Maps with 96km per side, no increase in size for a 50% increase in total ram. This got me thinking that perhaps mapper and terraformer are able to access only one module of ram irrespective of the total amount.

I got the trusty laptop out and installed 8GB of ram in it and fired it up with 64bit win 7, lo and behold I could import all the largest maps including Toronto (The Golden Horseshoe). Then I removed one module leaving only 4GB of memory installed, surprise, surprise I could still import all the large maps.

I fired the laptop up in 32bit win7 with the single 4GB module and could import all the largest maps except Toronto, breaking the 80km barrier on a 32bit machine. The caveat being it must be (a single) 4GB module(s) of ram. 2 x 2GB, 3 x 2GB or 6 x 2GB modules will not cut it. Further tests will determine the absolute size that will open in 32bit windows.

For the time being that just leaves one place to go with maps and that is with the latest 8GB modules recently released for the LGA2011 platform but can also be utilised in the LGA1155 platform. Be watching as I have recently built a new machine just to take advantage of this  ;<)

On the frailty front, the 980x has been absolutely stunning never once crashing , to the point about 4 months ago I decided to try multitasking while importing and exporting maps with both with terraformer and mapper. Never once did it fail and I started to ponder why these newer i3, i5 and i7 chips were so stable whilst the older dual core and core2 duo's and quads were so opposite. I remembered reading some years ago Intel stating that the newer chips would utilise an architecture of a single cpu with multiple cores while the older chips were like multiple cpu's on a single die. Perhaps this is what causes the crash as mapper or terraformer try to slip across the border of the individual cpu's so to speak. This got me thinking about the random CTD's with Simcity also, my older quads suffered these for no apparent reason as well but they have become a glitch in the past on both the 950 and 980x using 4 and 6 cores with multi-threading enabled.

Just to recap: It's not the total amount of ram one has, just ensure it is in 4GB modules to import my current largest maps.


So until the next installment thanks for reading and have a great day.


Cheers
dobs

[WC_EEND]

Very interesting post Dobs, I had no idea how crucial the kind of RAM sticks could be.
Oh, and even the 2600K is a beast (speaking from experience here). If you can get it stable around 5GHz, it will match the performance (more or less) of a 3960X. That's how good it is.

[dobdriver]

No neither did I until I looked at why I was adding more and more ram all the time and not getting the larger map sizes.

Basically for all the maps I and others have produced 4GB is enough.

Oh, and even the 2600K is a beast (speaking from experience here). If you can get it stable around 5GHz, it will match the performance (more or less) of a 3960X.

As for the 2600k at 5GHz, how long does it last?  Being in the tropics and no air conditioning mean air temps here at home are just that, mean. Up to a max of about 42C and the coolest will drop to about 20C for perhaps a week. I don't think my 2600k would last very long.

I shall see soon enough how well a 3960x will run, probably tomorrow. I just need to make a ghost of it then throw on mapper and terraformer.

[cogeo]

It must be understood that the memory available to applications is not the physical memory, but instead the virtual memory, that is the size of the pagefile (some claim that it's the size of pagefile plus the size of virtual memory). Of course, the larger the size of physical memory the less the swapping (between the physical memory and the disk) and thus the better the performance. So the RAM size has to do with performance, while the amount of memory available to applications is determined primarily by the pagefile size. But applications allocate memory in the so called Virtual Address Space. For 32-bit application running under 64-bit Windows, the address space is limited to 2GB or 4GB according to the executable's options. However an application can also allocate memory by employing memory-mapping techniques. It's unknown if the applications of interest (Terraformer etc) make use of memory-mapping, either directly or indirectly (through the DirectX module or the python engine). As for the HW archirecture (channnel, northbridge etc) should not make any difference (at least in theory) except for performance. Also the case with memory sticks sizes, as memory  management is virtual, that is an application can allocate a large memory block as long as there is enough memory available; the memory space does not even need to be (physically) contiguous, eg you can allocate 1.5 GB of memory (literally: address space), and have the 500MB on a contiguous block in chip B, 500MB on the pagefile and another 500MB on multiple page blocks (fragmented) on chip A! The application sees all this as a contiguous memory block, which can be accessed via a pointer, eg p[n] accesses the n-th byte in the virtual block.

Another consideration is the real amount of memory needed. A 80x80 km map is 20x20 large cities, containing 20x20x256x256≈26M vertices, or 52MB of memory, as each vertex is represented as a 2-byte integer (elevation in tenths of a meter), or 104MB if it's represented as a 4-byte real. This is nowhere close to the GBs you mnetion.

I wouldn't bother with overclocking at all, as reliability is the problem here. Even performance might not be benefited as much as you think or at all. By increasing the CPU spead by 20% or so, the computation time may be decreased by 16% or so, however the bottleneck might not be calculations but instead I/O (ie reading/writing, disk swapping and/or memory-mapping), as is the case for the majority of applications today. In addition there is the risk of unreliable operation or even damage.

[WC_EEND]

As long as you have a decent cooler and don't start putting silly volts through your chip (or push the baseclock beyond 103 for sandy bridge CPUs), there is little risk in damaging the CPU unless it was a bad model in the first place. I used the overclocking thing mainly as a metaphor to illustrate the immense power the 2600K actually has though.
I/O bottlenecks can easily be avoided by using an SSD instead of a regular harddrive (trust me on this one, once you've used an SSD, you won't want to use regular harddrives again) or faster RAM/GPU (depending on what kind of bottleneck we're talking about).

[dobdriver]

Xander I think I shall stick with the way I always run, boggy stock. ;<).  SSD's now they are a nice upgrade!!!

Cogeo, unfortunately I am not a programmer, scriptkiddie or coder except for a little dodgy html, but I think I understand what you are saying, just now opening toronto in mapper it is utilising 880,000 MB of memory as read in task manager. I have seen terra using about 1.3GB. but that is pretty much the max. Now the swap space thing is interesting as I do not run this in any usual way but allocate on the basis of 1 for 1. ie this machine has 24GB if ram and 24GB of swap space. (Surely mapper and tf can find some room but no.)

The other interesting thing was when I first fired up the i950 it had 12GB of ram (6x2GB)and 12GB of swap space and could not open as large a map as as my 32bit machines with 2x2GB of ram, it never crashed with a mapper error (ie. traceback etc) but did stop a couple of times on large maps with the generic 'this program has stopped working'. Upon checking the details it had always reached the same address space then 'kaput'. I may have even taken some piccies of the errant info but to find them now amongst the mess that is my filing system would take forever.

Whiles apps should run as you have outlined the truth is they only run as they are written and if there is a bug in the code then so be it. As you would have seen with mapper and terraformer there are enough differences in the code that the maximum size one can open is vastly different. 18x18 tiles versus 24x24 tiles. It is a shame Wouanagaine no longer has the source because as I mentioned previously the robustness is vastly improved with the lastest architecture and fixing the memory issues would be great. Roll on my 8GB machine (Almost ready ;<))

Now here's a thought, even though I presume Wouanagaine wrote it on a dual core/core 2 machine? It wasn't so suited, but the (i) model chips have no probs with them. Perhaps and this is a big perhaps the memory handling is suited to quad channel. Just a thought. I'll see soon enough ;<).
Perhaps he wrote it on an older pentium single core machine and that is why it has problems with the DC/C2 machines as it sees two or four physical CPU's. and the newer architecture it only sees one irrespective of the number of cores. There is always two sides. ;<)

Cheers

[dobdriver]

G'Day Devotees,


Well what a pizzling fizzer the 8GB sticks of ram turned out to be for terraformer and mapper. Photoshop loved wriggling around in 64GB of ram and was noticably faster in long drawn out tasks like saving a 10000px by 10000px image as a greyscale 16bit png. I decided to rejoin Kerguelen as the test and immediately terra refused to open the two largest maps, both are 68 by 72 and were originally made in terraformer. Not a good start, it managed the third map at 52 by 76 though. Well that was an epic fail. I used mapper then to break out the other two greyscales and joined all three in photoshop for a final map of 140 by 128. Here goes nothing I thought and that's exactly what I ended up with! Nothing.

I proceeded to cut the map back to 120 a side and had another go, still nothing. Another quick chop to 112 a side... and nothing, not even 3 extra blooming tiles per side. Back to the axe it went and down to 96 per side, the same as Toronto. Mapper still refused to load it. Bugger that's it I said and disconnected the 3960X, pulled the data drives and reloaded the 980X. It then proceeded to load the 96km map in about 25 seconds, thankyou very little!

So that's it, the biggest map possible is 98.304km per side or 24 large tiles and can be made with 1 single 4GB stick of ram and a 64bit version of windows. 32bit windows with a single 4GB stick is not much less, it is definately 22 tiles per side as I can open my San Francisco map but not 24. Although with a 256MB video card rather than 1GB it may just be possible.

And so ended the meglomaniacal dreams of a really huge map...
I've already made it. Oh well, que sera sera.


Cheers
dobs

[Kitsune]

it sounds like the key limitation is the program is only 32 bit... if a 64 bit version is/if made then it could use all the ram you want.