[Programming] Core Design

[Nique]

I've had contact with tomkeus (he has written a lot of Monte Carlo simulations for solid state physics), and he had a very good idea. (TomKeus, i'm quoting you here if you don't mind)

...As you mentioned Python, let me go back to my idea for separation of simulation engine and graphical engine into two applications. This implementation would use Python as scripting language and command-line interpreter. Simulation engine would be in form of modules for Python and it would be written in C++. This way allows for performance critical parts to be written in C++, and also exposes API through Python, so people can play with the game any way they want. I think, best way would be to give example approximately how simulation would work. You open Python command-line interpreter and type:


In: Import CityMania

In: OutPipe = CityMania.CreateOutputPipe(%file location or client's network adress%)
%This would serve as output pipe to inform graphcis engine of changes so they could be drawn in appropirate manner. This could be intermediate file or client-server connection%

In: London = CityMania.CreateCity(10000000, "C:\TerrainMaps\LondonTerrain.map", ... %other parameters%)
Out: City of London created. Budget: 10000000. Population: 0

In: London.MayorName("Defacto")
Out: ...

In: TransDept = London.GetTransportationDepartment()
Out: ...

In: DaVinciBlvd = TransDept.BuildRoad(RT_STRAIGHT, Tile(i,j), Tile(i,j), LN_SINGLE, TL_NO, "DaVinci Boulevard"...%Other options)
Out: "DaVinci Boulveard" Construction queued. Single Line, No Traffic lights...

In: NTeslaPP = London.BuildPowerPlant(PT_THERMAL_COAL, Tile(i,j), "Nikola Tesla"...%Other options)
Out: "Nikola Tesla" Plant construction queued

In: London.AdvanceOneMonth(%parameters%)
Out: Simulation complete. Commited changes:  "Da Vinci Blvd" Construction started. "Nikola Tesla" Plant construction started... "Da Vinci Blvd" Construction completed 10%. "Nikola Tesla" Plant construction completed 2%. Clock advanced one month.

In: for i in range(1,10)
      London.DoSomething()
      if(London.MakeDecision())
         London.DoOneThing()
      else
         London.DoAnotherThin()
      ...
   etc...



...This is just a crude sketch of how things would work. In this way city simulation is completely indpendent of the way we choose to graphically represent city. This also allows powerful way for people to script addons and basically do whatever they want with the game, including writing their own graphics engine.

There are development benefits also. I beleive writing graphics engine will be much harder job then writing underlying simulation. In this way, simulation could be tested independently without reliance on graphics engine features. Also, different programming languages could be used.
(...)

[croxis]

The way he described it makes perfect sense.  I don't know my programming paradigms very well, but this seems to lend itself to Model-View-Controller very well.

I also want to throw my support between a client-server model.  Being able to build a city with my friends in real time is something I always wanted to do, and SimCity 2000 Network Edition is the only city builder I know of that does this.  Fully implementing it should go on the back burner, but it keeps the possibility more open.

The other thing needing to be planned for is tiles vs freeform coordinates.  I agree with the others in the big thread that using a grid is just more easy, at the same time I really enjoyed making circle cities in the cityxl beta.  Perhaps the underlying simulation uses a free form coordinate system, but a grid is enforced at the ui level?  This way gridless features can be added over time without needing a serious engine rewrite.

[Nique]

The way he described it makes perfect sense.  I don't know my programming paradigms very well, but this seems to lend itself to Model-View-Controller very well.

I also want to throw my support between a client-server model.  Being able to build a city with my friends in real time is something I always wanted to do, and SimCity 2000 Network Edition is the only city builder I know of that does this.  Fully implementing it should go on the back burner, but it keeps the possibility more open.

The other thing needing to be planned for is tiles vs freeform coordinates.  I agree with the others in the big thread that using a grid is just more easy, at the same time I really enjoyed making circle cities in the cityxl beta.  Perhaps the underlying simulation uses a free form coordinate system, but a grid is enforced at the ui level?  This way gridless features can be added over time without needing a serious engine rewrite.

There is a solution for this. He came with that also. He was thinking about rendering the objects realtime, at a specific static angle that depends on the road that it is align. Big buildings need a lot of construction time. We can use the construction time for rendering the building. The object will become a sprite 'after' the player decided to put it over there. So the buildings actualy ARE full 3d models, but are rendered in realtime.

In this case, it doesn't matter on what angle the zones had been placed. The only problem i could think of here is that preview of 'plopable' buildings will be imposible because the game should render it instantly then. By the way, plopable's will still be plopable's by placing the object, but as soon put the building there, there will still be a grow process.

Multiplayer
Your multiplayer idea is fantastic and i think it shouldn't be that hard to implant such feature.

[croxis]

I don't know your experience with creating multiplayer, but my attempt so far is a lot harder than I thought.  It is just something I would like to lay the initial groundwork for.

For rendering buildings, so the building is a 3d mesh with textures, but it is rendered to a 2d texture on a rectangle (sc4 style)?   That is brilliant.  I would assume rendering would be on the order of microseconds, so even direct plop it shouldn't be noticeable.  Loading a saved game will be the part that sees some lag though.  The building under construction graphic might be an issue.  I think we could get around this by having the actual building construction as a 3d object in the scene, but then switch to a sprite when it is done.

[Nique]

I don't know your experience with creating multiplayer, but my attempt so far is a lot harder than I thought.  It is just something I would like to lay the initial groundwork for.

For rendering buildings, so the building is a 3d mesh with textures, but it is rendered to a 2d texture on a rectangle (sc4 style)?   That is brilliant.  I would assume rendering would be on the order of microseconds, so even direct plop it shouldn't be noticeable.  Loading a saved game will be the part that sees some lag though.  The building under construction graphic might be an issue.  I think we could get around this by having the actual building construction as a 3d object in the scene, but then switch to a sprite when it is done.

Yes yes yes!, and so we make full use of 3d (when we place the zone/building), and make it 2d as soon it needs to become static (when the building is placed). About the Networking part, i think we need a new topic for that.

[tomkeus]

TomKeus, i'm quoting you here if you don't mind

Of course not 

would assume rendering would be on the order of microseconds, so even direct plop it shouldn't be noticeable. 

Actually it's not how I though of it. What led me to the idea of in-game rendering was that nothing in the cities happens instantly. Huge building doesn't appear over night in real-life cities. It takes months and years to build, depending on the size. In this way, game could render buildings small piece by small peace, finally assembling full bitmap after, for example, a year of simulation time. Rendering whole buildings at once inside the game in great detail would be hardly possible without disrupting the game performance. Basically in this case, trying to be as realistic as possible is beneficial for the performance.

This strategy would also apply for the roads, railroads, demolition, repairs etc. Building everything will take time. For example, if you want to upgrade some important down-town road, you will have to suffer traffic mayhem resulting from it's closure during the works (which could take months or years, depending on the scale of the construction works).

As for previews of ploppable buildings, small resolution 3D models could be used. Once they are plopped, construction lot/animation is displayed until construction/rendering of the building is completed.

[Nique]

Of course not 

Actually it's not how I though of it. What led me to the idea of in-game rendering was that nothing in the cities happens instantly. Huge building doesn't appear over night in real-life cities. It takes months and years to build, depending on the size. In this way, game could render buildings small piece by small peace, finally assembling full bitmap after, for example, a year of simulation time. Rendering whole buildings at once inside the game in great detail would be hardly possible without disrupting the game performance. Basically in this case, trying to be as realistic as possible is beneficial for the performance.

This strategy would also apply for the roads, railroads, demolition, repairs etc. Building everything will take time. For example, if you want to upgrade some important down-town road, you will have to suffer traffic mayhem resulting from it's closure during the works (which could take months or years, depending on the scale of the construction works).

As for previews of ploppable buildings, small resolution 3D models could be used. Once they are plopped, construction lot/animation is displayed until construction/rendering of the building is completed.

in this case we benefit from 3d and 2d at the same time.

[tomkeus]

I don't know your experience with creating multiplayer, but my attempt so far is a lot harder than I thought.  It is just something I would like to lay the initial groundwork for.

As for multiplayer, the command-line way from the first post directly allows for more than one player/client to mess around the city. If you look carefully, you will see that any operation on the city is queued and implemented after AdvanceOneMonth() method is called. So any number of players/clients can issue commands to simulation engine, the graphics client that finishes updates from last month first sends AdvanceOneMonth command to simulation engine, and then simulation engine waits for all other clients to finish updating graphics. Meanwhile, any issued commands is queued for next month. All this would be very straight forward to implement.

The main problem is player coordination. We don't want one player building highway where another player is building park (though such things can happen in real-life city with poor management).

[Nique]

As for multiplayer, the command-line way from the first post directly allows for more than one player/client to mess around the city. If you look carefully, you will see that any operation on the city is queued and implemented after AdvanceOneMonth() method is called. So any number of players/clients can issue commands to simulation engine, the graphics client that finishes updates from last month first sends AdvanceOneMonth command to simulation engine, and then simulation engine waits for all other clients to finish updating graphics. Meanwhile, any issued commands is queued for next month. All this would be very straight forward to implement.

The main problem is player coordination. We don't want one player building highway where another player is building park (though such things can happen in real-life city with poor management).

Haha, you got a point here. I think that if people play in multiplayer mode they should plan what they are doing. We can make a hierarchy like: mayor and councilors. Or you can reserve space (by zoning an invisible layer) before you build on it. 

[croxis]

There are gameplay mechanics that could help with player coordination, but communication is also part of the fun.

So AdvanceOneMonth(), is this how sc and cxl does it (more obvious in cities xl although some things seem to update mid cycle) and fools us into thinking it is real time?  Could a tick system be used, a single event consumed each tick, with recalculations staggered across each tick (so tick 1, traffic; tick 2, land values... tick 10 crime; tick 11, traffic; etc)?

[Nique]

Hmm,

I hope tomkeus is willing to write a very basic skeleton for this as i have not enough experience with c++.

[tomkeus]

So AdvanceOneMonth(), is this how sc and cxl does it

Have no clue about that.

Could a tick system be used, a single event consumed each tick, with recalculations staggered across each tick (so tick 1, traffic; tick 2, land values... tick 10 crime; tick 11, traffic; etc)?

Let me see if I understand you well. You want to say that engine is running continuous calculations, adding one tick to the clock count after each stage of calculation is completed. Clock count is afterward converted to in-game time.

If this is so, this seems impractical for client-server model I had in mind. Queuing operations and AdvanceOneMonth (any other period could be used, as far as I'm concerned) seemed most logical when command-line interface is used. Graphics client would dictate how AdvanceOneMonth commands would be issued. For example, there could be preset interval of 1min between Advance commands. If graphics client is busy updating graphics around the city (rendering various stuff, implementing changes to city appearance from simulation server) it will wait until everything is done, and then issue new Advance command.

[croxis]

Ok now I see where you were going with this.  yes it makes sense with a command line client, making the simulation turned based.  My point which I failed to articulate was that with the standard city ui I'd like a more organic progression of time.  What I understand of real time engines (I've only made turned based ones) is that it is essentially the same as AdvancedOneMonth(), but a smaller unit of time and in real time.  I will look into it more so I have a better idea of what I am talking about.

[Nique]

Turn-based, but smoothly and automatically, (i prefer  AdvanceOneDay() above a month ) and it needs to be called every 4 game run time seconds?

[tomkeus]

Turn-based, but smoothly and automatically, (i prefer  AdvanceOneDay() above a month ) and it needs to be called every 4 game run time seconds?

Turn-based aspect will not be directly visible to the player. That's the reason there is separate graphics engine. Graphics engine will continuously animate city, construction sites etc. After each AdvanceOneMonth() call, city state update will be performed in background, without disruption of graphics flow. After that is completed, graphics engine will update city stats (population, budget, crime, pollution etc...) and, as for construction sites, it will tell graphics engine to start loop animation of next stage construction works.

If you look carefully SC4 updates stats once per month. Updating stats every day for big cities is not particularly wise if you don't have 10-15CPUs at your disposal. I mean look at the numbers: If the city has 2 000 000 inhabitants, each having 10 parameters, and 20 000 businesses each having 10 parameters, update would mean 2 020 000 of 11x11 matrix multiplications per update, and that is not taking into account traffic calculations, which will, I expect, take up even more CPU time.

So, we have to be reasonable.

[Nique]

Turn-based aspect will not be directly visible to the player. That's the reason there is separate graphics engine. Graphics engine will continuously animate city, construction sites etc. After each AdvanceOneMonth() call, city state update will be performed in background, without disruption of graphics flow. After that is completed, graphics engine will update city stats (population, budget, crime, pollution etc...) and, as for construction sites, it will tell graphics engine to start loop animation of next stage construction works.

I understand

If you look carefully SC4 updates stats once per month. Updating stats every day for big cities is not particularly wise if you don't have 10-15CPUs at your disposal. I mean look at the numbers: If the city has 2 000 000 inhabitants, each having 10 parameters, and 20 000 businesses each having 10 parameters, update would mean 2 020 000 of 11x11 matrix multiplications per update, and that is not taking into account traffic calculations, which will, I expect, take up even more CPU time.

So, we have to be reasonable.

That's why we need to use the GPU for graphics and CPU for simulation. I don't know how many calculations a average cpu can handle.

[croxis]

We also have the advantage of multiple cpu cores for multithreading.  The dependency for each aspect needs to be figured out (landvalue depends on polution, polution depends on traffic, and so on), but a lot of these can be threaded out and calculated in parallel.

[tomkeus]

OK, so I'll go into the little more detail on my reasoning behind the basic architecture and some simulation details.

I want to start from one observation about cities. The larger they are, the more stable they are. That is, when we have city of 5 million people it is extremely unlikely that, for example, entire population looses it's jobs at once, or for example that, suddenly, construction sites pop up all over the city. So any large change is gradual and takes time.

This fact is very useful if one wants to simulate city. When cities are small, they can change really fast, but since they are small, changes are small in absolute number. When cities are large, only small portions of them change so we are again left with small number of changes. To sum up: In every simulation step we don't have to deal with large number of changes.

This is one of the reasons I'm opting to have the GUI separate from the simulator. Because information exchange between GUI and the simulator should be limited in its extent due to the persistence of cities and we could also use other benefits of such division: such as multiplayer, possibility to have GUI and simulator on different computers for extreme city sizes, easier development, openness for modifications, content and functionality addition etc.

Another reason is that communication doesn't necessary have to be in real time. For example, when player lays down road for construction, GUI will set up animation and send information about new road construction to the simulator's queue. As much as player is concerned he sees immediately animation of beginnings of road construction on the screen. When the simulator calculates new time step it will calculate how much road construction has progressed and inform the GUI about that. GUI will then decide on what animation to use to represent construction progress.

Benefits of city inertia apply also to economy simulation. Simulator shouldn't blindly plow through all businesses in the city and perform update of their state. It is unnecessary because we know that within one month (which is time step I think is most reasonable) most businesses will continue to work the same way as previous month. So the simulator should take representative stochastic sample of businesses in the city and update their state. So, after the update is complete the simulator shall communicate to the GUI only the list of the affected businesses and changes that happened to them. Again, we save on the necessary bandwidth.

This also applies to the population related simulation.

When I'm here, let me mention. This is the basis of the Monte Carlo methods. We have some large system and we evolve it by making smartly chosen probabilistic changes to some smartly chosen subset of that system.

As croxis mentioned in the topic on traffic simulation, we could also try going Monte Carlo way for traffic though that could be very far stretch.

Another thing, that I also want to mention is that the GUI and the simulator doesn't have to know about same things. For example, when player builds something, the simulator is not going to check whether the player is building over something that is already built. Rules about that are implemented within the GUI. So for example, as much as simulator is concerned, we could have building and expressway residing on the same tile. Smart GUI will use this to implement bitmap of elevated expressway going over or under buildings. Also, mixed commercial/residential buildings are possible this way.

I hope you can see, that bottom line of this story is:

Make as few assumptions as possible, and keep things simple.

[Nique]

Of course not 

Actually it's not how I though of it. What led me to the idea of in-game rendering was that nothing in the cities happens instantly. Huge building doesn't appear over night in real-life cities. It takes months and years to build, depending on the size. In this way, game could render buildings small piece by small peace, finally assembling full bitmap after, for example, a year of simulation time. Rendering whole buildings at once inside the game in great detail would be hardly possible without disrupting the game performance. Basically in this case, trying to be as realistic as possible is beneficial for the performance.

This strategy would also apply for the roads, railroads, demolition, repairs etc. Building everything will take time. For example, if you want to upgrade some important down-town road, you will have to suffer traffic mayhem resulting from it's closure during the works (which could take months or years, depending on the scale of the construction works).

As for previews of ploppable buildings, small resolution 3D models could be used. Once they are plopped, construction lot/animation is displayed until construction/rendering of the building is completed.

How will these preview look like? Is there quality loss? 

[tomkeus]

How will these preview look like? Is there quality loss? 

Yes. Player will be presented with low poly 3D model with low-res textures. As soon as he plops it, GUI will remove 3D model and put nice construction animation instead. Preview only serves player for placement estimation. Nothing else.