SC4 Devotion Forum Archives

SimCity 4 General Discussion and Tutorials => SimCity 4 General Discussion => General Custom Content Discussion => Topic started by: eggman121 on December 29, 2016, 06:01:10 PM

Title: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on December 29, 2016, 06:01:10 PM
Hello everyone.

I am looking at ways to preserve the artistic talent of users and I believe that using free and open source programs such as Blender and other helper tools like FreeCAD (An open source CAD program) will help preserve the content creation capabilities for many years to come.

This is a general discussion and any valid input is welcome  :)

So my first question to anyone who knows, Is there a way too morph of extrude a texture onto a plain in Blender?

I have successfully been able to make some curves in FreeCAD and export them via dae collada and export them into blender.

But I have no clue of how to apply textures for starters and morph the textures along the curve.




So This is just the tip of the iceberg. If we could find a way too make a rollout for rendering textures from Blender onto a BAT than we could avoid the ever changing dynamic of 3ds max. This also means that users using different OS like Linux and Mac OS can make content to varying degrees as well.




So. This is an open discussion. Valid points and views are welcome.

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: matias93 on December 29, 2016, 06:11:58 PM
Well, I have no experience on 3D modelling outside of Sketchup, which is also proprietary software (and where extruding is very easy), but I guess that if we want to know more about blender, we should recur to the Cities Skilines modding community, where that application is more widely used.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tigerbuilder on January 21, 2017, 10:10:54 AM
I'm using blender to make low poly models for automata. It's a powerful program. I can use it because all I need is the S3D table data which I import using the reader to bring my model to the SC4 world. The texture UV data is transferred also during this import. I map it in Blender very easily. Research (YouTube) "Blender UV unwrapping". It's not so hard when you get it.

Of course we could model in Blender, but I'm afraid BAT and BAT4MAX is all we get to actually get our models into the game. Sure, if someone was to make a plugin that could do it, but that's unlikely. Surely Blender can make the 20 shots needed, but it won't be in SC4 format like BAT does. Converting those into a model file? That is something above my pay grade.

I find that using gmax/3dsmax is just easier to get to where you're going making something.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 21, 2017, 04:53:05 PM
I did briefly look at using Blender once. To be totally frank, I hated the interface and just found it totally alien. But that's more a reflection of having learned to use gMax. Rather than learn it all again, it made far more sense for me to move up to 3DS Max for my batting. Of course, that's not really helpful, I'm sure others will feel quite differently too...

The real problem with 3DS max is that everyone has gone along with Autodesk's will, being forced into a SaaS type of arrangement. Mostly because everyone uses a student licence, which always has an expiry date on it. As that software moves towards the cloud, along with other changes, compatibility with BAT4MAX is becoming a serious problem. So much so, I don't think you can use BAT4MAX with the current releases at all. Having no-one with the abilities of SimFox means it's been left without updates since his departure.

Whilst it would be nice to see someone update the scripts to work with newer editions of 3DS Max, that's a stop-gap solution at best. It's clear from the progression of that software that the problem will only come up again later on.

So what we need is an absolutely stable platform, in the sense that we can continue using an older edition for the purposes of compatibility. As such, Blender is probably the best candidate for this, being open source we're free to even fork the code especially for the SC4 community if so desired.

What we really need more than anything else is a plugin/script that can render an object created in Blender into the 20 views used by SC4 Models. There shouldn't be any technical barrier to doing this, it simply requires someone with the knowledge to do it and the will to make it happen.

It's probably useful to explain how BAT4MAX works a little. Whilst all the bells and whistles of the script are handy, they aren't really essential. Everything except the initial rendering process can be handled by SC4BAT/gMax, much like it is now.


The real issue is finding someone with knowledge of the inner workings of Blender. Someone who could write a script/plugin that would enable the application to render the images in the format required. Ideally that would include some default lighting rigs and scenes, suitable for making bats. It was suggested (over on ST I think), that we could maybe look for a third party who would be able to handle this on our behalf. That may mean paying someone outside of the community as a one off commission. Assuming no one with the knowledge is around to handle it for us. But, I think it would be possible to hire a non-professional, for what is a reasonably small amount of work, without it costing huge sums of money. That may mean crowd sourcing the cost of this one-time service to achieve, but I think that's possibly the best solution here. Since having done this once, we'd have a stable and free platform for BATs for the future of SC4.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: matias93 on January 21, 2017, 06:20:46 PM
You made that sound even easy!


In any case, it seems that Blender is fairly transparent on its scripting, and uses --what else-- Python to automate things (https://www.blender.org/manual/es/advanced/scripting/introduction.html).


As I said previously, I guess someone on the C:S community, where Blender is more widely used, could be of help to learn what is needed to automate on the modeling side, and the exporting side would be the only part requiring full, novel work.


Considering also the abundance of scripts for Blender (https://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts), and its general open source nature, it wouldn't be strange to find something useful to fork, or someone available to give a hand (or even to take charge of the project). And, as you said, this would only require to be done once, so is a no-strings-attached situation for any developer that could help.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on January 28, 2017, 08:26:11 AM
Quote from: matias93 on January 21, 2017, 06:20:46 PM
Considering also the abundance of scripts for Blender (https://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts), and its general open source nature, it wouldn't be strange to find something useful to fork,

I installed Blender and spent some time poking around, I know nothing about using it, but I noticed there is a very long list of import / export scripts. are none of those, or a combination, useful?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 28, 2017, 11:00:18 AM
Quote from: Handyman on January 28, 2017, 08:26:11 AM
I installed Blender and spent some time poking around, I know nothing about using it, but I noticed there is a very long list of import / export scripts. are none of those, or a combination, useful?

I wouldn't have thought so, simply because modelling in SC4 is unique (compared to modern 3D games). We actually apply 2D models to a 3D LOD, but each Rotation/Zoom combination has it's own static texture. With the exception of certain 3D objects, most items can only display correctly based on the isometric viewpoint of SC4. Even if scripts exist for other 2.5D isometric games, it's doubtful they will conform to the standards used in SC4. I'll try to do some digging, but alas I fear such a task is beyond me. I poked around the 3DS Max scripts and I just don't know enough about the modelling side of things to make any progress on them, even if the logic is mostly understandable.

However, a brief search on Google does show that similar things have been attempted (https://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Add_Mesh/Create_IsoCam), more here (http://www.gamefromscratch.com/post/2015/11/20/Creating-Isometric-Tiles-in-Blender.aspx). So there may be third party plugins we can make use of.

What I'm not sure of is whether all Isometric games require the same formatting. But maybe there is something to start looking into. I'll try to see if I can make something out of this.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on January 28, 2017, 04:45:48 PM
I checked out your links, and while I don't understand all the processes, it seems there are similar concepts at play. Hopefully we can find some helpful information.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 28, 2017, 06:08:40 PM
I dug in a bit further and as previously mentioned, I don't see any technical barrier to making Blender do what we want.

That said, I got the impression that the linked information (and everything else I came across) was aimed at sprite-based modelling. Where you model each individual 2D tile, which isn't what we're after at all.

The key barriers here are working out the following:


Once we know this information, it should be possible to render the images we need I think. But bear in mind, that's 20 camera angles that need switching between for a complete render. If we can get a successful render, then we can look at scripts to automate the process and automatically setup the scene for end users.

And here once again we hit a bit of a roadblock, because I've not got the slightest clue what any of those settings might be. I'll dig around through 3DS Max and see if the BAT4Max scripts shed any light on this. Likewise, somewhere in the forum archives, SimFox (or ALN / ChrisAdams) may have documented these settings. It could also be somewhere in the SC4BAT scripts too. But these will need to precise, it's not going to work well to just use trial and error here.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on January 29, 2017, 05:56:19 AM
Quote from: mgb204 on January 28, 2017, 06:08:40 PM
I dug in a bit further and as previously mentioned, I don't see any technical barrier to making Blender do what we want.

That said, I got the impression that the linked information (and everything else I came across) was aimed at sprite-based modelling. Where you model each individual 2D tile, which isn't what we're after at all.

The key barriers here are working out the following:


  • Where to place the "cameras" so that the render appears correctly.
  • The specific angle of that camera in relation to SC4s isometric view.
  • There will no doubt be a need to set up the lighting too. I should state, at least in the first instance, it's unlikely we'd have DarkNite renders with this method. Of course that's just a specialised lighting rig for 3DS Max. But the renders from Blender will not necessarily be identical to what we have now.

Once we know this information, it should be possible to render the images we need I think. But bear in mind, that's 20 camera angles that need switching between for a complete render. If we can get a successful render, then we can look at scripts to automate the process and automatically setup the scene for end users.

And here once again we hit a bit of a roadblock, because I've not got the slightest clue what any of those settings might be. I'll dig around through 3DS Max and see if the BAT4Max scripts shed any light on this. Likewise, somewhere in the forum archives, SimFox (or ALN / ChrisAdams) may have documented these settings. It could also be somewhere in the SC4BAT scripts too. But these will need to precise, it's not going to work well to just use trial and error here.

This looks like a positive step mgb204. You know a lot more about this stuff than I do, so I am encouraged by your response.

When I was scanning through a tutorial, I remember seeing a section that described working with single tiles or a group of tiles. ( don't know if this is what you are referring to above ).
I also spent some time with the program and tried to familiarize myself with some of the tools (so many tools &mmm). I was able to add and move cameras, add and move lighting, I was able to import a S3D file and manipulate it. Don't know if any of this helps, just kinda spit balling.
I'd be happy to peruse some of the scripts if you like, just point me in the right direction.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: APSMS on January 29, 2017, 11:48:22 AM
Just for clarity, I was under the impression that SC4 uses a trimetric projection. Previous 3d SimCity titles used isometric perspectives.

This is what causes the foreshortening and weird perspective angles when trying to properly model standard shapes in all perspectives.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 29, 2017, 12:03:44 PM
hm, iirc simcity actually uses it's own unique axonometric projection, which is not proper isometric nor trimetric. Like, there're slight differences in projection for each zoom level, and some weirdness in the height scale.

That said, I've been following this discussion with great interest as some time ago I came across a set of modular medieval buildings for blender. Thus far I've only imported them into Max successfully (1 (https://www.dropbox.com/s/e3spcnab251fpda/modular%20medieval%20fbx%20preview.jpg?dl=0), 2 (https://www.dropbox.com/s/baltjbiwspchgtd/modular%20medieval%20fbx%20preview%202.jpg?dl=0)) but that's about it. And given I've picked up an interest in programming lately I may start to tackle this case from the opposite end, i.e. fool around in Python a bit, look up library for fsh compression. .  &Thk/(

Also, there's an awesome tree generator plugin for Blender. .  ::)   Ah well, lets not get ahead of one self. .  :D
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 29, 2017, 12:17:36 PM
Quote from: vortext on January 29, 2017, 12:03:44 PM
And given I've picked up an interest in programming lately I may start to tackle this case from the opposite end, i.e. fool around in Python a bit, look up library for fsh compression. .  &Thk/(

Feel free to do so, but as it stands right now, that's really unnecessary. The tools from BAT4Max can be used for that, in fact those tools are just command-line based versions of the FSH tools we are using already, they aren't specifically for BAT4Max, the scripts used by BAT4 Max merely sends them the commands to run. When I made the Diagonal addons for Cal's Canals, I couldn't get the scripts to work properly with Max9. So I had to manually take the exported textures and create my own scripts to pack them. So I know for a fact, if we can get Blender to render the images correctly, that side of things is a trivial matter. The hardest part, randomly (or programmatically) assigning the IDs for the BATs. That said, this is handled by SC4BAT when it exports the LODs. So the scripts can just take the ID from there for importing the new textures. Again, the tools to read/write DBPF data already exist for this.

However, Python knowledge would come in handy for scripting in respect of the rendering process. Since having figured out the 20 angles needed for a successful render, a script to automate that would be a high priority. But, having worked with BAT textures myself, I know even when you have the z5 camera setup correctly, the scaling is not as simple as you might think for the other zoom levels. I.e. z4 textures are not exactly half the size of z5, the texture size might be, but the part that renders is not. This is really the key to unlocking this, getting the settings for rendering. Armed with that, the remaining parts are relatively easy to make functional. At that stage, we could look at incorporating useful features like Auto-LOD creation, different lighting setups and all the other bells and whistles that we can utilise with tools like BAT4Max. Those would require more knowledge and effort to include. But a basic functional script that renders the textures should be the first priority obviously.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 29, 2017, 12:26:26 PM
Ah I see, so basically what's needed at first is a script for Blender to take the camera render and send the output to FSH tool? 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: art128 on January 29, 2017, 03:07:31 PM
What would be interesting too is a script for blender to replace sc4bat completely. That way we could create BATs under Linux, from modeling to LODs, without having to use wine or a vm.

That aside, this is a very interesting thread which I will be following closely.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on January 29, 2017, 04:52:56 PM
Quote from: art128 on January 29, 2017, 03:07:31 PM
That way we could create BATs under Linux, from modeling to LODs, without having to use wine or a vm.

I like that idea too. :thumbsup:
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 29, 2017, 09:51:00 PM
Quote from: vortext on January 29, 2017, 12:26:26 PM
Ah I see, so basically what's needed at first is a script for Blender to take the camera render and send the output to FSH tool?

Pretty much. The script is for keeping things simple. Setting up twenty different camera angles and hitting render would otherwise be tedious. Assuming that outputs BMPs like 3DS max does, everything else from there would work much as it does now.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on January 29, 2017, 10:16:23 PM
If I may add we could try to export the current camera angles from gmax or max and import them into blender.

I have some scripts that allow the export from gmax to blender so If we can get the camera angle rig out and import it into blender that may be immensely helpful.

Just my two cents. I like where this discussion is going.

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on January 29, 2017, 10:36:44 PM
The entire process is in the Gmax\Gamepacks\BAT\scenes and scripts folders

This is quite readable (if a little convoluted) and someone (programmer/scripter type) knowledgeable in Blender and gmax could reasonably assess whether its doable.

Some time back I extracted all of the basic info on the projection and wrote this article to explain it (mostly to myself)

  http://users.tpg.com.au/rivit/rvt-simcity-articles2.htm (http://users.tpg.com.au/rivit/rvt-simcity-articles2.htm) 

and I've recently used this to allow Bender to project planar tiles so we can edit rails or roads encased in Bats.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on February 01, 2017, 05:57:58 PM
I have been checking out the BAT scripts rivit mentioned, and the python scripts in Blender, there seems to be similar functions in both cases. If the BAT scripts can output the necessary co-ordinates for the cameras I don't see any reason why I, or someone can't get these into a python script to set up Blender, and export content in a usable format.

Also followed your link rivit, I thought it was well written, just a completely new concept for me. I had to go over it a couple times.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 01, 2017, 07:44:00 PM
I've been using Blender and have found it pretty easy to learn once you get the hang of the interface. Getting things from Blender into gmax/BAT is a bit of a pain, though. There are some old tutorials on this site that are of use; so far the big thing I've found is that textures need to be rendered in in Blender's cycles render, not its internal Blender renderer
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on February 01, 2017, 07:51:21 PM
Just for clarification, the purpose of this is not simply so we can use Blender, that doesn't actually provide any tangible benefit if we export to SC4 BAT and render models there. Since SC4 BAT aka gMax does not produce renders of a very high standard. That's why most people went to 3DS Max, to achieve better quality renders, whilst retaining compatibility. So the real purpose of this thread is to find a way to render in Blender, retaining the better quality we've all come to expect.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: art128 on February 02, 2017, 12:18:06 AM
Eeeh, I believe Blender can render in a quality just as good as 3DSmax's mentalray.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 02, 2017, 06:25:43 AM
Quote from: mgb204 on February 01, 2017, 07:51:21 PM
Just for clarification, the purpose of this is not simply so we can use Blender, that doesn't actually provide any tangible benefit if we export to SC4 BAT and render models there. Since SC4 BAT aka gMax does not produce renders of a very high standard. That's why most people went to 3DS Max, to achieve better quality renders, whilst retaining compatibility. So the real purpose of this thread is to find a way to render in Blender, retaining the better quality we've all come to expect.
You can happily render w either of Blender's renderers and then bake those textures into a map you can import into gmax, if that's what you mean?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on February 02, 2017, 08:05:09 AM
Quote from: tomvsotis on February 02, 2017, 06:25:43 AM
You can happily render w either of Blender's renderers and then bake those textures into a map you can import into gmax, if that's what you mean?

Right, but when you import them into gMax, how exactly are you getting them from there into the SC4Model file? If you have to render (Export) in gMax, then you are going to end up with a poorer render from it's engine (not to mention slower).

But if you are saying that gMax can take the renders from another application and somehow use them to build and SC4Model including it's LODs automatically, then in essence that's exactly what we're trying to achieve here. So if that's the case, could you tell us how you do it please? But, I must confess, that's not how gMax works as far as I understand it. Not that I'm an expert in this application by any means.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vester on February 02, 2017, 08:15:08 AM
Quote from: mgb204 on February 01, 2017, 07:51:21 PM
That's why most people went to 3DS Max, to achieve better quality renders, whilst retaining compatibility.

Better renders but also faster (if you don't use the more advance materials that comes with 3dsmax)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on February 03, 2017, 03:37:33 PM
I have some questions;

When you open Gmax or 3DSmax, do you have to set up the environment for SC4 production every time?

When you export the content what format is it in?

What is the next step?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on February 03, 2017, 05:44:25 PM
The short answer is that it exports .bmp pairs, which means a texture and a paired alpha channel, which simply dictates the transparency of the main texture. When converted to FSH, those will be combined together.

And the way 3DSMax works is that the scripts placed in a certain location are loaded automatically upon startup of 3DS Max. But for each scene, you need to initialise it's lighting and camera rig, handled by extra buttons which the script adds to the UI.

SC4BAT is launched separately from gMax, but is basically a plugin/script too. So assuming we had such a script, when ran, it should ideally handle this somehow.



I'm going to make the assumption you weren't looking for the short answer... Warning, wall of text incoming.

First up, if you need a model you can render for testing purposes with gMax, see here (http://sc4devotion.com/csxlex/lex_filedesc.php?lotGET=1509). Similar models exist for 3DS Max too, I picked the canal because it's a reasonably simple model.

SC4BAT is a plugin for gMax, so it's that which contains the scripts and settings for exporting SC4 Models. gMax can render stuff, but natively that's fully 3D models.

Exporting is a two-step process, the first just creates something called LODs. A LOD is basically a box that covers the entire area of the model. SC4 Models need three of them, one for Z5, Z4 and Z3, hence they are named LOD3, LOD4 and LOD5 in your scene. The LODs are created, 4 for each zoom level, covering the rotations/viewpoints of the game (z1 and z2 LODs are a very simple box which is auto-generated, but does not exist in the scene). This is the part that 3DS Max users will still do using SC4BAT. Part of the BAT4MAX scripts help to create suitable LODs automatically, but they are imported as .3DS models into SC4BAT. This step also denotes the ID, picked based on a timestamp, which has been great at avoiding conflicts. All of this can be done by making three suitable LODs in Blender and Exporting them as a 3DS file into SC4BAT. Then you end up with an .SC4Model file, which is full of blank textures, but has the LODs, IDs and UV Mapping settings needed for the next step...

Now the part that we are trying to achieve in Blender, let's assume we've got the 20 camera angles/lighting rig all setup correctly. It just needs to render 20 .bmp files. Here's how BAT4MAX's scripts handle that... so it reads the TGI of the model (the IDs for the textures) from the filename of the SC4Model file. I'll walk you through an example:

So if you look through your plugins folder, note how many files will be named such that the TGI or ID is included in the filename. Sometimes they are removed to tidy things up, but in their unmodified state, a typical example would be:

          MGB_RetailBanner_Centra-7ab50e44_e4f71a8f_30000.SC4Model.

All models should have the same Type ID (the First one, Black) and use the same Instance IDs (Third one, Blue). Also note that 30000 is short for 00030000, where preceding 0's exist, it will cut them, but each ID is always 8-digits in length. Now, that means for every new model, only the Group ID (Middle, Red) will be altered, this basically makes the model unique.

<I have to mention it, but sometimes a model will actually use the Instance ID as the unique identifier, not the Group ID. This is pretty rare to see in practise, but I think relates to failing to install a patch for SC4BAT. Rather than muddy the waters, for now, let's assume we just have to deal with the Group ID as the identifier.>

So the Instance IDs all work very logically from here on out. 3XXXX is the ID range for each zoom/rotation, which breaks down as follows:

          0x0003TZRU



               
               
               
               
               
               
T = TimeofDay    Z = Zoom            R = Rotation        U = Unique ID     
0 - Day Render4 - Zoom 50 - South0-F*
8 - Night Render3 - Zoom 41 - West
2 - Zoom 32 - North
1 - Zoom 23 - East
0 - Zoom 1

* = The unique ID is used when a texture needs to be larger than 256x256px as part of the model. In such cases the texture is broken up into smaller segments, each will have it's own unique identifier from 1 to F (1-16). That's also why no model can be so large that it needs more than 16 textures at zoom 5. Because that's a hard limit of the ID system used for them. Update: Actually this turns out to be incorrect, once the limit is hit for IDs (16), it can re-purpose IDs to the left that are unused. For example, TimeofDay only ever uses 0 or 8, so more unique IDs can be created if need be. This information is actually commented by a coder in the original script, although I can't remember exactly which digit or digits can be used. Although there is a limit somewhere as to the maximum size of an SC4 model, plus it's rare to need more than 16 images too.

Because these are all fixed, the scripts can work out from the ID of the SC4Model file, the exact IDs needed for each texture that's rendered. Although in practise, those are named simply 0000.bmp / 0000-a.bmp (Alpha or Transparency Channel). The scripts create twenty or more folders, which have the actual IDs. Two batch (.BAT) scripts are generated as part of the process to facilitate this. So the scripts just need to know which folder to place each set of .BMP pairs inside. To help you see what's going on here, I've attached a sample output from the 3DSMax scripts (note: this is only the night render, which overwrites the day render). The scripts can handle switching the T ID from 0-8 as necessary. So it only creates one set of folders for both day/night images.

All of this is based off the BAT4MAX process. But the tools used with the scripts are things we can re-use easily. One a render's images are all output, it converts them to FSH based on the folder names (IDs) and replaces the previously blank images with the newly-render model's ones.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on February 03, 2017, 07:16:29 PM
Thanks for the info Robin, your insight and investigation is absolutely awesome :thumbsup:

I know many hands make light work as a community we can find a solution to make Blender render models. The discussion here is just plain awesome. No doubt we will need to re-train ourselves to use blender but I do feel that we are closing in on a solution for making BATs with blender.

Quality BAT making keeps drawing users back to unleash their artistic and architectural desires and making the tools more readily available will be a boon to the content creators and user base alike. Keep it up everyone!

-eggman121 (Stephen)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on February 03, 2017, 09:12:19 PM
You assumed correctly. Thank you. I will digest this and very likely have more questions.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: woodb3kmaster on February 07, 2017, 08:50:55 PM
A thought just occurred to me: Assuming we can set up the camera/lighting rigs correctly in Blender (which should be a snap thanks to the data rivit linked to earlier), it should be possible to create the SC4Model file in gmax (as we already do with BAT4Max) and use something like an automatic version of GoFSH to convert the BMP textures to FSH. At that point, it mostly boils down to writing a Python script that splits the textures into 256x256 chunks, if I'm understanding the technical challenges correctly. This is seeming more and more feasible.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on February 08, 2017, 05:25:55 AM
I have been studying the Blender python api and it seems like setting up the camera and lighting is going to be possible. I have not spent much time looking at the render processes yet, though it also looks promising. I have read a lot of articles and received some good info here, but I am still in the early stages. I have experience with python, but zero with 3D processing, so it will take time. Now I am going to take it one step at a time and just see where it takes me. Can't make any promises, but I am cautiously optimistic.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: bombardiere on February 08, 2017, 07:44:23 AM
This is interesting to read. Sadly I don't skills in this field so I will remain as an observer. However, I will look forward for what ever progress may yet to come.

I have tested Blender before, but personally I prefer 3DS MAX. May be it is because of my GMAX history but I like how it works. When I look at the Blender, I just get confused. However, It is a big scare that there will be a day when 3DS MAX is not suitable for batting. In every new version they brake something. I have about a year and half left of my 2014 Student licence. After that there may not be a suitable version available for batting. :( So if there can be a Blender script I am willing to learn the Blender.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on February 24, 2017, 05:47:22 AM
  I have been able to spend some time working on scripts for Blender. So far, just some simple UI stuff, added a panel and created 3 buttons to move things around on the xyz axis. Not very exciting but it helps me get a feel for the syntax and format.

  Unfortunately I am getting bogged down trying to implement the actual procedures pointed out in the info posted here. This stems from my complete lack of knowledge related to using Blender and modelling in general. I find myself spending way to much time trying to learn enough about these things so I can script the procedures. Unfortunately I just don't have enough time, right now, to take this route so I would like to try breaking it into smaller pieces.

  I have extracted some quotes from current posts and pasted them below, along with some thoughts and questions, in the hope of moving this along one step at a time.

mgb204 - posted
  "What we really need more than anything else is a plugin/script that can render an object created in Blender into the 20 views used by SC4 Models."....

  ...."The key barriers here are working out the following:
    Where to place the "cameras" so that the render appears correctly.
    The specific angle of that camera in relation to SC4s isometric view.
    There will no doubt be a need to set up the lighting too."....

From rivit's linked article:

Az123456    67.5 deg    Range  190m
ViewPoint   m,deg        D deg          E deg           Fx               Fy               Fx
El 456              45 deg       16.32495     59.63881          0.96269       0.75712       0.70711
El 3                 50 deg       14.90920     57.20219          0.95607       0.70648       0.76604
El 2                 55 deg       13.36473     54.16472          0.94960       0.65365       0.81915
El 1                 60 deg       11.70092     50.36073          0.94349       0.59986       0.86603

  It seems, to me, that these all concern the same piece of the puzzle.

  Is this correct?
  If yes:
  Can someone show me exactly where to input rivit's co-ordinates into Blender?
  (Please use screenshots, if possible)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tigerbuilder on March 05, 2017, 02:17:44 PM
I've been reading around on the net on my off time, I thought I'd share some observations I saw that seem interesting areas that could apply here.

The BAT/BAT4max must be a "maxscript" I'm assuming. There is information available about converting from maxscript to python. I thought that was interesting.

The cameras and scenes and all easily scripted. The problem is the transfer of that render to the LOD we need. I haven't seen much about that part.

I thought I would post to maybe stir this up again
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 05, 2017, 03:27:19 PM

That's good news. In the main script (I think BuildingMill.ms) in Gmax, there is code that takes the model as in the scene and rotates it to the correct orientation to export for an S3D the model file in SC4. Otherwise the LODS (which are really just a hat rack to hang the textures off) can be exported as a 3DS.

The rendered projected images need to be broken up into maximal 256x256 pixel blocks and correctly numbered before export. That code too is in one of the scripts.

Now Gmax uses a proprietory dll to get from Render and LOD to numbered S3D and FSH, but we may have the knowledge and code to do this if we can get them out as S3d and say BMP( or PNG) with alpha.

Can Blender accept home grown plugins? i.e. could we code a plugin to do this for us?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on March 05, 2017, 05:22:45 PM
Thanks for all the reply's everyone.

Yes I think we are getting closer to a solution for this. Everyone who has contributed in this thread should be commended.

To keep SC4 modding alive I do believe we need too get programs that are at least free licensing but even better open source for the continuation of the making of quality BATs.

I wonder if a tool Like GoFSH could be adapted to take the renders of the model and apply the required FSHs in the BAT?  :-\

That would be a good test stage until we uncover a universal Blender solution.

-eggman121


Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on March 06, 2017, 05:08:18 AM
QuoteThe BAT/BAT4max must be a "maxscript" I'm assuming. There is information available about converting from maxscript to python. I thought that was interesting.

tigerbuilder, could you post links to the info you referred to?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tigerbuilder on March 06, 2017, 09:43:04 AM
This is an example, handyman.

https://sourceforge.net/p/blur-dev/wiki/Py3dsMax/

There was more reading...I was googling things like "maxscript to python converter".

I have zero scripting experience..but I thought, what about the scripts that we already use...could they be used? That led me to looking around on that topic.

Oh by the way...this ST thread came up from 2008...boy it would be nice to have Simfox around.  His answer in this thread basically is where we are still.

http://community.simtropolis.com/forums/topic/22179-blender3d/

Here is something interesting about the rendering to a lod. I'm on a chase for this information.

http://blender.stackexchange.com/questions/46891/how-to-render-an-environment-to-a-cube-map-in-cycles

Cheers, tiger
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on March 06, 2017, 05:27:32 PM
Thanks for the links tigerbuilder, lots to look at. :thumbsup:
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: null45 on March 06, 2017, 11:00:07 PM
Quote from: eggman121 on March 05, 2017, 05:22:45 PM
I wonder if a tool Like GoFSH could be adapted to take the renders of the model and apply the required FSHs in the BAT?  :-\

The FSH conversion in BAT4Max 4.5 and later uses a command line tool that I wrote to replace FSHTool.
The DXT compression algorithm used by FSHTool produced a large number of compression artifacts when used by the DarkNite truNite export process in BAT4Max.
SimFox posted more on that issue in the BAT4Max 4.5 thread (http://community.simtropolis.com/forums/topic/34613-bat4max-2k-edition-v-45/?do=findComment&comment=886510).
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Handyman on March 07, 2017, 03:44:55 PM
I have looked through the links provided by tigerbuilder, and a few more. At this point, I don't think this will serve our purpose as it all seems to be aimed at using a Python api to wrap maxscripts for use with Max. Even though Blender runs, mostly, on Python, we would still need an environment that can support the maxscripts. I did a number of searches with this in mind and came up empty, so far. :'(
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on March 07, 2017, 07:49:33 PM
I think perhaps there is some misunderstanding re my previous postings here.

In short, the only thing we need to get Blender to do is render a model into BMP files, making each texture using the correct camera angle for each zoom/rotation needed.

The export process would be identical to 3DS Max + BAT4MAX, i.e.:


So in short, the only part we need concern ourselves with is getting the correct textures from Blender. Everything else that's required for this to work exists and can be reused. UV Mapping is handled by SC4BAT, because that is applied based on the LODs, which will be 100% accurate as they originate from the model in Blender.

I hope that clarifies things.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on November 16, 2017, 08:22:02 PM
Has anyone managed to make any more progress w this? I've picked up modeling again recently, in Blender, and have come up against the same wall, i.e. that it seems nigh on impossible to import a Blender model into Gmax. It's driving me insane.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on November 16, 2017, 10:21:16 PM
Quote from: tomvsotis on November 16, 2017, 08:22:02 PM
Has anyone managed to make any more progress w this? I've picked up modeling again recently, in Blender, and have come up against the same wall, i.e. that it seems nigh on impossible to import a Blender model into Gmax. It's driving me insane.

You can save a model as .3ds and import from there.

I am pondering weather I should make public the method of making True 3d models with FSHs attached like transit Pieces open to others. I have been able to get models out through the Quake .md3 method to make true 3d Models for NAM and thus ditching 3ds Max (With its overpower and prohibitive License tag).

Theoretically you could have true 3d for some parts and combine them with a normal BAT from Gmax. The way this could be done is with Cogeo's Model Tweaker which combines different Types of models into one. That is one tool I am truly grateful for  ()stsfd()

That way you could assign FSHs to the model parts that can be modified and pasted onto a BAT to give the illusion of different texture types.

Thoughts?

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on November 16, 2017, 11:58:13 PM
Yeah, to clarify, the prob i'm having isn't importing the model; it's successfully applying the textures i baked in Blender. Has anyone managed to do this? I baked a texture map, but I can't seem to get Gmax to import the UV map from Blender w the 3ds file, and even if I do manage to get the map into Gmax (via exporting from Blender as a .obj file), the baked textures don't seem to follow the map and thus show up on the model as a giant mess. Ugh.

And eggman, your method sounds fascinating, but it's well beyond my experience at this point!
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: ????? . on December 17, 2017, 09:00:09 PM
Blender to Gmax (importer for gmax)
UVmap OK.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on December 17, 2017, 10:16:36 PM
Thanks for the Reply!

This will be very useful for getting models out of Gmax and into Ilive's Reader tool.

Many thanks too you  ;)

I use the quake route for making models but this is equally valid.

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on December 18, 2017, 01:21:14 AM
Oh, AWESOME. I will look forward to playing with this!
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 22, 2019, 06:08:06 AM
Played around a bit over the weekend with the 'new' blender (thanks tomvsotis for mentioning that (http://sc4devotion.com/forums/index.php?topic=18349.msg529967#msg529967)  :thumbsup:) and first I must say the ui definitly is an improvement.

At any rate, with the help of rivit's article (http://users.tpg.com.au/rivit/rvt-simcity-articles2.htm) and some digging around in the game files to find the sun angle I managed to recreate the game perspective & lighting somewhat decently I reckon (for the closest zoom any way, and no foreshortening applied yet). Though there're some hardly explained settings for the ortographic camera so I eyeballed a few things.

Here's the viewport perspective (with eevee render) showing a 16x16x16 cube with a 'roof' on top:
(https://i.imgur.com/74vLY50.png)

And here's the actual render (using cycles render):
(https://i.imgur.com/orSgIJ1.png)

Also got a barebones addon script working because as it turns out enabling / disabling addons is the fastest way to reload python scripts in Blender. Lots of things to tackle still but definitly seems doable. .  ::)

The major question on my mind atm is how gmax / max handles the camera allignment for rendering. That is, if you look at a fsh file it always show the renderer model alligned to the left. Guessing camera allignment is somehow related to LOD size but howww. .?  &Thk/(  :D
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on January 22, 2019, 08:05:44 AM
Hats off to you for getting this far. Does this work for all 4 rotations in zoom 5?

Re: Camera Alignment, I guess the first question is does the exported texture Map correctly to the LOD exported by SC4BAT? Because if it does, then whilst it may not be completely optimised, at least it's a working render.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on January 22, 2019, 08:13:41 AM
The first BAT rendered with Cycles would be like stepping on the moon, I guess.
 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 22, 2019, 08:53:30 AM
Quote from: mgb204 on January 22, 2019, 08:05:44 AM
Does this work for all 4 rotations in zoom 5?

No, atm it just add a camera & light to the scene, that's it.

Quote from: mgb204 on January 22, 2019, 08:05:44 AM
Re: Camera Alignment, I guess the first question is does the exported texture Map correctly to the LOD exported by SC4BAT? Because if it does, then whilst it may not be completely optimised, at least it's a working render.

Hm yeah, about that. Looked for the s3d export, couldn't find it anywhere and a quick search result seems to indicate the 3DS export is not ported (yet?) to version 2.8 (https://developer.blender.org/T57001). .  Guess it's back to 2.79 for the time being. . Or import the LOD as another format into gmax. . e.g. obj mentioned below  :-\

And perhaps should've been a bit clearer, currenlty no view rendered with blender has been brought ingame, nor has a 'lod' been exported.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on January 22, 2019, 10:27:20 AM
Great work here vortex (Erik) :thumbsup:

This is a really solid breakthrough  &apls

I really appreciate you making this happen and look forward to what we can do with blender.

+1 for your effort

-eggman121 (Stephen)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 22, 2019, 10:52:17 AM
Thanks Stephen!

so as a rudimentary answer to the question;

Quote from: mgb204 on January 22, 2019, 08:05:44 AM
does the exported texture Map correctly to the LOD exported by SC4BAT?

A very quick trial indicates 'no', along with host of additional issues. Then again, I opted to give the obj route a go and who's to say that's desirable to begin with.

At any rate, exported a 'unit' cube as obj from Blender, then using this script (http://hce.halomaps.org/index.cfm?fid=3451) imported it into gmax. No scaling required, but had to change the axes on export from Blender. Next saved the rendered image from Blender as bmp, and used GoFSH to import it into a .dat. Finally replaced the appropriate texture in the sc4model.

sc4model export from gmax 
(https://i.imgur.com/xT7dwq2.jpg?1)

rendered texture applied
(https://i.imgur.com/Icfidcy.jpg?1)

Note the shadow is also not correct, probably due to lacking the proper alpha (tho the exported bmp was argb it appeared to be lacking alpha in gofsh as well, perhaps it would need to composited in Blender. . ? &Thk/( )

Here's a better indication of what's going on with regards to the uv.
(https://i.imgur.com/hkXusJ1.png)

All things considered it looks like the camera needs to be pulled back a bit, or would it soley be due to no foreshortening applied yet? In the article Ron also mentions the camera having a 'range' of 190 meter in the default rig. However I don't quite understand what it means; is range the distance from the location of the camera to the origin of the scene?   
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on January 22, 2019, 02:19:18 PM
@vortext,
   that's already a great advance on where we were yesterday, so to speak.

I interpret "a 'range' of 190 meter" as the camera distance from the object, everything else is done by the zoom i.e. changing the field of view by making the render wider/narrower in the given ratios. Alternatively the Range*146/Zoom Ratio will do the same ie 146,73,32,16,8  for the Zoom5 down to 1. As you've deduced SUN is always 90 from the left so that shadows fall horizontally. Elevation varies with Zoom. Foreshortening is entirely handled in the projection parameters - ie is intrinsic to them. so you don't need to do anything for that. Just make sure you're not rendering in normal 3d perspective.

As for alignment I remember that gmax has a variable called viewportslop which is the left margin of the render, and I'm pretty sure its aligned by finding the leftmost projection point from the model and the view (ie maths) but there may be intrinsic functions in Blender to help deduce thus.

If you have the original render bmp it might pay to check that there is indeed alpha in it by looking in photoshop. Windows doesn't play nicely with alpha esp. when exported from a program relying on windows libraries. Hence it may have once had alpha that has now become all ones. If you can export as PNG that will prove it. That will also load into GoFSH correctly.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on January 22, 2019, 09:21:18 PM
Amazing! I'm glad I mentioned this :)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on January 24, 2019, 03:23:11 AM
Btw, I've done all my modeling up to this point in Blender 2.79, because it's what I'm familiar with, but i'm now playing around w 2.80, and it's SUPER impressive. if you guys aren't familiar w Blender, this is prob a good time to investigate it -- the UI is wayyy better than it used to be, and it's also got a fancy new real-time rendering engine, which is interchangeable with the more photorealistic inbuilt Cycles renderer. It's really pretty great, especially because it's free and open source.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 27, 2019, 05:38:42 AM
Quote from: rivit on January 22, 2019, 02:19:18 PM
Foreshortening is entirely handled in the projection parameters - ie is intrinsic to them. so you don't need to do anything for that.

Thanks for clearifying that Ron, indeed I was under the assumption foreshortening had to be applied, however reading up on orthographic projection it makes a a lot of sense it is the result of it.

So since the dreaded 'm' word has come up, it seems I've managed to calculate the camera position for all zooms using a fixed range of 190 meters. In each render the camera is looking at the center of mass of the lod (0,0,8) and uses the same orthographic scale and no offsets (more on that in a bit). On a sidenote, in gmax I noticed the camera looks at a position slightly above the lod, would be interesting to know what the look at point is in studio max.

(https://i.imgur.com/YGkOif5.png)


Now onto what seems crucial to get right: the orthographic scale of the camera, and closely related the x & y camera offsets. Blender describes the scale as being 'similar to zoom', and online I found the following explanation (https://blender.stackexchange.com/questions/52500/orthographic-scale-of-camera-in-blender);
QuoteThe Orthographic Scale factor represent the maximum dimension (in scene units) of the portion of space captured from the camera.
Or to put that yet another way, when the camera is looking straight down to the ground the dimensions of the surface area captured by the camera correspond to the scaling factor (and this holds true irrespective of camera height location).

So if for example an orthographic scale of 32 is used (as I had done initially for zoom5 as it seemed to make some intuitive sense) and the camera looks down, the area captured by the camera measures 32 x 32 meters. However by trail-and-error I arrived at the following settings for the zoom5 render. Also note the offsets to manoeuvre the model to the topleft corner. The offsets use a 0..1 range whereby an offset of 1 moves the camera by the scale value along the axis, thus I reckon it should be viable to use them as viewport stop values.

(https://i.imgur.com/4gwfsNW.png)

And side-by-side comparison. 

(https://i.imgur.com/XzVoKG9.png)

Also made an interesting find playing with some of the numbers in Ron's article. Here's the relevant numbers for zoom 5;




LevelZoom xRel RangeEquiv Range mPixel in m
Z5146126.10.102

The finding is that if an orthographic scale of 26.1 is used, i.e. the Equivalent Range in meters, the 16x16x16 model fits snuggly into the camera viewport heightwise (provided no offsets are used, and the camera is looking at the center of mass). I'm intrigued the corresponding Relative Range of 1 seems to indicate some 'true' scale, and is echoed by the heightwise fit of the model relative to the camera exclusivly. 


(https://i.imgur.com/mO3Qrc9.png)


So yeah all things considered the orthographic scale seems to be quite crucial indeed, and I stronlgy suspect it can be calculated using the known camera positions and render dimensions but alas, my maths game is not that strong.  $%Grinno$%
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Barroco Hispano on January 27, 2019, 07:15:33 AM
@vortext

(https://nsa40.casimages.com/img/2019/01/27/190127041719358394.png)

Z5: 8 Meters Z4: 16 meters Z3: 32 meters Z2: 73m Z1: 292m    (Z axis)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 27, 2019, 08:43:19 AM
Yeah I tried to those zoom ratios, however, they seem intrinsic to how gmax / max handles things and appear to have little use in Blender. For example here's the zoom5 camera with an orthographic scale of 8;

(https://i.imgur.com/5A1Kqqt.png)

Likewise applying a scale factor of 292 is way too much. And keep in mind pulling the camera back makes absolutly no difference to how much of the model is in the camera viewport (nor does changing the rendered output dimensions btw). As said, the orthographic scale is similar to zoom but not quite the same.  &mmm

Guess the question is; if the gmax /  ingame cameras are pointed down towards the ground, what are the dimensions (in meters) of the surface area captured? At least if I understand correctly this should correspond to the orthographic scale used by Blender.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on January 27, 2019, 12:03:57 PM
These are very interesting discoveries!

Does this help any with the camera render scale? Note: Maths and I aren't friends, so I could be completely off-piste $%Grinno$%

-- set camera fov large enough so width of building appears in view
      renderFov       = 2 * (atan (targetWidth2/curCam.Target_Distance))
      
      if usingThumbnailCam then
      (
         renderWidth    = (146 * 2.0 * targetWidth2)/(cellMaxX - cellMinX)
      )
      else
      (
         renderWidth    = (zoomSizes[curZoom] * 2.0 * targetWidth2)/(cellMaxX - cellMinX)
      )
      renderWidth    = ((renderWidth  + 0.5) as integer)      -- round off
      renderHeight    = (((meshMaxY + viewportSlop)/targetWidth2) * renderWidth)
      renderHeight    = ((renderHeight + 0.5) as integer)      -- round off
      
      renderScaleX    = (0.5*curCam.Target_Distance) / targetWidth2
      renderScaleY    = (renderWidth * renderScaleX) / renderHeight
      
      true
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on January 27, 2019, 12:25:29 PM
@vortext

You've made great progress - this is starting to look a lot better every time you post - do you feel like an explorer in 1492 yet?

@mattb325

  yes I recognise those - fundamental unwrapping of the 3D projection back into 2D.

 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on January 28, 2019, 05:46:30 AM
Yeah that helped Matt as I managed to get confirmation about the orthographic scale.  :thumbsup:

First I calculated the fov according to script, using a targetWidth2 of 24 (composed of mesh width 16, and viewportSlop 8 as defined) and Target_Distance of 190. Next using the fov angle and known camera height in the Blender scene (142.35) I calculated the length of the opposite side of an imaginary triangle, as if the camera was looking down, which came out as 36.54528. Very close to what I arrived at manually indeed (and btw the hypotenuse of this triangle is 146. . ).

However this seems a bit roundabout way of getting the scale, seeing the orthographic camera in Blender does not have a field of view property (and which appears to be different from max as the render camera does have a fov set, however, orthoProjection is set to true as well). Besides it still leaves the camera offsets and output dimensions to be calculated as well. . so yeah..  &Thk/(

However looking over the entire script a few times I got a vague idea of the steps involved rendering a model of arbitrary size, so I'll have to do some digging in the Blender api and refresh my trigonomy to figure things out.


Also note to self: do not give the export from Blender a filename starting with 'lod' as this will somehow cause gmax to actually wipe it from the scene .. was a fun half hour figuring that out  &sly

btw Ron tbh I'm starting to feel more like this (https://media.giphy.com/media/l0IylOPCNkiqOgMyA/giphy.gif) . . the angles and the scale..  it's, like. . all related man   :D
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on January 28, 2019, 02:29:14 PM
"the angles and the scale..  it's, like. . all related man"   Yeah I know exactly what you mean. A lot of the time its "miracle occurs here" until something drops into place.

https://cellcode.us/quotes/miracle-occurs-here-cartoon.html (https://cellcode.us/quotes/miracle-occurs-here-cartoon.html)

If I can help in some way don't hesistate to ask - perhaps I should download Blender and see if I can parallel your efforts so we have notes to compare? Im a bit of a noob with modelling programs though.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on February 03, 2019, 04:21:58 AM
Quote from: tomvsotis on November 16, 2017, 11:58:13 PM
Yeah, to clarify, the prob i'm having isn't importing the model; it's successfully applying the textures i baked in Blender. Has anyone managed to do this? I baked a texture map, but I can't seem to get Gmax to import the UV map from Blender w the 3ds file, and even if I do manage to get the map into Gmax (via exporting from Blender as a .obj file), the baked textures don't seem to follow the map and thus show up on the model as a giant mess. Ugh.

And eggman, your method sounds fascinating, but it's well beyond my experience at this point!

If I export 3ds from Blender, most of the time it doesn't work in gmax for me ("improper format" or so).

You may try this:

a) download this converter written with gmax in mind (by a ms flight simulator fan): https://www.scenerydesign.org/modelconverterx/ (https://www.scenerydesign.org/modelconverterx/)
b) from Blender export to *.dae format
c) import the dae into the model above converter and export as 3ds.

Please tell me if this works for you too. Thanks.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 10, 2019, 06:17:31 AM
Quote from: rivit on January 28, 2019, 02:29:14 PM
If I can help in some way don't hesistate to ask - perhaps I should download Blender and see if I can parallel your efforts so we have notes to compare? Im a bit of a noob with modelling programs though.

As been said this would be a good time to try out Blender for sure, though couple of things to keep in mind. First it still is quite a daunting piece of software to navigate as it is capable of so much more than just modelling (animation, simulation, fx, post processing, etc) so be prepared for lot of 'which damn menu was this thing again. .' 

Secondly, the new version does introduce an unwanted side-effect in that if you search for 'how do I do this specific thing in blender' around 95% of the results are for older versions. Some of these will be applicable 1:1 with the new version, other results will be completely useless. For example something as simple as changing the major gridlines in the viewport took me more searching around time than it should (it's now in the overlays menu btw (https://www.katsbits.com/codex/grid/) ). So this can be tad frustrating, but of course this will change over time as more and more people adapt to 2.8

At any rate, I'm confident the scaling can be resolved one way or another. Either by doing it in the roundabout way of calculating a fov first, then somehow fitting it in camera view. Or by taking a cue from the (g)max scripts and using transform & camera matrices to work things out.

Either way both solutions wil need a proper functioning add-on to make use of, so that's what I decided to focus on for the time being. Shown below first glance at the add-on located in the 'output' context menu, which I figured was appropriate. Besides is isn't as cluttered as some of the other context menus so scrolling is kept to a minimum.  :)

(https://cdn.discordapp.com/attachments/383313438065033230/544151654451249172/blenderbat_1.gif)


Next thing will probably be figuring out how to handle rotations. I noticed the (g)max script actually seems to rotate the camera (and I presume lighting) around the object. However at the moment I'm thinking to rotate all the scene objects around the world origin, as this functionality could potentially also benefit making diagonal and far renders. Moreover I'll need to go over all the scene objects anyway in order to fit the LOD. So yeah, lots of details to be hammered out still.  :-\
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 10, 2019, 03:01:16 PM
Thanks for the tips - I have downloaded it and taken a look and I agree its like standing in a blizzard looking for a particular snowflake. At least I don't have the advantage? of having seen previous versions.

However I did also take a look at the manual for scripting and it appears nearly all of the internal data model is available with appropriate code. I'll need to wrangle the language first.

I've given myself the project of seeing if I can make an action panel based on one we know well from (g)max and see if I can gradually hang things off that. At least once its set up I won't have to look about as much. Your example is the sort of thing I had in mind. Then I will also have got a bit of a grip on the syntax and feel of a tiny piece of the data model.

Quote from: vortext on February 10, 2019, 06:17:31 AMHowever at the moment I'm thinking to rotate all the scene objects around the world origin, as this functionality could potentially also benefit making diagonal and far renders. Moreover I'll need to go over all the scene objects anyway in order to fit the LOD.

I ended up here in thinking too - since SC4 always uses the same lighting regardless of viewpoint rotating the model should work. (although given its under a button in the end its probably just as easy to rotate cam and lights)

And then of course the script sort of reduces to

for day &/or night
   set environment
   for zoom is 4 downto 0
       set projection     
       for rotation is 0 to 270 by 90
          setup render and orient
          render, slice'ndice, number things, make fsh, project LODS, make s3ds, save the bits
     next rotation
  next zoom
next

easy?  nope. Each word is like an explosion - gets big fast.

~~~
A question - can you set Blender up to be/work in Left Handed, XZ base plane, Y pointing up Coordinates ie in DirectX mode a la SC4, or is it Righthanded XY baseplane, Z up only?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 10, 2019, 05:01:47 PM
Quote from: vortext on February 10, 2019, 06:17:31 AM
Quote from: rivit on January 28, 2019, 02:29:14 PM
If I can help in some way don't hesistate to ask - perhaps I should download Blender and see if I can parallel your efforts so we have notes to compare? Im a bit of a noob with modelling programs though.

As been said this would be a good time to try out Blender for sure, though couple of things to keep in mind. First it still is quite a daunting piece of software to navigate as it is capable of so much more than just modelling (animation, simulation, fx, post processing, etc) so be prepared for lot of 'which damn menu was this thing again. .' 

Secondly, the new version does introduce an unwanted side-effect in that if you search for 'how do I do this specific thing in blender' around 95% of the results are for older versions.

Yeah, that's def true. I've gotten pretty good w Blender over the last couple of years; if there's anything specific you're struggling with, feel free to ask, and maybe I can give you the answer?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 10, 2019, 06:45:27 PM
Quote from: fantozzi on February 03, 2019, 04:21:58 AM
*snip*
Please tell me if this works for you too. Thanks.
Thankfully, I've managed to get a copy of 3ds Max since then, so I haven't had to worry about exporting from Blender to Gmax. If it's any help, exporting from Blender as an .fbx and importing that into 3ds Max works pretty much flawlessly. (The exception is curved surfaces, for which there is a workaround that involves flipping the normals on the object and then adding a Normal modifier to flip them back agin. I have no idea why this works, but it does.)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Jasoncw on February 11, 2019, 08:39:43 PM
BAT handles the various rigs by linking them to a helper object ("TB2CameraHelper" or something like that), which everything else is linked to. Right now in BAT4Max the cameras are in their own scene(because the cameras never change) which is xreffed, and linked to the helper object. The lights used to be separate from the scene as well, but now they're within the scene itself, because they're constantly getting changed to switch between day and night(s).

So in BAT, when you change the rotation, the only thing it's doing is rotating one object, the helper object. If there's an equivalent in blender, the lights and cameras should be linked to a helper object, and that way you just need to rotate that one thing every time. Rotating the entire scene is more prone to introducing problems. What if all the building parts get rotated inconsistently? Or when you rotate the view for a preview render, does that change which facades show up in each viewport, and which parts of the building correspond to which axis directions changes every time you want to do a different preview render?


FAR and diagonal BATing could be approached different ways. I don't think an export setting would be as useful here, because when you make these buildings you usually need to customize them a lot first. It's not just a matter of it needing to be rotated, unless the building is completely freestanding and doesn't line up with roads or anything. The difficulty of those things is having to work both diagonally and orthogonally at the same time for different parts of the building. In 3ds Max, it's possible to create grid objects (which you could then rotate, so you could have your normal grids and viewports but also have grids/viewports for your FAR/diagonal facades), and having buttons to press to create those automatically would be useful, if that exists in blender.


Unfortunately, while I have some memory of BAT4Max getting developed, the most I've ever done myself are very very simple edits. I've also never used blender before. But when it comes time, I can help out more directly by helping to put together good lighting and rendering settings with the new renderer, so that things look good, match well, and are hopefully fast to render.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 13, 2019, 02:02:12 PM
The (software) engineer in me has found a need to specify what I'm doing:

Bat4Blender

Goal:
Using the Blender Application, to be able to produce .SC4Model files for rendered 3DModels, compliant with the parameters demanded by the SimCity4 application and with final outputs as produced by the gmax BAT tool.

Functional Specification v1.0 rivit
Art Features:
   To take advantage of the Blender Application Rendering capabilities to produce art compatible with SimCity4
   + advanced application dependent rendering techniques
   + control of exposure, color gamut, shadows, etc etc
   
   Legacy Compliance with SimCity4 and derived works
      - produce Buildings/Props and Foundations
      - SC4 Standard calibrations e.g. Sun and Shadows, Day, Night
      - Maxis Variable Night  i.e. Day with Night renders a blended overlay
      - BAT4Max v6? fixed DarkNight (or a variable equivalent) Day and Night both fixed renders.
         - MN is inconsistent with Maxis Night colors and should be dropped (is not functionally relevant if we can make a Variable Maxis night)
      - High Definition renders of Zooms (3,4),5 in addition to Standard Rendering

   Desirable
      + add support for baked textures for Automata

Technical Features:
   Simple dedicated application Tool Panel which controls all aspects of the BAT production
      should look/act like the gmax BAT panel
      
   Setup/Manipulation of Camera, Lights for Day and Night(s).
      Do we really need 6 cameras? or just camera settings according to Day/Night/zoom/view/resolution/renderstyle
      
   Produce automatic LODs of models for zooms (3,4),5 - preferably close-fitting accurate shells
   
   Support manually created LODs
   
   Legacy Compliance with SimCity4: must produce like outputs from gmax BAT.
      Support for randomised or manually entered GID
      To Produce Render and Package a .SC4Model file comprising
         4 Views and 5 zooms of each Model, Day and optionally Night
         Correctly TGI numbered components
         up to 64 panels per Model view at max zoom (SD or HD)
         FSH of the rendered model - DXT1 for Day, DXT3 for Night
         S3D files of the segments of the rendered LODs of the model
         Thumbnail Images

   Desirable:
   + Add support for exporting (s3d) of Automata and their Textures (DXT3)
   
Notes:
   This custom application for Blender v2.8 and following, is largely a porting operation from gmax BAT scripts, with all internal gmax program/function references taken up by Blender/Python equivalents. The gmax Scripts comprise 7000 lines of code - even at only at 5 min per line (understand, translate, test) this is 600 hours of code translation.

   There is a large language difference between gmax script (BASIC-like) and Python, although functionally the latter is likely to be richer given the age of gmax. The current scripts are very bound to gmax, rambling, untyped, not fully qualified and hence ambiguous to some extent - I have started (20% in) to port these (original gmax BAT) to VB.NET so that they can be at least checked for completeness by compilation. This allows me to separate script-only code from internal gmax functions which need to find Blender equivalents. Eventually it gives me a baseline protoype app shell independent of gmax which should make it easier to subsequently recode into Python (by someone who can). (Needs someone who knows the Blender API well)

   However, since we need to make such a large transition in code, we are not obliged to keep all features or methods applied in gmax, nor even functional constraints as long as we produce Simcity4 compliant outputs. Hence simplification should be a catch-cry wherever we can. Also newer techniques now exist for some things e.g. Does anyone use the old nightlighting techniques in gmax BAT? Making detailed LODs automatically should be easy nowadays with Shell Mesh functions.  We should take advantage of this if we can. (Needs someone who knows Blender well)

   Also, I'm not sure that all of the "parameters" of BAT are in the scripts - where are the art things actually defined? Documented? There are some things done outside of the script by MAXIS code I think - not clear yet. 
      
It's a project, Jim, but not as we know it

   It's a big project however you look at it. If it is to succeed in a useful timeframe it will need a lot of interaction and communication and a team of about 10 (not necessarily concurrently), split into
         Art/Technical types,
         Blender/Max types,
         Python/BASIC types,
         Experimental/Structural types,
         Starter/Finisher types.
         Document/Test types
         
   There needs to be a lead on Art, one on Technical and one on Prototyping/Testing.
   And Someone in charge of the charge.
      
  The work should proceed from the Art (Blender) towards the production of Correct Render and LOD, the technical from outputs backwards (from SC4) to there. This allows for a synch point. There is an overlap where technical control of the app is needed to produce the art - Menus, Rigs, Command Buttons etc. hence coordination will be necessary. We will have to bootstrap up from Prototyping to Structured development - its a LEGO challenge.
      
   This is a project equivalent in complexity to a NAM release that revises legacy with something new (think Flex replacing Puzzle Pieces) - will need the same dedication, perserverance and patience - is it worth doing? or are we too late? can we get this sort of team to stick? can we find enough of the right talent? real volunteers?
      
   please discuss and refine (what is missing?, what can we drop?). I will collate specification items.
      
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 13, 2019, 06:39:19 PM
Damn, you can manage one of my projects any time!

I'd love to help however I can. My previous posts in this thread were made when I .... well, I had no idea what I was doing, tbh. Now that I've actually used the existing Bat4Max, I have a better handle on how the process works and what the scripts actually need to do. I'm not a programmer, unfortunately, but I am fairly familiar with Blender and how it works, having used it to model and export several buildings that I rendered with 3ds Max.

I've also done a fair bit of experimentation with Cycles Render -- before I got 3ds Max, I textured and rendered several models in the hope of being able to bake textures etc and import them into gmax to render as .SC4Models. (That didn't work, sadly -- I couldn't get gmax to successfully import the baked textures and UVs, which is what one of my previous posts here was about.) But yeah, I have a pretty good understanding of how to texture models and how to fiddle with Cycles' settings to produce something that looks reasonably BAT-ish.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on February 13, 2019, 06:46:28 PM
Happy to help, Ron, if you need me  :)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 14, 2019, 07:44:43 PM
Thanks Tom and Matt, I really appreciate your replies, and will take advantage from them  ;D

Tom : given you've had more Blender experience that most of us some things that would be useful to know:

a) How do you record Macros and get them out of Blender into a script? Using macros tells us a lot about how to code the scripts later and may save us quite a bit of coding. Its a trick I use in Excel when I havent got a clue how to code something - recording actions as a macro and then editing it is easier than inventing it from scratch.

b) How do you make/automate LODs in Blender? - given in Max they're just Bounding Boxes, but we'd like better than that


Matt: given your experience in Max and deeper understanding of BATing techniques in general and you've just gone through a full install and calibration of Max2017

a) can I have a copy (zip) of the BAT gamepack folder for BAT4Max as it is in your setup, please. I dont have 3dsMax at all. You probably have the most recent completely working BAT4Max setup that I know of. I can then compare these scripts (and executables) with the gmax scripts to see exactly whats changed over the years and also figure out more of how Bat4Max works. Some snapshots of the tool Rollout would also be useful.

b) would you consider the following idea for a reference BAT. Imagine something one could put to use in-game but that anyone can primarily use to test the scripts and strengths/weaknesses of gmax, Bat4Max and Blender (eventually). Also it might help people calibrate/improve their color setups.   

It should be:

Of course, you may actually already have one you prepared earlier given a number of your recent FAR (out) productions.
   
cheers (its Friday)  Ron
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on February 14, 2019, 11:34:43 PM
Can it be we were eye witness of the birth of the ...

... Blender Gang?

()meeting()

Exciting.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 15, 2019, 05:33:30 AM
Quote from: tomvsotis on February 13, 2019, 06:39:19 PM
Damn, you can manage one of my projects any time!

Yeah no kidding, I've got a few projects laying around which would stand to benefit from such analysis!  :D

Thanks for the breakdown(s) Ron, and also thank you Jason for your input. I've wondered what those handlers were exaclty. Also hadn't thought about the positioning issue when rotating models, and given the other issues you mention it may indeed be more beneficial to rotate the cameras & lights instead. I'll have a look around if Blender has similar capabilities for linking objects together in a scene.  :thumbsup:

Quote from: rivit on February 13, 2019, 02:02:12 PM
The current scripts are very bound to gmax, rambling, untyped, not fully qualified and hence ambiguous to some extent - I have started (20% in) to port these (original gmax BAT) to VB.NET so that they can be at least checked for completeness by compilation. This allows me to separate script-only code from internal gmax functions which need to find Blender equivalents. Eventually it gives me a baseline protoype app shell independent of gmax which should make it easier to subsequently recode into Python (by someone who can). (Needs someone who knows the Blender API well)      

This is really excellent, because the (g)max scripts seemed a bit incoherent indeed. I figured it was my lack of knowledge regarding the scripting language and gmax in general which made it seem that way (though chunks of commented out code in a 'production' environment are never a particulary good sign . .). At any rate, while I do not know the Blender api well (yet) it's just a python library and by now I do know python well enough to make use of it.  :)


Quote from: rivit on February 13, 2019, 02:02:12 PM
However, since we need to make such a large transition in code, we are not obliged to keep all features or methods applied in gmax, nor even functional constraints as long as we produce Simcity4 compliant outputs. Hence simplification should be a catch-cry wherever we can.

Yes, this I feel is quite paramount to keep in mind for the time being as your outline, while comprehensive and feature complete, is quite daunting because of it. Yes custom lods, nightlights and tgi generation wrapped up in an automated process which executes with a simple button press would be very nice to have in the final end result. However if I understood Robin's post here correctly (http://sc4devotion.com/forums/index.php?topic=17650.msg515370#msg515370) the majority of these steps can be taken care off manually with already existing tools.

So I guess another way of putting it is I think the analysis could benefit from prioritisation of what goes into the proof of concept, or the minimal viable product, or however one wants to call the thing where a dozen manual steps are still needed to get the desired result.   

Friday indeed, plan on some tinkering time over the weekend. If it wasn't obvious already I'm the experimental starter type.  :D

~~

already got some tinkering in and have Blender auto-completion working in pycharm and found a way to run external files from within Blender. The latter is really nice as the native text editor leaves a lot to be desired, to put it mildly. Also made a github repo (https://github.com/GoldenQubicle/BAT4Blender), though the project structure is likely to change. 

Quote from: rivit on February 10, 2019, 03:01:16 PM
A question - can you set Blender up to be/work in Left Handed, XZ base plane, Y pointing up Coordinates ie in DirectX mode a la SC4, or is it Righthanded XY baseplane, Z up only?

yes that's possible, however, I'm not sure if changing the axis of an already existing blend file will affect the model, or have unwanted side effects otherwise. So perhaps it's best for now to leave it at default orientation as I reckon the majority of users and blend files available from share sites will probably use the default. And from my limited experience thus far with the exported .obj the defaults works fine when imported into gmax. 

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on February 15, 2019, 11:10:18 AM
Quote from: vortext on February 15, 2019, 05:33:30 AMHowever if I understood Robin's post here correctly (http://sc4devotion.com/forums/index.php?topic=17650.msg515370#msg515370) the majority of these steps can be taken care off manually with already existing tools.

Once we've rendered BMPs, provided the script can pass the right data along, we can make use of the same image conversion/importing process used by BAT4Max. When exporting the LODs, they will be imported into gMax (SC4 BAT), which will give our model an ID.

So if we follow the BAT4Max process, i.e. selecting that model file when rendering. It can pick up the ID from the SC4Model filename? In essence, everything is handled inside Program Files\Autodesk\3ds Max 20##\gamepacks\BAT. Once you've rendered at least one model using BAT4Max, what resides there is pretty straightforward to understand. I've attached a sample .BAT file, which uses DatCmd / fshwrite as command line tools, for reference. So long as the Blender scripts can create/run the script, and place the rendered images into a similar folder setup, all the post render stuff is taken care of. Again, I think you'd want to mimic however BAT4Max handles this process, how straightforward that is, I can't say.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on February 15, 2019, 02:32:15 PM
Quote from: rivit on February 14, 2019, 07:44:43 PM

a) can I have a copy (zip) of the BAT gamepack folder for BAT4Max as it is in your setup, please....... Some snapshots of the tool Rollout would also be useful.

I've sent you a link for the file. I've also left the output files for the last several Bats that I have shown. While none has custom LODs, they may fit the bill, or just at least be useful as a reference.

The tool rollout is relatively similar in feel to gmax. There's a couple of extra features. It's in the utilities Maxscript area:

(https://i.imgur.com/ufqHaLC.jpg)
Clicking on each header reveals further options:
(https://i.imgur.com/cd5MjmF.jpg)

(https://i.imgur.com/sx5tzI9.jpg)

(https://i.imgur.com/kMwfPSu.jpg)

Quote from: rivit on February 14, 2019, 07:44:43 PM

b) would you consider the following idea for a reference BAT. Imagine something one could put to use in-game but that anyone can primarily use to test the scripts and strengths/weaknesses of gmax, Bat4Max and Blender (eventually). Also it might help people calibrate/improve their color setups.   

It should be:

  • smallish, somewhat asymmetrical , say 3x2 textures, so as to trigger the slice and dice in the script
  • night lit using modern methods to test the day and night rendering quality, shadows, transmitted light and cast light
  • tricky enough geometry to stump default LODs and hence need custom LODs and exercise the Ray tracing well
  • exportable to gmax, Blender (I don't know what it accepts)
  • renderable in gmax, Max and Blender (we have no data to compare).
  • If its got a few snafus in textures/colours that would be good too - say sharp edges, contrasts, patterns


Here's one I just started. In the output files that I have sent you only one has transparency. So with this model I intend to give you transparency, custom LODs, foliage, both procedural and bitmap textures. The works. I can also add a 45 or 90 degree angled piece at the base if you want.

(https://i.imgur.com/YP4Mu3W.png)

Whatever you think will help.

It must be noted that the way I model is really peculiar to me. Apart from Alex - xannepan - I don't know anyone else who uses these methods, which, for square buildings is mostly just lines/splines. Only when I need to do anything more complex or organic do I collapse to mesh or polygons. But both of which slow me down intensely (we are talking the difference between seconds to model, say a wall, with lines and days to model it with polygons).

Later I can collapse these if we run into issues. But it is something to bear in mind.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 15, 2019, 04:06:11 PM
Working Backwards

Thankyou Matt - package received - a quick look suggests that BuildingMill has grown by 1000 lines of code (65%) to 2500 lines. Ironically many of the 128 changes are removing old commented code so the change is actually bigger.  The camera Rig is almost the same - just 31 changes. Pics of rollout are now in the archive.

Progress report: I have found that about 500 lines of code in the old gmax scripts are either dead or unused.

The building is quite the things I had envisaged. My logic is such - if Blender can't actually match Bat4Max in outputs then we need to find out where, how to compensate and make a decision if its worth it. Same goes for handling imports/exports. To get there we need something to compare with. The gmax bit is so that we have target files with which to compare the outputs into SC4, as Bat4Max produces files that are 2x larger than gmax to start with.

This should also give the creator types something to get their teeth into as far as render settings etc. So thank you so far - brilliant, and I'm sure the Engineering Dept will find good use.

TODO: for someone - nail down what Blender/3dsMax/gmax/ (Reader takes .3ds) can actually exchange with each other. Not just file format - but content within - including textures, uvw etc., settings, any exotics like splines, NURBs, lights, scenes.  Document (with examples) and make available. This is independently needed as useful info, and crucial to data exchange between tools - esp. for testing.

@Robin (MGB) Yes that why the synch point is Render and LOD. If the first part works, and the existing facilities can be used to get outputs into SC4,  then we can bootstrap the second later. Having said that getting the renders is still a big unknown.

@Fantozzi - like it - next you'll be telling me our theme tune should be "Life in the Fast Lane"

@vortext
Thanks for all off your additions/alterations, the name of this Product and the GitHub Repo. It's good news you're getting good with python as we're definitely going to need it, while for me its readable but very weird. And yes I'm very conscious of this not becoming a Ben Hur - the faster we get something working the more likely we are to finish it or conclude its not worth it.

btw Our Oz SBS TV broadcaster is showing the original Monty Python series (this time its in color!), it being 50 years ago today, that they first surfaced. I was 14 at the time, and I'm amazed at how much I still remember of it - it clearly seared into my brain







Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 16, 2019, 04:34:06 AM
Quote from: rivit on February 14, 2019, 07:44:43 PM
Thanks Tom and Matt, I really appreciate your replies, and will take advantage from them  ;D

Tom : given you've had more Blender experience that most of us some things that would be useful to know:

a) How do you record Macros and get them out of Blender into a script? Using macros tells us a lot about how to code the scripts later and may save us quite a bit of coding. Its a trick I use in Excel when I havent got a clue how to code something - recording actions as a macro and then editing it is easier than inventing it from scratch.

Hmmm. I don't know that you can *record* macros in the Word/Excel sense, but here's the next best thing: every command you enter in Blender is logged in the program's internal console. So you can effectively "record" the code for your actions by performing a few actions and then checking what the code for them looks like in the console.

The console is viewable by switching one of the viewports to "Info" mode. You can see it demonstrated here (https://www.youtube.com/watch?v=zMxkU350SaE), from about three minutes in. Note that the technique he uses at about 3:25 - dragging the divider immediately above the main viewport downward to reveal the info viewport -- doesn't seem to work in Blender 2.80. Instead, the easiest way I've found is, using the default "Layout" workspace, drag the animation viewport (at the bottom of the screen) upwards:

(https://i.imgur.com/Jt3AUx7.jpg)

...and then change it to "Info":

(https://i.imgur.com/a0rMRIi.jpg)

Then, as you go about your modeling, you can see the Python syntax for what you're doing. As an example, here's the log for selecting the default cube, scaling it 200% along the x axis, entering edit mode, and then applying an edge loop along the x axis - as recorded in the console:

bpy.ops.object.select_all(action='TOGGLE')

bpy.ops.transform.resize(value=(2, 1, 1), constraint_axis=(True, False, False), constraint_matrix=(1, 0, 0, 0, 1, 0, 0, 0, 1), constraint_orientation='GLOBAL', mirror=True, proportional='DISABLED', proportional_edit_falloff='SMOOTH', proportional_size=1)

bpy.ops.object.editmode_toggle()

bpy.ops.mesh.loopcut_slide(MESH_OT_loopcut={"number_cuts":1, "smoothness":0, "falloff":'INVERSE_SQUARE', "object_index":0, "edge_index":11, "mesh_select_mode_init":(True, False, False)}, TRANSFORM_OT_edge_slide={"value":-0, "single_side":False, "use_even":False, "flipped":False, "use_clamp":True, "mirror":True, "snap":False, "snap_target":'CLOSEST', "snap_point":(0, 0, 0), "snap_align":False, "snap_normal":(0, 0, 0), "correct_uv":True, "release_confirm":False, "use_accurate":False})


This code can then be entered into the program's internal text editor and saved/exported as a Python script. (Note that all such scripts need the initial command "Import bpy", which imports the Blender Python module.)

Hope that helps?

Quote from: rivit on February 14, 2019, 07:44:43 PMb) How do you make/automate LODs in Blender? - given in Max they're just Bounding Boxes, but we'd like better than that

Bounding boxes are easy enough, and manually making LODs/exporting them as .3ds files to gmax is also pretty straightforward and do-able. I haven't tried automating it, but I can't imagine it'd be that difficult to interrogate the program to get the bounding box of a scene and then draw an object to the scale of that box. I can certainly look into how to do this.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 16, 2019, 04:23:22 PM
Tom,
  thats terrific! It might be a bit clunky but it means that we don't actually have to imagine all of the code we need - just by manually doing the actions we need we can get a lot closer than we think. I'm sure that's going to get a lot of use and be a huge time saver.


Progress Report: Having taken a somewhat dangerous dive into the BAT4Max scripts (I didn't know how deep they'd be) it turns out that almost all of the vast code change that CP did was to handle the fact that the MAXIS executables no longer worked in 3dsMax. A couple of others fiddled with it (quite safely) and then Simfox added/changed things to deal with his TruNight and some other snafus 3dsmax threw up (over) itself.

All up this has made a significant (albeit working) mess of the BuildingMill - the Render routine is HUGE -  and I think sticking to gmax principles and finding out what Blender can and can't do is the way to go. However I will still plough through the Bat4MAx stuff to make sure I've seen it all and understand it as best I can.

Nuggets to come from this are the Sun,Sky and Exposure Settings Simfox defined for truNite variations
To be found here https://1drv.ms/b/s!AphvaLJG-tShg4wkcDAenWlkTsbRbA (https://1drv.ms/b/s!AphvaLJG-tShg4wkcDAenWlkTsbRbA) is a Pdf of a preliminary pretty print of the relevant details from those routines.

TODO: Art types for whom this means something are requested to see if/how Blender does these things and incorporate them in the code base. The Recording mechanism Tom outlined above will be quite handy.

TODO: There are some questions that need answering about Sun/Sky Colors and Sun Positioning and these answers might lead to changes in the Bat4Max scripts.

TODO: can someone take the Sun/Sky/Camera/Target objects in a Bat4Max and a gmax scene and dump/print out/record all of their attributes so we can compare these properly. We need to know they are absolutely the same/equivalent and it would be advantageous to record/set these in the scripts rather than have them float around in some scene where no one can see them.

TODO: can Null45 who I think made the FSHwrite.exe encoder tell us whether there were any fancy things in there or if it was all vanilla encoding. All the FSH made are DXT3 when the Day ones are DXT1 in SC4 so are 2x larger than necessary.  I can also do this with the code I have for GoFSH and can also now do VariableNight but would like to know if there are any "specials" in this code to worry about re truNite.

TODO: Who wrote/found the Datcmd.exe that makes the .SC4Model up? What are the command line parameters needed to run this?  (MGB may have answered this already).  Is source code available?

TODO: Still havent worked out how the S3Ds are encoded. Presumably I will find out once I get through BuildingMill.

TODO: What platforms are we intending to support Windows/Linux/Apple?  Current stuff is very Windows-centric - it affects .exes more than anything - in principle it could all be python (which would need to be written from scratch) but we have existing code in .net which could be ported (read largely recompiled) to .Net Core. Also there's quite a bit of java code about from earlier contributors. Basically we need to be able to do Bitmap manipulation (which is OS dependent), FSH encoding, S3D encoding, DAT assembly. Everything else seems inside the 3DTools.


  Sorry but TODOs will stack up for a bit - we are starting to get into the nitty-gritty of how things are made. This is a good thing.
   Ron
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 16, 2019, 04:54:29 PM
Fitting a LOD was on my radar for a while already and as far as I can tell there isn't an straighforward way to get a bounding box which encompasses all the objects in scene. Few answers I found online boiled down to; get the bounding box for all objects and work it out from there. So that's what I did today, and actually just finished it.

(https://cdn.discordapp.com/attachments/476300301872398349/546490405533909006/blenderbat_2.gif)

Perhaps there's another way and it'll be proven obsolete but hey, at least I'm a little more familiar with the Blender api. In terms of code it's less than a 100 lines atm (here in case interested  (https://github.com/GoldenQubicle/BAT4Blender/blob/master/LOD.py)), though I made some assumptions to speed things up.
Notably starting from the world origin to find the LOD dimensions. So say for example a scene contains a single offset object (e.g. the x dimensions for the bound box are -12,-5) the fitted LOD will extend towards origin anyway (-12,0) and thus the LOD width will not be correct. I figured it an acceptable compromise for the time being.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 16, 2019, 07:22:08 PM
Looks pretty functional to me. If you initialize the boundingbox with large negative maxes and positive mins then it will  encompass only the extent of the object by the time you've fed everything through it - even if it was only one point.

Is there no Blender function to get the Object Extent? == BoundingBox, or its max or min dimensions?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 16, 2019, 10:19:39 PM
Quote from: rivit on February 16, 2019, 04:23:22 PM
Tom,
  thats terrific! It might be a bit clunky but it means that we don't actually have to imagine all of the code we need - just by manually doing the actions we need we can get a lot closer than we think. I'm sure that's going to get a lot of use and be a huge time saver.

Yay! I'm glad it's helpful :)

Quote from: rivit on February 16, 2019, 04:23:22 PM

TODO: What platforms are we intending to support Windows/Linux/Apple?  Current stuff is very Windows-centric - it affects .exes more than anything - in principle it could all be python (which would need to be written from scratch) but we have existing code in .net which could be ported (read largely recompiled) to .Net Core. Also there's quite a bit of java code about from earlier contributors. Basically we need to be able to do Bitmap manipulation (which is OS dependent), FSH encoding, S3D encoding, DAT assembly. Everything else seems inside the 3DTools.

Considering that Blender and Python are both cross-platform, it'd seem a shame not to support Linux and macOS, esp since they're already pretty poor relations as far as SC4 goes. The whole reason I started w Blender was that I only had a Mac, and thus couldn't run gmax/3ds Max. I think not spreading the ability to create BATs via Blender as widely as possible would be a mistake; we have an opportunity here to extend the BAT-ing community to people who might previously not have been able to join it. My 2c, anyway.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 17, 2019, 02:59:45 AM
Quote from: rivit on February 16, 2019, 07:22:08 PM
If you initialize the boundingbox with large negative maxes and positive mins then it will  encompass only the extent of the object by the time you've fed everything through it - even if it was only one point.

Yes that would work as well, though imo the optimal solution would be introducing an additional step wherein a single bound box corner coordinate (in world space) is compared to 1) the other corners belonging to that bound box and 2) all other corners in the collection to figure out the min & max for x,y,z. That way there'd no need to initialize / make assumptions about the initial LOD size to begin with, as it would follow from said comparisons.

Quote from: rivit on February 16, 2019, 07:22:08 PM
Is there no Blender function to get the Object Extent? == BoundingBox, or its max or min dimensions?

So yeah, I wanted to show you this but apparently there's no bound box display yet in 2.8 ..  (https://devtalk.blender.org/t/about-cycles-in-2-79-and-2-8-and-bound-bounding-box/3154/2). Anyway, ample examples online of course, shown below the bound boxes as Blender fits them around objects.

(https://i.stack.imgur.com/201Dp.png)

Note the bound box location, dimensions and rotations are all in local space. As far as I've been able to tell there're no methods available to get the world space dimensions of a bound box direclty. Hence the LOD script first calculates the corners of a bound box in world space, and then iterates over all corners of all bound boxes.

Also Tom, are you able to export models as 3ds in version 2.8? If so, how exaclty?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 17, 2019, 03:37:24 AM
Quote from: vortext on February 17, 2019, 02:59:45 AM
Also Tom, are you able to export models as 3ds in version 2.8? If so, how exaclty?
Hmm, I hadn't tried in 2.8, and it seems there's no native export function. But if you want to export LODs to gmax, i think you're better off doing it as an .dae, as outlined by Fantozzi earlier in this thread:

QuoteIf I export 3ds from Blender, most of the time it doesn't work in gmax for me ("improper format" or so).

You may try this:

a) download this converter written with gmax in mind (by a ms flight simulator fan): https://www.scenerydesign.org/modelconverterx/
b) from Blender export to *.dae format
c) import the dae into the model above converter and export as 3ds.

.....but I'm away from my Windows PC at the moment, so I can't check that it works :-/
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 17, 2019, 05:59:24 AM
Hm yeah, shame there's no native s3d export (yet?), thought perhaps you had stumbled upon an alternative. I tried the .obj route previously (described here (http://sc4devotion.com/forums/index.php?topic=17650.msg530121#msg530121)) which worked seemlingly okay but did require an additional import script for gmax as well. Guess automating LOD export should be put off for the time being untill the pros & cons of various methods are fleshed out.

Also regarding LOD fitting came to realize (while doing some cleaning  :D) a far simpler solution would be to just initialize the dimensions from the very first corner vertex in the list (instead of world origins, or super large values, let alone comparing all corners against one another), and proceed from there. And indeed that works fine, script has been updated (https://github.com/GoldenQubicle/BAT4Blender/blob/master/LOD.py) to handle offset objects and added small check to keep from adding LODs to the scene.


Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 17, 2019, 09:42:42 PM
Quote from: vortext on February 17, 2019, 05:59:24 AM
came to realize (while doing some cleaning  :D)
An excellent time to solve any sort of thinking problem... which reminds me I should probably do some.

I've continued to plough on through code (O so repetitive code - very little refactoring has been done) so don't have a lot to mention except that the exporting of the LODs seems a rather unnecessary evil - I read somewhere that its to 'not break the Maxis EULA' that we export to gmax and make up the s3ds from there. Why?

Call me a Noob %confuso but if all we are doing is importing/creating S3Ds from slices of a LOD mesh projection after mapping UV coordinates to textures and then putting both S3ds and FSH into a DAT, then why is does that break the EULA?  "$Deal"$ Reader can make S3Ds, and do packaging of DATs, just as well as a number of other tools manipulate dats, s3ds and FSH. Do they not break the EULA? Whats special here? We are not reverse engineering any Maxis executable code in this - we have reverse engineered the data.  I am reminded of the Unisys GIF patent...

Even more oddly, it seems the BAT4Max scripts still do all of the work that mapping and slicing entail but so far I can't find anywhere they are output, so its busy work that wasn't turned off - and a quite meaty part of the processing.

I'm also somewhat unimpressed with the Output file conventions/folders/names - leads to a lot of work internally in the scripts and is awkward to use externally. Maybe I'm just getting tired  &mmm

Also a pita about the no .3ds export - however code for writing them (very simple ones) does exist in NetWorld I expect, and is probably not that hard although messier than .objs which are very elementary. I have (written) code to read them .3ds and obj many years ago.  If I understand the process we really only need points and mesh triangles - everything else is deduced.

TODO: verify what image formats are output by Blender from a Render - in particular we would like 32bit ARGB bmps, but 24bit RGB will do but then we'll need two - one for color, one for alpha. PNGs are serviceable but much harder to manipulate natively in code - Windows does load them more-or-less correctly but doesn't save them well.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 17, 2019, 10:34:17 PM
Quote from: rivit on February 17, 2019, 09:42:42 PM

I've continued to plough on through code (O so repetitive code - very little refactoring has been done) so don't have a lot to mention except that the exporting of the LODs seems a rather unnecessary evil - I read somewhere that its to 'not break the Maxis EULA' that we export to gmax and make up the s3ds from there. Why?

Huh. I always wondered about this, too. It's particularly annoying in this context as it single-handedly undermines the whole cross-platform nature of the enterprise: SC4 is cross-platform (at least until Apple completely discontinues support for 32-bit apps, but that's a separate discussion), Blender is cross-platform, Python is cross-platform, but gmax... isn't. It's prob worth finding the EULA in question and seeing what it actually says.

Quote from: rivit on February 17, 2019, 09:42:42 PM

Also a pita about the no .3ds export - however code for writing them (very simple ones) does exist in NetWorld I expect, and is probably not that hard although messier than .objs which are very elementary. I have (written) code to read them .3ds and obj many years ago.  If I understand the process we really only need points and mesh triangles - everything else is deduced.


Just for completeness' sake: there IS a .3ds export plug-in, but it's, well, a plug-in. It hasn't been updated for 2.80, but it did work (albeit not super-reliably) in 2.79. I have no idea how much work would be required to make it work in 2.80, but it does exist.

Quote from: rivit on February 17, 2019, 09:42:42 PM

TODO: verify what image formats are output by Blender from a Render - in particular we would like 32bit ARGB bmps, but 24bit RGB will do but then we'll need two - one for color, one for alpha. PNGs are serviceable but much harder to manipulate natively in code - Windows does load them more-or-less correctly but doesn't save them well.

Blender's export settings for BMP are... pretty minimal:

(https://i.imgur.com/aRM9Hdr.jpg)

...and, as per the manual (https://docs.blender.org/manual/en/dev/data_system/files/media/image_formats.html), BMP export is 8-bit only. (That manual page is, afaik, the most up-to-date reference as to what formats Blender supports.)

PNGs are somewhat more flexible:

(https://i.imgur.com/L5Do4Zz.jpg)

This is by no means my area of expertise -- i am good at photo editing, but not at manipulating image files via code -- but if we want lossless files w high colour depth, can we use TIFFs? (Or even TGAs?) Blender also supports 32-bit EXRs, but i know nothing at all about that format.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 17, 2019, 10:37:10 PM
Or, wait, does "8-bit BMP" mean 8 bits per channel, i.e 24-bit colour but no 8-bit alpha channel? This stuff has always confused me  ()what()
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 17, 2019, 11:32:22 PM
Yes, 8 bits per channel - so
the BMPs look to be BW = 1bit, RGB = 24bits  - BW for 1 bit alpha, RGB for color
and the PNG 1bit BW ?, 24bit RGB or 32bit RGBA
   
really wouldn't want to go TIFFs they can be huge, but TGAs are OK.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mgb204 on February 18, 2019, 01:38:19 AM
Quote from: rivit on February 17, 2019, 09:42:42 PM
Call me a Noob %confuso but if all we are doing is importing/creating S3Ds from slices of a LOD mesh projection after mapping UV coordinates to textures and then putting both S3ds and FSH into a DAT, then why is does that break the EULA?  "$Deal"$ Reader can make S3Ds, and do packaging of DATs, just as well as a number of other tools manipulate dats, s3ds and FSH. Do they not break the EULA? Whats special here? We are not reverse engineering any Maxis executable code in this - we have reverse engineered the data.  I am reminded of the Unisys GIF patent...

I think the real issue is not so much whether we'd actually be in breach of the EULA, but whether EA would consider it a breach. In times past, EA was certainly keeping an eye on modding developments and it was probably just a case of being careful to avoid their wrath.

In the here and now, I'd imagine EA probably doesn't care and it's highly unlikely they'd notice such a thing, since I can't imagine SC4 modders are on their radar. But, should EA ever decide to send in their team of lawyers, right or wrong doesn't come into it, who could fight them? Simply in a financial sense, rather than factually or morally. That's ultimately the real problem, if EA decide they don't like it or think something is in breach, there wouldn't be an awful lot anyone here could do about it. It would have to be taken down from any sites or they'd sue us to oblivion very quickly.

Personally speaking, I agree, we're not re-using Maxis/EA code, so there is no actual breach.

Quote from: tomvsotis on February 17, 2019, 10:34:17 PM
Huh. I always wondered about this, too. It's particularly annoying in this context as it single-handedly undermines the whole cross-platform nature of the enterprise: SC4 is cross-platform (at least until Apple completely discontinues support for 32-bit apps, but that's a separate discussion), Blender is cross-platform, Python is cross-platform, but gmax... isn't. It's prob worth finding the EULA in question and seeing what it actually says.

Perhaps it won't make me popular, but SC4 is not multi-platform. Sure you CAN play it on Linux and a consistently borked Mac version exists. But few of the modding tools, including none of the official ones, ever worked outside of Windows.

The enterprise as I understand it, is not to make modelling possible on other platforms, although it would be a nice fringe benefit. The idea was simply to provide a foolproof method that would allow us to continue making quality models for use with SC4. The key issues being that 3DS Max and it's ever-changing nature, coupled with a lack of anyone to maintain the scripts for BAT4Max, have created a ticking time-bomb. Short on everyone returning to older versions of 3DS Max, which is very expensive and let's be honest, almost everyone is using the Student Licence to obtain 3DS Max for free. We'd be back to having to use gMax (SC4BAT) once more to create/render models, a huge step backwards. With Mental Ray being removed from 3DS Max recently, sans a re-write of the BAT4Max scripts, in a few years we're basically boned here.

I said when this was first discussed, one key benefit of Blender, as an open source application, we can always be sure a working version will be obtainable. Because we wouldn't suffer the forced-upgrade path that Autodesk are able to exert on everyone. So even if Blender outgrew the scripts for SC4, we wouldn't be deprived of a working solution as we could use an older version that did work.

I really feel that this should remain the #1 priority, ensuring a future-proof way of making quality models for SC4. If we can make it work on Mac/Linux, great, there is literally no downside to that. It could even transpire that this is something that works without expending any effort. But, if that is going to require any significant extra development time and resources, it should be very much a secondary or low priority feature. Why?, well simply because almost everyone making models already uses Windows. Just like almost everyone playing SC4 does. It makes no sense to divert resources to support a small percentage of potential users, in the first instance we should focus on what will work for the many, not the few.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 18, 2019, 02:57:01 AM
Quote from: mgb204 on February 18, 2019, 01:38:19 AM

Perhaps it won't make me popular, but SC4 is not multi-platform. Sure you CAN play it on Linux and a consistently borked Mac version exists. But few of the modding tools, including none of the official ones, ever worked outside of Windows.

True, although I guess the point I'm making is that one very appealing feature of the very nature of the solution we're looking at — i.e. Blender rendering + Python scripting — is that it IS cross-platform, and it'd be a shame to then tie it back to gmax if it transpired that doing so wasn't necessary. You're correct that the modding tools have never worked outside of Windows — believe me, I've tried 'til I was blue in the face to try to get PIM-X to work in Wine — but I wouldn't entirely discount the Mac and Linux community as far as playing the game goes. And anyway, BATing is a quite separate enterprise; as with any modding activity, there are some people who enjoy it more than the game itself. Ultimately, for the long-term health of the community, we should be trying to extend the capability to create content to SC4 as widely as possible.

Anyway, I think we're basically saying the same thing — that the priority should be getting an equivalent to Bat4Max working in Blender, and that if as a result cross-platform compatibility is maintained, that is, as you say, ultimately a fringe benefit. But I don't think cross-platform compatibility should be thrown out unnecessarily, either, so if we can find an alternative to tying the whole thing back to LOD rendering in gmax, we should.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on February 20, 2019, 12:16:49 AM
Quote from: rivit on February 15, 2019, 04:06:11 PM

The building is quite the things I had envisaged. My logic is such - if Blender can't actually match Bat4Max in outputs then we need to find out where, how to compensate and make a decision if its worth it. Same goes for handling imports/exports. To get there we need something to compare with. The gmax bit is so that we have target files with which to compare the outputs into SC4, as Bat4Max produces files that are 2x larger than gmax to start with.

This should also give the creator types something to get their teeth into as far as render settings etc. So thank you so far - brilliant, and I'm sure the Engineering Dept will find good use.


So I've finished the Engineering department. I've uploaded the game version on the LEX and will prep the model files so they can be dissected.

(https://i.imgur.com/E5nT0y4.jpg)

It has
* custom lod (just simple - the LOD is rotated 45 degrees, but it will be easy to troubleshoot rather than a really complex one)
* bitmap materials
* various 3ds procedural materials
* transparency + reflectivity
* basic + complex shapes (eg the foliage)
* standard primitives, extruded lines and meshes
* night lights + glare (i don't normally do glare)
* Day, MN and DN

So we should have most of the elements there to make a start.

I'll also take a further look at your todo items over the next few days to see if I can give any meaningful help there...
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 20, 2019, 04:57:53 AM
Matt,

  that's absolutely a great test piece, totally fitting. It reminds me so very much of the Engineering School building I attended at Auckland University in the 70s. I suspect this will be a quite stiff test to meet as you've managed to pack quite a bit in there. Thank you very much, and challenge one for someone will be seeing how to get it into and then rendering it in both gmax (probably wont go) and Blender. I truly hope a few people will give it a go - even if just to try Blender - as this is real stuff to get on with.

My end, I've now transcribed the gmax scripts to VB.NET and have only 81 compiler errors remaining from the 8000 odd I started with 10 days ago. Even if it compiles that doesn't mean it works, of course, but I am nearly through phase 1 (unravelling how it does its thing) and I have started to sketch out the structure of a plugin - scaffolding classes and UI.

Still need to parse CP's BuildingMill for all the special Night Window stuff - which having never used, I don't understand at all from a process perspective. Found one or two coding errors in the gmax scripts along the way. Overall though the gmax code is reasonably tight within the context of scripting, and the prime cause of code 6 is probably unsuffficient or scrambled workspace/memory. I suspect its more likely if the LODs are trickier/large.

Overall its been interesting, if tiring, to comprehend the 'spaghetti nature' of the meaning of the code caused by having a scripted language with 2 or 3 ways of referencing the same thing, everything implicitly typed and much global and declared on the run. Unravelling it and refactoring into clear classes without a lot of app dependencies is not going to be entirely trivial.

Research suggests we are going to end up exporting PNGs and OBJ files from Blender as support for BMP is thinner (maybe OK) and the OBJ format seems the most evolved of the export formats (although .3ds can be done). The Blend file format seems inpenetrable. Converting these to BMP and .3ds for LODs isn't that hard to do if necessary so we should be able to do the gmax trick at first. I would like to aim for removing gmax from the workflow in the longer run.

In other news I read that Blender uses a XYZ axes with Y facing forward, Z up, 3dsmax is XYZ with Y facing backwards, Z up, and SC4 is XZY with Z forwards and Y Up. This affects thinking about transforms and other maths operations we use when comparing operations with 3dsmax. It also seems to have a fairly rich (if less evolved than 3dsmax) set of mesh operators. The renderer also appears to be reasonably advanced and can take advantage of Graphics cards for ray tracing. I will be very interested to see what its capable of.


Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 22, 2019, 05:21:15 AM
Sounds great Ron, if a bit tedious work but well worth it if eventually we can cut gmax out of the render equation.  :thumbsup:

Quote from: rivit on February 20, 2019, 04:57:53 AM
In other news I read that Blender uses a XYZ axes with Y facing forward, Z up, 3dsmax is XYZ with Y facing backwards, Z up, and SC4 is XZY with Z forwards and Y Up. This affects thinking about transforms and other maths operations we use when comparing operations with 3dsmax.

Yeah this tripped me a up a bit when it came to making the camera object configurable using the game angles. However apart from a few minor quirks it does work as intended now (here (https://github.com/GoldenQubicle/BAT4Blender/blob/master/Camera.py)) and won't be too difficult I reckon to hook it into gui later on.

Also found this method (https://docs.blender.org/api/blender2.8/bpy.types.Object.html#bpy.types.Object.camera_fit_coords) which, given a list of coordinates, will return the orthographic scale to fit all the coordinates in the camera view. It works as advertised, and takes into account camera nicely (i.e. returns different scale depending on rotation & zoom). However, it does not return offsets so it remains to be seen what use, if any, it will have in practice.


Quote from: rivit on February 16, 2019, 04:23:22 PM
TODO: can someone take the Sun/Sky/Camera/Target objects in a Bat4Max and a gmax scene and dump/print out/record all of their attributes so we can compare these properly. We need to know they are absolutely the same/equivalent and it would be advantageous to record/set these in the scripts rather than have them float around in some scene where no one can see them.

Yeah I'm interested in this as well, would be much appreciated it we can get more info on the cameras, world settings and all related 'global' properties.

~~~~

The issue of wrangling the camera offsets has become managable by this super helpful method  (https://docs.blender.org/api/blender2.8/bpy_extras.object_utils.html?highlight=world_to_camera_view#bpy_extras.object_utils.world_to_camera_view) hiding in sidelines. Not quite sure if the orthographic scale is applied correclty (actually pretty sure it's not) but at least it provides a way to get the LOD in the upper left corner of the camera view consistently with a fixed number of pixels between it and the edges (janky code here (https://github.com/GoldenQubicle/BAT4Blender/blob/master/Renderer.py))

(https://cdn.discordapp.com/attachments/476300301872398349/548591180066324491/blenderbat_4.gif)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: c.p. on February 22, 2019, 09:37:48 AM
Quote from: rivit on February 17, 2019, 09:42:42 PM
I've continued to plough on through code (O so repetitive code - very little refactoring has been done) so don't have a lot to mention except that the exporting of the LODs seems a rather unnecessary evil - I read somewhere that its to 'not break the Maxis EULA' that we export to gmax and make up the s3ds from there. Why?

I did the original bat4max scripts, and I can assure you that EULA considerations had nothing to do with it, at least not for the original version of bat4max.  I had no programming background (Refactoring? what's that?  :D ), so was restricted to what I could do via maxscript.


Quote from: rivit on February 17, 2019, 09:42:42 PM
Even more oddly, it seems the BAT4Max scripts still do all of the work that mapping and slicing entail but so far I can't find anywhere they are output, so its busy work that wasn't turned off - and a quite meaty part of the processing.

If I remember correctly, the LOD geometry is used in 3dsmax to calculate the size and position of the rendering.  (The slicing probably isn't used though.)


Quote from: rivit on February 20, 2019, 04:57:53 AM
Still need to parse CP's BuildingMill for all the special Night Window stuff - which having never used, I don't understand at all from a process perspective.

If I can help with specific questions about the night windows code, ask away.  I'm not sure anyone but me has used the "night library" feature, so that might not be worth devoting time to.  The general point of the pixel-by-pixel post-render processing was to create an alpha channel for the night render, so only the nightlighted areas show up in-game.  (Which incidently makes them compatible with both "Maxis Night" and customs night mods).  That is also why a third render was done for certain types of nightlighting.  If you need to know what the specific gbuffer channel info is that the code is pulling from the render, I can figure that out for you, but a lot of that might be unneccessary/irrelevant in Blender.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on February 22, 2019, 01:21:51 PM
Awesome to see you here, CP...your 3ds mastery will be invaluable...way better than my non-existent script knowledge, that's for sure.  :thumbsup:

Quote from: vortext on February 22, 2019, 05:21:15 AM


Quote from: rivit on February 16, 2019, 04:23:22 PM
TODO: can someone take the Sun/Sky/Camera/Target objects in a Bat4Max and a gmax scene and dump/print out/record all of their attributes so we can compare these properly. We need to know they are absolutely the same/equivalent and it would be advantageous to record/set these in the scripts rather than have them float around in some scene where no one can see them.

Yeah I'm interested in this as well, would be much appreciated it we can get more info on the cameras, world settings and all related 'global' properties.

~~~~



Okay, so here's what I have so far: anyone please feel free to correct me.

In both gmax and 3dsmax "Zpx = 4000" is a constant between both programs in the camerarig scripts etc. In the gmax scriptutils, this is listed:

--------------------------------------------------------------
-- Get a ray corresponding to a pixel on the screen
-- only works with ortho cameras
-- Obsolete
--------------------------------------------------------------
fn FnSafeScreenToRay pixelPt = (
   local p0, p1
   local projXform
   local resultRay = stRay3D()
   
   --gw.setTransform (matrix3 1) -- paranoia
   
   projXform = inverse (viewport.getTM()) -- xform takes view space to world
   
   -- get point behind pixel in viewspace
   p0 = mapScreenToView pixelPt 4000.0 -- 4000 was empirically determined to be what mapScreenToWorldRay uses
   -- transform point into world
   resultRay.pos = FnTransformPt p0 projXform
   
   -- get point a little further behind pixel in viewspace
   p1 = mapScreenToView pixelPt 3990.0
   -- transform point into world
   p1 = FnTransformPt p1 projXform
   
   resultRay.dir = p1 - resultRay.pos
   
   normalize resultRay.dir
   
   return resultRay
)



That may be useful to you, I don't know....

As for camera/lighting here's some visuals on what I have taken from the systems. Note down the bottom of each pic is the XYZ co-ordinates of each element:

A) GMAX. I'm really rusty with the commands, so there could well be more to extract, but it will take some time for me to remember all of the commands (assuming they're needed, the lighting rig in gmax is very old....)

(https://i.imgur.com/Ne814qp.jpg)
Lighting rig. Each individual light (8) has the same co-ordinates, which I find odd...I'm probably missing something

(https://i.imgur.com/Y0WaO6G.jpg)
Camera Array

B) 3dsMax:

(https://i.imgur.com/XO8fjjd.jpg)

(https://i.imgur.com/uoFSdZA.jpg)

(https://i.imgur.com/bchY9IY.jpg)
IES Sky, IES Sky Day Object Properties, IES Sky Target Object Properties

******************

(https://i.imgur.com/gitvHiy.jpg)

(https://i.imgur.com/AUaLcRQ.jpg)
MR Sun & MR Sun Object Properties

******************
(https://i.imgur.com/jFRbsma.jpg)

TB2 Camera Handle properties


Hopefully this assists, let me know if anything else needed.

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 22, 2019, 04:26:03 PM
@CP So very pleased to see you in this thread, you're like a missing master for me - I see your name everywhere on things I like and have in SC4.

I'm very impressed with the stuff you added into the building mill - given the awful environment to work with and the quirky script language I'm amazed that anything that big was built and worked. I had "sort of deduced" the alpha fiddling was to do with producing a variable alpha to fit with Maxis mode, but the depth of code still has me as a rabbit in the headlights. Must have taken ages to get it to work as intended. In fact, because I couldn't follow that code properly it made me find another way to make alphas for variable nights. I will ask if I have questions thankyou.

@Matt the stuff associated with the CameraHandle seems to stop us seeing info for each individual camera - I can't figure out a way to break that group into individual items. While we do have camera details (I can't remember how I found those back in 2009, so they need to be verified) we don't know the camera target in particular.

@vortext - glad to see you've found those little improvements - bit by bit getting closer to getting the views right.  The 'orthographic scale to fit all the coordinates in the camera view' is essential to framing the views and determining the end UV coordinates per sub mesh so that will be rather handy. In the BAT scripts this is then modularised into multiples of 256 for zoom 5, and smallest power of 2 to fit for the other zooms for the rendertexture sizes.

I have now got a compiled version in VB.NET.  It doesn't do anything useful but this has allowed me to identify/abstract those functions coming from the "rendlx" plugin, and from gmax/3dsmax in general. A pretty printed pdf version is available here. https://1drv.ms/f/s!AphvaLJG-tShg4wbSqa1_8cIZ4sy2w (https://1drv.ms/f/s!AphvaLJG-tShg4wbSqa1_8cIZ4sy2w). It's also let me find those functions in the code that are dead i.e. not used. About 500 lines. I have started the design process on a new plugin structure - main goal will be to keep Blender code in one place so that the dependencies are easier to manage - we'll see how that works out...  :-\

  ASIDE: Refactoring is a process by which repetitive/verbose/complex code is abstracted into simpler functions to improve code quality by reducing the number of places mistakes could occur.  These new chunks can then be used everywhere - eg. if I need a single file name everywhere instead of writing it out in code each time in full I might refactor that into a constant which I then use everywhere. That way I can mistype it only once.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 23, 2019, 04:23:23 AM
Quote from: rivit on February 22, 2019, 04:26:03 PM
The 'orthographic scale to fit all the coordinates in the camera view' is essential to framing the views and determining the end UV coordinates per sub mesh so that will be rather handy. In the BAT scripts this is then modularised into multiples of 256 for zoom 5, and smallest power of 2 to fit for the other zooms for the rendertexture sizes.

Regarding the textures sizes, played around a bit this morning with different LOD sizes and I noticed gmax will actually produce smaller textures if the LOD is equally smaller as well. For instance an 8x8x8 cube results in a 128x128 px texture for zoom 5. Likewise a 2x2x2 cube renders to a 32x32 px texure for zoom 5 (and zoom 1 to a whopping 4x4 px  :D).

Consequently it seems the orthographic scale can be better thought of as a orthographic constant. That is to say, if I calculate the orthographic scale for the 8x8x8 cube following the procedure followed here (http://sc4devotion.com/forums/index.php?topic=17650.msg530262#msg530262), it results in ~23.5. Plugging that into Blender, however, things seemed off. Setting the gmax render as the background image for the camera it quite clearly is not correct, as shown below.

(https://i.imgur.com/2DQqa0v.png)


Now taking the orthographic scale found for the 16x16x16 cube (~36.5), dividing it by 2 and applying the result as orthographic scale for the 8x8x8 cube and things look a lot better.

(https://i.imgur.com/OVsT9EQ.png)

The jagged edges from the gmax render are still visible, however, it's a much better fit overall.

So it seems the render procedure for Blender will need to utilize a 'constant', pre-determined orthographic scale which is halved / doubled depending on whether the orthographic scale for the whole LOD is smaller or bigger than said constant. And of course the output dimensions need to be set accordingly as well.

Also, on the subject of the smallest possible model. I've exported a LOD measuring 0.25 x 0.25 x 0.25 meters. Gmax renders without a problem, sc4pim shows the model just fine as well and sets the correct occupant size. All that said the model just won't show up ingame. Not as a building, nor as a prop, and not even when zooming in all the way with the sizeoff command. It appears there is somekind of size limit inherent to the game graphics rendering engine itself?


Thanks Matt for the info. Regarding the 8 lights all having the same coordinates, it would make sense if these were global illuminating lights. For instance in Blender the actual position of the sun doesn't matter, only the angle at which parallel light rays are cast. And I imagine this is pretty much the same in all 3d modelling software. Then again why would there be 8 global lights in a scene.. 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 23, 2019, 01:17:24 PM
@vortext - that is a very useful advance, and yes the zoom 5 sizing is dependent on the size of the LOD. The n by 256 is when the LOD is too big to fit in one texture. Remember to keep a small constant margin all round (say 4 or 8 pixels) so that nothing bleeds past the texture edges when rendering and it will also lead to cleaner FSH.

In work going on in the background a milestone - Matt has sucessfully loaded the reference building into gmax, and (albeit sans textures) into Blender - but it's version 2.79. We have not been able to get it into 2.80 as the .3ds functionality isn't there. FBX fails in both.

TODO: Decide for sure which version of Blender we go forward with - 2.79 is stable, 2.80 a Beta. From a development perspective using the latest stable version is to be favoured over an incomplete Beta.  Is there anything so far other than "look and feel" that isn't in 2.79 that we have identified we need? If not I vote to use 2.79 first as that will maximise our chances of success for now. 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 23, 2019, 04:40:45 PM
Quote from: rivit on February 23, 2019, 01:17:24 PM
@vortext - that is a very useful advance, and yes the zoom 5 sizing is dependent on the size of the LOD. The n by 256 is when the LOD is too big to fit in one texture. Remember to keep a small constant margin all round (say 4 or 8 pixels) so that nothing bleeds past the texture edges when rendering and it will also lead to cleaner FSH.

In work going on in the background a milestone - Matt has sucessfully loaded the reference building into gmax, and (albeit sans textures) into Blender - but it's version 2.79. We have not been able to get it into 2.80 as the .3ds functionality isn't there. FBX fails in both.

Yeah, the textures are the hard bit. Exporting meshes FROM Blender to gmax/3ds Max is pretty straightforward via FBX, but textures/UVs don't play nice.

Quote from: rivit on February 23, 2019, 01:17:24 PM
TODO: Decide for sure which version of Blender we go forward with - 2.79 is stable, 2.80 a Beta. From a development perspective using the latest stable version is to be favoured over an incomplete Beta.  Is there anything so far other than "look and feel" that isn't in 2.79 that we have identified we need? If not I vote to use 2.79 first as that will maximise our chances of success for now.

I think you're right that 2.79 is the better choice for now. The interface can be... a bit confusing, but it's fine once you get the hang of it. 2.80 is meant to be out of beta in "early 2019", fwiw — but when that happens, i don't think there should be anything to prevent us porting our scripts etc forward to 2.80.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 24, 2019, 12:16:53 AM
Agreed developing against a stable release is more sensible, and the lack of 3ds support in 2.8 is another reason to do so.

As for potentially porting scripts forward, from what I understand there're some fundamental changes internally as to how a scene is structured in 2.8. It used to be organized in layers, going forward however so-called collections will take their place. This change also directly affects the python api concerning how the scene is accessed. Something to keep in mind I suppose.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 24, 2019, 12:30:38 AM
Quote from: vortext on February 24, 2019, 12:16:53 AM
Agreed developing against a stable release is more sensible, and the lack of 3ds support in 2.8 is another reason to do so.

As for potentially porting scripts forward, from what I understand there're some fundamental changes internally as to how a scene is structured in 2.8. It used to be organized in layers, going forward however so-called collections will take their place. This change also directly affects the python api concerning how the scene is accessed. Something to keep in mind I suppose.
Yeah. Fwiw, what they're calling "collections" are actual layers in the sense that anyone who's used Photoshop and a whole lot of other programs would recognize. 2.79 "layers" are a pretty poor approximation of layer functionality; you're limited to 10 of them, you can't name them, etc. All of which is to say, i think 2.80 layer functionality will, if anything, make things easier; but if we stick to 10 or less layers in our 2.79 scripts, we should (in theory, anyway) be able to port them straight over to 2.80.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 24, 2019, 11:23:43 PM
Well I was going to summarise how we were doing today, but I've been staring at a code editor for too many days in a row, so got taken by the bug that is trying out a new program called Blender. Because Matt had managed to get it to import the Reference BAT, I wanted to see for myself what it was like. Keep in mind I'm no 3D Editor hero - I have only played with gmax and 3dsmax on occasion.

Well I have to say, I'm not going to get those several hours back in a hurry. My first impression was that of a GUI where everything you need is hidden, nothing you need frequently stays where you found it and that it needs the dexterity of an octopus to wrangle. It is written with a workflow I really don't understand intuitively (definitely not Windows) - probably really good to prototype in, but for exact work may well be extremely tedious - jury out on that. Someone who actually knows how to turn primitives into something like a 3dModel will be better placed to judge. That simple things like input fields don't work as you expect is disconcerting but will presumably grow on you.

My overall impression was like driving a mid-class vehicle with some very powerful modifications requiring a zillion settings to prevent yourself running off the road. Don't get me wrong, this can probably do what we need it to but so far for me its going to be like a relative you may only tolerate to see when you have to. Too harsh?  :o Maybe. It is a first impression. Even after 1000 bugfixes 2.79 still can get itself in a dither, or show faulty imagery. And why, oh why, is there no prominent UNDO button?

Firstly the import of the 3ds model worked - with reservations. It brought everything in, but mangled the shrubbery badly. It brought in the Sun, Sky and 3 cameras - but none of them were meaningful - just empty entries. It brought in the LODs OK. Also a lot of other dummy entries. It did create some sort of material library - but didn't bring in the materials themselves or their labels. Even after editing materials I couldn't get any bitmap materials to work - but that's probably me. I did get some procedural ones to work.

So given that the Reference Model and its intricacies was all beyond me I thought I'd concentrate on the Renderer(s) - an interest of mine from a long way back. There are two, an internal renderer and the Cycles renderer which uses the GPU. The internal renderer seems to make a standard Sun and Sky (don't know where is standard and can't see any flexibility as yet, except for atmospheric effects which are cool), and renders simple things well. Color control was surprisingly primitive but then most things are done with textures. It may well do, if and when we can see what gets done with textures.

The Cycles renderer was, for me, a bit like playing with the code from a series of published scientific articles on rendering (of which I've read many) so I recognised most of this - it can do quite a few rather neat things, yet once again also can be fooled easily into incorrect renders. May be me again too here. Lots of settings to get right. Transparency was one place I got strange results. This is an infinitely configurable beast - so it too can do what's needed - just need a PhD to work out how.       

Image processing looks quite strong both compositing and fiddling - definitely PNG oriented, but may do some post processing for us for the variable night alphas. Its strongly supportive of animation, and it also has a Game engine mode for making moving things for games, which I didn't try out. There are many public domain addins (some of which may give us hints how to code things), and the best of them cost $$$ - no surprises there. Only tried one out - the dynamic Sun/Sky but didn't understand how to drive it or the results - will need to try this again.

Be interested to see what others make of Blender - I remain open-minded. No Monkeys were harmed in the making of this review.

Here the only lasting result of my playing today - Cycles renderer: metal,foam,velvet. Blender Orthographic Camera in the wrong place but Standard Sun/Sky.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 25, 2019, 01:15:05 AM
Quote from: rivit on February 24, 2019, 11:23:43 PM
Well I was going to summarise how we were doing today, but I've been staring at a code editor for too many days in a row, so got taken by the bug that is trying out a new program called Blender. Because Matt had managed to get it to import the Reference BAT, I wanted to see for myself what it was like. Keep in mind I'm no 3D Editor hero - I have only played with gmax and 3dsmax on occasion.

Well I have to say, I'm not going to get those several hours back in a hurry. My first impression was that of a GUI where everything you need is hidden, nothing you need frequently stays where you found it and that it needs the dexterity of an octopus to wrangle. It is written with a workflow I really don't understand intuitively (definitely not Windows) - probably really good to prototype in, but for exact work may well be extremely tedious - jury out on that. Someone who actually knows how to turn primitives into something like a 3dModel will be better placed to judge. That simple things like input fields don't work as you expect is disconcerting but will presumably grow on you.

My overall impression was like driving a mid-class vehicle with some very powerful modifications requiring a zillion settings to prevent yourself running off the road. Don't get me wrong, this can probably do what we need it to but so far for me its going to be like a relative you may only tolerate to see when you have to. Too harsh?  :o Maybe. It is a first impression. Even after 1000 bugfixes 2.79 still can get itself in a dither, or show faulty imagery. And why, oh why, is there no prominent UNDO button?

Firstly the import of the 3ds model worked - with reservations. It brought everything in, but mangled the shrubbery badly. It brought in the Sun, Sky and 3 cameras - but none of them were meaningful - just empty entries. It brought in the LODs OK. Also a lot of other dummy entries. It did create some sort of material library - but didn't bring in the materials themselves or their labels. Even after editing materials I couldn't get any bitmap materials to work - but that's probably me. I did get some procedural ones to work.

So given that the Reference Model and its intricacies was all beyond me I thought I'd concentrate on the Renderer(s) - an interest of mine from a long way back. There are two, an internal renderer and the Cycles renderer which uses the GPU. The internal renderer seems to make a standard Sun and Sky (don't know where is standard and can't see any flexibility as yet, except for atmospheric effects which are cool), and renders simple things well. Color control was surprisingly primitive but then most things are done with textures. It may well do, if and when we can see what gets done with textures.

The Cycles renderer was, for me, a bit like playing with the code from a series of published scientific articles on rendering (of which I've read many) so I recognised most of this - it can do quite a few rather neat things, yet once again also can be fooled easily into incorrect renders. May be me again too here. Lots of settings to get right. Transparency was one place I got strange results. This is an infinitely configurable beast - so it too can do what's needed - just need a PhD to work out how.       

Image processing looks quite strong both compositing and fiddling - definitely PNG oriented, but may do some post processing for us for the variable night alphas. Its strongly supportive of animation, and it also has a Game engine mode for making moving things for games, which I didn't try out. There are many public domain addins (some of which may give us hints how to code things), and the best of them cost $$$ - no surprises there. Only tried one out - the dynamic Sun/Sky but didn't understand how to drive it or the results - will need to try this again.

Be interested to see what others make of Blender - I remain open-minded. No Monkeys were harmed in the making of this review.

Here the only lasting result of my playing today - Cycles renderer: metal,foam,velvet. Blender Orthographic Camera in the wrong place but Standard Sun/Sky.

LOL. I have never quite understood why people hate Blender, but this is ... a pretty detailed explanation.  :D

A couple of points:

Cycles is def the renderer we want to use. Blender's internal renderer is antiquated and lacks support for a far few modern features. 2.80 includes a new renderer called Eevee, which is basically interchangeable w Cycles — materials created for the former work in the latter, and vice versa — and is easier to use, so that may be our longer-term answer, but for now, I think we stick w Cycles in 2.79.

The dynamic sun/sky is fun to play with, but just using a basic HDR also gives pretty acceptable results. And we'll presumably be placing lights etc via script?

2.80 fixes a lot of the UI-related quirks that 2.79 users have come to know and, ahem, love. So again, I think that's prob our best long-term option once it's out of beta. But 2.79 is OK (honestly!) once you get used to it. Here's my latest project, all modeled in 2.80: you can get some pretty decent results, and I personally find it WAY less tedious than 3ds Max (although, of course, that's prob a case of familiarity with one workflow over another):

(https://i.imgur.com/YhhNC48.jpg)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 26, 2019, 11:38:57 PM
Hey all, how would we feel about setting up a Discord server for this project? I've taken the liberty of snaffling the Bat4Blender name; if you guys are keen, lemme know and I'll send you the link.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 27, 2019, 04:26:27 AM
A bit of progress!

I've imported Matt's model into Blender 2.79 as a .3ds. This seems to be the only import method that's working at the moment — FBX throws me a heap of errors, annoyingly. I'm going to have a play around with Matt's model in Max to see if I can simplify it a bit, and see if that helps (or, failing that, to at least identify the sticking points.)

Unfortunately, as discussed here before, using .3ds as an import/export format seems to break materials. There doesn't seem to be any way around this except manually recreating materials; the progress here is that I've recreated a couple of Matt's materials in Cycles, and they look basically the same as they do in Max. Hurrah!

(https://i.imgur.com/RAxUMFr.jpg)

(https://i.imgur.com/jvCR4Xg.jpg)

(Also worth noting: you can see from the first pic that the UVs generated by Max's UVW Map modifier look like a dog's breakfast in Blender. This is a problem I came up against trying to export from Blender to gmax via the .3ds format, too.)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on February 27, 2019, 08:37:32 AM
he yeah Ron all those points you mention regarding the unintuitive UI and elaborate nature of the workflow is exaclty why the re-design of 2.8 is such a big deal. That, in addition to the real-time renderer and along with the aforementioned restructuring of things under the hood in fact makes it quite a different piece of software compared to previous versions, and some argue (https://youtu.be/lPVpg4_POww?t=30) it should've been called Blender 3.0 to demarcate this.

Nice progress Tom!  :thumbsup:

Can't help with the max materials as I don't have max to begin with, which in fact is one the motivating forces behind all this (and yes, acquiring a student license for 3dsmax would be a lot easier than trying to create an viable alternative to it but where's the fun in that!  :D)

As for a discord server, I suppose another channel of communication doesn't hurt. Just as an FYI there actually is a sc4d discord server, which I happen to be the owner of (shameless plug  (https://discordapp.com/invite/APzvgZq)  ;D ) and another channel w member roles is easily created as well. 

On my end I ported the addon to v2.79 and barring any unforseen obstacles I reckon a working prototype should be ready somewhere by the end of next week, which as far as I see it needs to be able to do 2 things (well 3, really);

- able to render all zooms & rotations
- able to add/delete/export LOD as 3ds

Lots of caveats and remarks to be had here. For starters the first point implies the ability to rotate the sun object as well. I assume (dangerous .. ) this will work pretty much along the same lines as rotating the camera object, however, I have not yet implemented this. So, fingers crossed there.

Second of all I fully expect the alligment of views will need to be tweaked multiple times as we work through various test cases, i.e. do not expect perfect allignment. No promises on correct output dimensions either, and slice 'n dicing may need to take place outside of Blender to begin with anyway. Same goes for converting png to fsh and assignment of tgi, though GoFsh can do a lot of heavy lifting there.

As for the yet unmentioned 3rd point; the addon should be able to be distributed and installed on other machines. Spend yesterday evening working out the kinks on that end and while the result is just a simple .zip file, there's a (un)surprising amount of things which can go wrong ..  ::)

Once the addon can do those 3 things more stuff can be added to it like; preview renders, adding/deleting sun & camera from the scene, custom LODs etc. And while on the subject of LODs, what is the purpose of the LOD3, LOD4 & LOD5 as seen in gmax? I suppose these correspond to the zoomlevels, but I don't quite get what their purpose is? Is there a different geometry to them per zoom, and if so how does that benefit mapping the rendered image? 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on February 27, 2019, 04:34:59 PM
Nice work you three  :thumbsup:

I figured that the way I model (super-quick and tailored for SC4, but not 'proper' in terms of 3d modelling) might cause some issues. Tom, the textures are looking good...I also got the model to export/import as an .obj file. That process also created a texture library; unfortunately my blender knowledge is insufficient to do anything else with that other than note that it occurred!  ???

The LOD3/4/5 are all the same size in gmax (they are literally cloned) but each represents the zoom size of the pasted image in game. In that regard, I guess you'll also need to replicate thumbnail cameras and z1 and 2 unless that is just going to be done by a calculation....
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on February 27, 2019, 08:16:04 PM
Quote from: mattb325 on February 27, 2019, 04:34:59 PM
Tom, the textures are looking good...I also got the model to export/import as an .obj file. That process also created a texture library; unfortunately my blender knowledge is insufficient to do anything else with that other than note that it occurred!  ???
Hmmm, just had a look — same deal as the .3ds, I'm afraid, in that they're all just blank materials. It seems that either way, we need to recreate the materials and (possibly) some of the UVs. I think it'd also be the same w FBX. This is why I came to the conclusion for my models that it was gonna be easiest to just texture them in Max — all of the various inter-program export formats seem to have trouble w materials and textures :-/
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on February 27, 2019, 09:23:04 PM
Hi all - nice work everyone,

seems like a day for a number of reveals and progress milestones. I have finally completed the Hatrack for the GUI and can present that here. Its working code inasmuch as when you hit a button it tells you what its going to do, but doesn't actually have the backing code yet. That's where the stuff the vortex has been producing will fit straight in. It takes the form of an exe right in this folder https://1drv.ms/f/s!AphvaLJG-tShg4waHjo9pVt58Fadqw (https://1drv.ms/f/s!AphvaLJG-tShg4waHjo9pVt58Fadqw) so feel free to download it and punch it around a bit and think about whether I have everything covered. If it survives then it can be built into a GUI for the plugin. Ive also started on the downstream processing thinking.

The other big thing is that I've managed to get all of the Camera and Light data out of the rig. Oddly enough just loading the rig into gmax as if it was a model was the key at getting to this data.  I am pleased to see its largely consistent with what I had derived by measuring bitmaps so many years ago. Very close but subtly different.

All of this data is also to be found in the docs folder named above as a text file and as an .xlsx file.

There are in fact 7 cameras defined altogether, but in our implementation we really only need one and change its settings as we need to. I also have the coordinates of the target for the cameras.

As far as lights go I was very surprised to see there  were 10 lights defined - in addition to the prime light a series of fill lights play the roll of smoothing out the light all around a model in a studio-like lighting setup. They also play a role in getting a fairly straight light response to texture color.  Now in principle we will only be using a Sun - corresponding to the primary light but we have the data on the colors as well, and so some clues as to enhance the result.  see 2nd pic

This data has thrown up an anomaly with BAT4Max - it appears the Sun here is not quite at the same vector as the gmax rig - its lightly lower so shadows cast in BAT4Max should be longer and north of those in gmax. Also the target (in previews at least)  is taken offset from the centre of the LOD, rather than in gmax a fixed offset from the ground which presumably slightly changes the positioning of the render i.e. where it ends up on the bitmap.

I'm looking forward now to using those bits already made and will take some time to think through the main processing - in particular the slice and uv mapping of the LODs. I'm also going to rtfm some of the bits I obviously don't get just by fiddling with Blender.

Ron
 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Jasoncw on February 28, 2019, 09:01:16 PM
It's great seeing progress on this project. :)

The GUI looks good in general (and I love seeing that model name and group id area) but there is an issue I see.

There are buttons for setting Day, MaxisNite, and DarkNite. These change the light settings and also the rendering settings.

Then there are two preview buttons, and pressing one of the buttons changes the rendering settings and starts a preview. The problem with this is that it prevents the user from changing the rendering settings themselves. If you edited the rendering settings, you wouldn't be able to do preview renders, because pressing the preview button would change the rendering settings back. So if you wanted to include a button to make draft rendering settings, you would want to make those buttons separate from the preview render button.

In BAT4Max there are also buttons for making the night brighter or darker, and what those do is switch between MaxisNite and DarkNite without affecting (overwriting) rendering settings which the user might have customized. I've personally never used these buttons, and just manually re-changed my custom rendering settings, but I don't know if other people used them.

The draft and final rendering settings would have been easy to implement in BAT4Max but iirc the purposeful choice was made to use one set of generally good settings, to make BATing as WYSIWYG as possible. And then for special cases, to encourage people to change the settings themselves. I personally agree with that decision (and the rendering settings in Cycles are more intuitive than Mental Ray's), but this is a different situation and so some presets like that might be useful.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on March 01, 2019, 08:20:33 AM
Guys, this thread goes above my head. I just applaude to the sound of knowledge.  &apls

I think the advantage of cycles render is it's free. One can switch from cpu to gpu rendering. It can make use of highly specialized hardware on 3D rendering - gaming cards. Especially cycles renders can benefit from nvida cards and those so called cuda cores created for 3D games. So you can speed up rendering by combine graphic cards instead of buying whole new pcs.

Disadvantage is: the whole project has to be loaded at once into graphic cards memory - means: the object and the texture, everything has to be present in the gpu memory. Cyckes doesn't support file swapping.

But on the other hand there is no limit in render quality. You just choose infinite render and when you like it you just stop rendering. You can stop rendering at any time without losing data. This is what I like about cycles render engine.

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Barroco Hispano on March 01, 2019, 08:41:05 AM
Blender cycles and Evee is compatible with PBR materials so that will be very helpful  ;D
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 01, 2019, 03:53:13 PM

Hi Jason and many thanks for taking a reviewers eye to the GUI proposal. I appreciate the feedback and I'll answer your valid point on the Previews in a slightly roundabout way.

The logic behind the Plugin Design is to minimise the code in general so the following principles are followed:

What I'm trying to avoid here is the repetition of code I see in the maxscripts to check and recheck state on the run instead of nailing that down before starting things. Since we aren't experts in Python we need to keep the code clear and simple to avoid burying ourselves. We have the advantage of a clean slate - we are not modifying existing code.

So I envision that there is a given State associated with Day, Maxis Night (we are not doing MN), and Dark Nite - this gets set up when you select that rig. Resetting the rig reinstates those settings at will; Rebuild Rig sets things up from scratch back to Day. These are insurance - you could just as easily select the Rig setting again. Previews and Export will set the check/set the Camera/Render parameters, then do their thing.

Draft Render (which I reintroduced to the design and which may not survive) is an anomolous (presumably much less fancy and hence FAST) state and so it should save the Standard state before and restore it at completion. The same sort of logic will be taken in the LODs, and the files management. Make and maintain standard state.

Now its still not at all clear what the "locked" Settings and Parameters for Environment and Render will be (and they will need some form of consensus), but I would hope to restrict these as much as possible. It will depend on what the art experts (like yourself) need to recreate the right look and feel of the result. But once set, thats our standard.

It should be easy to add/change things but difficult to break the standard.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on March 03, 2019, 03:30:38 AM
OK people, here is is: Matt's Engineering Dept, imported as an .OBJ, and rendered in Cycles. Ta-da!
(https://i.imgur.com/N2kMyjZ.jpg)

i had to recreate all the materials, because the materials imported as part of the .OBJ were, er, interesting:
(https://i.imgur.com/sNCkeJz.jpg)

How/why this node group was created and how it reflects the Max materials are ... unclear, but it was easy enough to nuke them and re-apply Matt's textures. I created the glass material (which is reflecting the sun in the HDR i used) and the lights from scratch, with Cycles' Glass and Emission Shaders respectively.

But anyway, I'm happy with this! It looks good, the OBJ import has recreated the shrubs etc well, and the little details etc have rendered fine. Hurrah! We have a reference building in Blender!

(https://i.imgur.com/tvg9NSV.jpg)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 03, 2019, 04:01:44 AM
Well done Tom - a huge step forward for sure!
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 03, 2019, 09:41:04 AM
Great work indeed, looks nice!  &apls

Thanks for the info dump Ron regarding the camera & lights, usefull reading material.  :thumbsup:

Meanwhile the gui is hooked up to the camera. Was a bit more work than anticipated due to having to register a set of variables with Blenders UI manager. Kinda makes sense though, and passing the same info along to other methods should be relative straightforward now. The addon is relocated to the Scene context menu btw, since v2.79 doesn't have an Output menu. And minor nitpick but it's a bit annoying Blender doesn't seem to have proper radio buttons (i.e. circular). . ah well.

(https://cdn.discordapp.com/attachments/476300301872398349/551816795531640863/blenderbat_5.gif)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: c.p. on March 03, 2019, 02:15:56 PM
Looks like some pretty good progress :thumbsup:

Are you going to be integrating this with gmax, or are you going to attempt to create your own SC4Model files?  (I imagine the UV mapping calculations to be a bit of a nightmare).
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 03, 2019, 03:20:49 PM
@vortext - also nice progress - you're all starting to make me worry about the next bit.

@c.p.

   The short term goal is to get Renders and LODs out to be able to use gmax. My ultimate goal is to take gmax out of the equation. My main question a the moment is will it be in Blender Python or externally with say .NET

From the maxscripts I have what needs to be done - its essentially this given a Render, and a LOD:

1) Project (transform) the LOD to the view of the render, i.e. look at it from the camera viewpoint then u=across and v=down. That's the texture coords.
2) cull all faces facing the back or completely obscured
3) transform the LOD mesh into SC4 coordinates
3) make a S3D from the resultant mesh and use the u,v  for each vertex. Save the 1 FSH.

The problems arise when the render is bigger than 256x256 then
1) Do 1 as above.
2) First divide the mesh horizontally at the 256 boundaries by planes made from the corners of the 256x256box and the camera point.
3) then for each horizontal slice do the same vertically at the 256x256 boundaries.
4) fix the uv for the vertices so produced
5) Transform the resultant meshes back to SC4 coordinates, cull hidden faces
6) Make one s3d and FSH for each 256x256 part of the render which actually has a mesh.

There's also some ID housekeeping to do.

  Conceptually not ridiculous, but 3d geometry is always a brainscrambler. 


A discovery:
When doing a little work with MattB to verify the difference in sun angles between gmax and 3dsmax he mentioned in passing that the game actually doesn't have correct shadows either. This isn't something I've ever noticed but he's right. Somewhere in the MAXIS machine the viewpoint azimuth (angle from N) and the sun azimuth have been interpreted as the same thing, meaning the in-game shadows are out by 14.5 degrees when compared to the shadows on BATs.

The two images below show the difference - the zip file is a little .dat that fixes the LightingExemplar to carry the correct angles.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on March 03, 2019, 09:00:48 PM
Great progress again! Excellent work on the engineering department textures, gui and new shadow mod!

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Barroco Hispano on March 04, 2019, 10:17:03 AM
When the sun direction is modified, this happens:

(https://nsa40.casimages.com/img/2019/03/04/190304071803852269.jpg)

Some time ago I spent hours trying to solve it but it was impossible...
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on March 04, 2019, 01:28:20 PM
I guess it is worth mentioning that a batted object is rarely seen in isolation; it will always be surrounded by other objects, and, as shadows are not cast upon bats by other bats, small glitches tend to go unnoticed inside the clutter of a typical scene.

Also it depends on how the bat was made and by which designer. From my observations, it appears that each designer had their own separate version of Max and in addition to each being different from the other, there didn't appear to be any versions that were an exact match for the gmax that the community was given. The rigs weren't consistent, either (you can see this by looking at the California Plaza shadows. All the landmarks were made by one or two Maxis designers; industrial by another; residential another, etc, etc).

Add to that each designer's own inevitable artistic signature and I'm amazed that the Maxis buildings are congruous. I suspect the materials palette covered any glaring inconsistencies: by using Maxis materials I can still get 3dsMax 2017 to appear 'old school' as it were....

Your example probably has an unintentional underground lod.

You can see the same effect in one of the examples (3rd from the bottom) in this:
(https://i.imgur.com/hackXDJ.jpg)

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on March 04, 2019, 04:02:11 PM
I must say. This is awesome work that is going on here.

This is truly an inspirational and collaborative effort by all that are involved. Kudos to you all  :thumbsup:  &apls

To add some flavor, I use some scripts from the Quake m3d scene to export True 3d Models from Gmax to Blender. I can't help wonder if such scripts can help users get there Models form gmax to Blender for users that have a large repository of models that can be touched up with the rendering power of Blender.

This would also help with workflow for users and content creators that have experience with gmax but not so much with blender. So you could do the modeling in Gmax and Texture in Blender? &idea

Anyway I am keenly following these developments. You all need a pat on the back for the effort shown.

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Odainsaker on March 04, 2019, 07:42:25 PM
@rivit:  Even though it arose from a tangential issue spurred by an offhand aside, your shadows fix resolved a glaring bit of incompetence that has irked me for years...

(https://i.imgur.com/3AaxU1U.jpg)

Finally, I can sleep in peace!  Thank you!  Thank you!  Thank you!

Ancient Greek geographers used the lengths and angles of sundial shadows cast in Alexandra and Syene in Egypt to determine the astounding circumference of a spherical Earth.  SimCity 4 astronomers would have looked at the wacky shadow of my column and instead been astounded by the inanity of their world.  No longer will they be deceived by the digital shortcuts underlying the mysterious workings of their universe.

Hehe, actually, I recall that SimFox used to lambast Maxis for this blatant mismatch in its BAT lighting and game shadowing setup.  I also seem to hazily recall that at some point he had mentioned that his sun position in BAT4Max had deviated from the primary "sun" light Maxis setup in Gmax.  With Gmax and even Maxis's original buildings already not matching the game, it may have been decided that longer BAT shadows and different angles of incidence of sunlight against typical walls were more graphically effective than purity to Gmax's and Maxis's fuzzy setup.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 09, 2019, 05:34:58 AM
So a very puzzling bug threw me off the past few days. It boiled down to the render not being alligned correctly whenever the camera was (re) initialized in the scene, even though all prerequistes checks for scene context were being met when rendering. Turns out whenever new objects are linked to the scene an additional update method needs to be called as well . . great. .  &sly .. no idea what it does exaclty, but it did ultimatly resolve the issue though it took a while to get there.

Anyway, preview and rendering are working now at least. Though I suspect the orthographic scale is still not correct for all zooms. More specific, I suspect a single orthographic scale needs to be used across all zooms, rather than recalculating it for each zoom based on camera height as it is doing currently.

(https://cdn.discordapp.com/attachments/476300301872398349/553926491000406058/blenderbat_6.gif)

Bit annoying the render preview window positions itself at the mouse cursor it seems. Would be nice if it could have a fixed position on screen. I also wanted to show the render previews when rendering all zooms & rotations however that's not as easy as it appeared (exact same issue & possible solution here (https://blender.stackexchange.com/questions/71454/is-it-possible-to-make-a-sequence-of-renders-and-give-the-user-the-option-to-can) ). So yeah, putting these issues on the backburner for now.

Instead I want to implement the LOD export this weekend, however, I'm having trouble going through the steps manually to start with. Either I'm not using the correct export settings in Blender, or perhaps importing it wrong in gmax. Either way the dimensions always come out wrong, specifically the it always becomes a regular box shape in gmax (i.e. width, depth & height are the same) even if the LOD in Blender is an irregular shape.  %wrd

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 14, 2019, 06:18:14 PM
Despite apparent silence things have been progressing in the background. Today I would like to introduce a new program called OBJxS3D which is another brick in the wall needed to get from Blender to SimCity4.

This program is designed to accept an OBJ file and produce a S3D file which can then be inserted into a .DAT with Reader. It is designed to swing both ways - its also possible to take an exported S3D file and turn it into an OBJ file. If appropriately numbered then the S3D file and its Textures will be correctly imported and linked in Reader. This is a fundamental step in the BAT4Blender export process, where we export texture mapped LODs. Done this way it will make the gmax step obsolete in the export process. 

This program should eventually also be useful for making network models for the NAM (shoutout to the Eggman), or automata in tools like Blender, MilkShape or even 3dsMax (anything that can edit and export OBJs actually) as no extra work will be required if the OBJs and their materials are correctly named.

The materials lib it uses/produces will propagate the texture references given, and these are/should be named as they are in GoFSH (T-G-I-C0.bmp) which can be used to produce the textures needed for the materials. 

Currently the program is in TestMode i.e. it will produce two files for every OBJ file it can read namely _SaveOBJtoOBJ.obj and _SaveOBJtoS3D.s3d. For every S3Dfile it reads it will produce _SaveS3DtoS3D.s3d and _SaveS3DtoOBJ.obj and _SaveS3DtoOBJ.mtl files.

The point of making these particular test files is that the OBJ files should give the same result as the originals when loaded into a Modelling program, and the S3ds when imported into Reader. Don't try big models - <100 vertices should be more than enough to test with. Round trips OBJ->S3D->OBJ and S3D->OBJ->S3D seem to be working correctly so its time to test more widely. Reader can be used to save decoded S3D and FSH files.

The program can be found here:  https://1drv.ms/u/s!AphvaLJG-tShg4xvy66rkCFJZGnJuA (https://1drv.ms/u/s!AphvaLJG-tShg4xvy66rkCFJZGnJuA)
OBJxS3D.exe is a .NET 2.0 exe so should work on any Windows from XP to Win 10. Just put it in any temp folder.

I would ask that a few people please test this program to see that the following things do work properly as I suspect results may vary depending on the Modelling programs used or the source of the OBJ in the first place.

OBJ files are supposed to use Right Handed XY Z(Up) Coordinates, Simcity uses Left Handed XZ Y(Up) Coordinates - so expect models to be oriented incorrectly if the OBJ file is true to standard.  I don't know Blender well enough to do the Texture tests.

Look out for:
   Please feedback your questions/findings to me with the name of Modelling Program used to produce the OBJ and which of the above 7 points failed and how. If there's a predictable pattern I'll change the program accordingly. Obviously Blender is the main target for compatibility. Milkshape is a useful program too - a lot like gmax.

  I have so far only tested with Milkshape - it uses the same axes as Simcity but flips the textures Horizontally. It takes the OBJ literally and doesn't appear to worry about axis orientation.
   
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 16, 2019, 08:18:33 AM
Quote from: rivit on March 14, 2019, 06:18:14 PM
Despite apparent silence things have been progressing in the background.

Yes this it true for me as well, suppose it's time for the weekly show and tell.  :D

At any rate, things have been progressing nicely with the addon. Notable additions include;

- fully functional gui
- lod export as .3ds (and export as .obj in the works)
- png written with tgi as filenames
- 'tiled' rendering

And here's how it looks atm.

(https://cdn.discordapp.com/attachments/476300301872398349/556489848094064641/unknown.png)
Options for HD, nighttime, etc will be added as things progress.

With regards to the tiled rendering, instead of slicing the complete rendered image after the fact it turns out Blender actually has the option to render part of the camera view, and save the partial render as an image. Rather handy indeed.

(https://cdn.discordapp.com/attachments/476300301872398349/556187847330365480/unknown.png)
Ignore the filenames here, it was for testing. For context you're looking at a 32x32x32 diagonal cube.

However there's one thing still to be tackled here, which is to figure out if a partial view is empty and disregard it before writing (as writing empty images will exhaust the naming scheme rather quick, as explained in Robin's post here (https://sc4devotion.com/forums/index.php?topic=17650.msg515495#msg515495)). The most straightforward way I can think of is to check the alpha values of the pixels array, though I suspect some matrix magic might do the trick a lot faster. . but yeah projection matrices really are hard to wrap ones head around.

At any rate, there're few other things on the to-do list still but very close to a first version of a functional addon.  :)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 16, 2019, 03:36:44 PM
You've made a lot of progress  &apls.

Well that partial render looks rather handy. I wouldn't be surprised to find Blender knows when its empty by virtue of some variable or method it has because it saves a lot of render time to know when the model is not visible.

  If we export LODs to 3ds for gmax its just a model as gmax does the projection work. But if we go ourselves then we still need to attach texture coordinates to the LOD - that is the time to check - if the LOD model doesn't encroach into the rendered view the calculated texture coordinates will be >1 or <0 because the model wont intersect the render. The LOD needs to in the view projection to produce the texture coordinates, but its still saved in model coordinates rotated accordingly for the OBJxS3D conversion to result in the correct perspective for SC4. 

Do also check that you have the view Rotation enum correctly numbered 1=S, 2=E, 3=N, 4=W which is how it shows in gmax and Bat4Max.  Also rendered textures are always produced and numbered from top-left, across then down. The correct formula for the IID is given below from the Bat Scripts (up to 64 FSH per view rotation)


    ''--------------------------------------------------------------------------------------------------
    '' Function:   TextureOutputIDStr
    '' Param:      texIndex                     
    '' Desc:         
    ''--------------------------------------------------------------------------------------------------
    Private Function TextureOutputIDStr(Optional ByVal texIndex As Integer = 0) As String

      Dim highSwizzle As Integer = 0

      '' awful kludge to be able to handle more than 16 textures per view
      '' sharing the second digit between rot & the high bit of the texture index
      Dim digit2 As Integer = texIndex \ 16
      texIndex = texIndex Mod 16
      If (digit2 > 3) Then
        MsgBox("This building is huge! It's just not going to work!")
      Else
        highSwizzle = digit2 * 4
      End If

      Dim guidStr As String = ModelID() & (zoom - 1).ToString("X1") & (highSwizzle + (rot - 1)).ToString("X1") & texIndex.ToString("X1")

      Return guidStr

    End Function
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 17, 2019, 11:46:43 AM
Thanks Ron!

Actually had the rotations enums backwards, fixed now as well as going in the correct order for tiled rendering.  :thumbsup:

With regards to the tgi, don't quite understand yet what they're doing, however as tgi gen does work for small scale stuff I returned my attention to the orthographic scale and yeah, speaking of awful kludges. .  ::)

Using the same orthographic scale for all zooms wasn't correct. So I made it take camera height into account again, however, this didn't appear correct either. After some manual fiddling and comparisons in Blender I noticed a pattern, the os returned from the gmax formulae decreased per zoom while it should be increasing. Long story short I ended up with the following, wherein the 'default' is the os for zoom5 as calculated by gmax as well..

final_os = default_os + (default_os-os_gmax)

This gave better results, here's how the blender rendered zooms hold up to the uv mapping.

(https://cdn.discordapp.com/attachments/476300301872398349/556903792713662464/unknown.png)

(https://cdn.discordapp.com/attachments/476300301872398349/556903628724764682/unknown.png)

(https://cdn.discordapp.com/attachments/476300301872398349/556903501461061636/unknown.png)

(https://cdn.discordapp.com/attachments/476300301872398349/556903375812427817/unknown.png)
(https://cdn.discordapp.com/attachments/476300301872398349/556903286762897419/unknown.png)

Zoom 1 & 2 are still rather poor fits compared to the gmax renders, however the routine is using 1 pixel x & y margins for all zooms so there's some wiggle room to improve things. 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: mattb325 on March 17, 2019, 02:39:12 PM
That's incredible progress, guys  :thumbsup:

The fit for the z5 is perfect...for z1 & 2, from my observations, the buildings are actually standard orthographic, whereas z3-5 are the SC4 camera angle (if that helps any)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 17, 2019, 04:21:14 PM
Well all things being equal that's not bad. Z5 spot on, the Z4 is only an offset of 1 up,left  away. The others look to be offset and distorted, and I wouldn't trust the Z0 coordinates as they look wrong in gmax.

To be frank, we won't get this right till we can produce the texture coordinates ourselves to see what the numeric differences between them and gmax are. Once we have that at least we can mathsage it back into shape.

Thinking further about the partial renders - we still will need to chop up the LODs for all the parts. There is a slice by plane function in Blender that we may be able to put to use for this - I suspect it will need some playing with before we know how it works for us. We need to slice by a plane that goes from the view point, through the sides of the partial render to reduce the projected LOD to what SC4 needs - then cull any backfacing faces.

I've done some futher testing with OBJxS3d and it seems Blender picks up the OBJs of exported s3ds quite well. Like Milkshape the mesh comes in correctly (ie faces the right way optically) and only the texture v coords are flipped vertically. So that was surprisingly good. Haven't gone back the other way yet but I don't expect a lot of problems. Will fix the program to take CLI arguments so we can invoke it directly from either a .bat file or from the plugin itself by using the OS Run or Exec.

below the result of importing one of my SC4 automata into and then rendering with Blender. Its pretty cubist but thats what you get with a very low poly model.

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 26, 2019, 12:30:36 PM
welp seems like last time I forgot to mention I was away for the week . . :-[  . . so not much progress since last update.

At any rate, back at how to deal with empty tiles. Unfortuanlty it seems the python api has no direct way of telling if there's nothing in the camera view. Worse, there's no direct way to access the pixel array of the rendered image, unless using nodes as described here (https://ammous88.wordpress.com/2015/01/16/blender-access-render-results-pixels-directly-from-python-2/).

So I implemented that, however, it is really slooow .. In fact, for some time I thought there was a bug in the checking procedure because Blender seemed to freeze up, but nope.. it is just a stupid slow process.. Like, it took ~45 minutes to 'render' a 32x32x32 cube. Obviously things can be sped up by not checking every single pixel in the image and check every Nth pixel instead, however that does give rise to the chance an image is discarded as empty while it is not.

So yeah, this is not ideal on both counts and may have to come up with alternatives. I was thinking perhaps raycasting could be of help here, as well come in handy for LOD slicing later on? Any other ideas how to go about tackling this?   &Thk/(
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Jasoncw on March 26, 2019, 04:03:40 PM
The sliced images would have a fixed number of dimensions, right?

If they were empty, wouldn't there also be a fixed number of file sizes? For example wouldn't an empty 256x256 image always have the same file size, and a 128x256, and so on?

The only potential variation I can think of in a genuinely empty area would be that the background color would be different for different renders, but it would always only be one color.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: rivit on March 26, 2019, 04:32:52 PM
@jasoncw
That's a fair observation - PNGs should compress to a constant (small) size if empty (0) or only one color (r,g,b) throughout for a given image dimension. Probably need to save it first to get it though. Other formats like BMP are a constant size regardless of their contents so filesize isn't helpful in working out if they're empty.

@vortext  I think we needed a break anyway - I have finished OBJxS3D and its usable now. But I haven't done anything else either. All of my experiments with Blender itself left me frustrated so I haven't been near it for a while.

Thinking about the empty image thing there seem to me to be a few possibilities

1) don't worry about empty textures for now - it won't go wrong in SC4 since nothing will refer to them.

2) don't do partial renders but rely on the chopping up part to segment everything. That can work out if its empty by knowing no vertex coordinate falls in that 256x256 segment. This involves raycasting from viewpoint through the vertex coordinates onto the texture to get u,v and slicing the meshes accordingly. It does depend on seeing if we can chop things up efficiently in Blender.  1 view/zoom -> 1 LOD with 1 texture -> n  LOD parts with 1 part texture each.

3) don't bother to try checking for empty inside Blender, as checking for empty textures is trivial and very fast in GoFSH - I already do this for a number of things. If textures need chopping/renumbering on the run then so be it - these are also in GoFSH code already. I haven't yet built in the special processing for Bat4Blender but will be reusing everything I already have and adding things won't be that hard.

4) we might need to do the whole segmenting/chopping thing outside Blender - given OBJs of the LODs 1-5 and PNG renders of View 1-4, Zoom 1-5. Then all of that code goes into an .net exe. Not ideal but it may be way faster.

Favouring (4) If small objects that are restricted to <=256x256 pixels rendered can be produced directly from BAT4Blender to feed through OBJxS3D and GoFSH (PNG->BMP->FSH) then that is a major achievement already. Then we can expand to larger items.

We always knew it was going to get quite hard at some point - and this LOD slicing has always seemed the worst part (not forgetting you've already solved a number of other difficult things) - the longer we can avoid having to code it the more chance we have of understanding it.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on March 27, 2019, 03:52:36 AM
Just a "fantozzi thought":

Blender has a really mighty community with many developers and modelling enthusiasts. If you ever get stuck it might be helpfull to present your work so far in a topic somewhere over there (https://www.blender.org/community/ (https://www.blender.org/community/)). Could be we get even some "blender guys" involved with SC4. Making up a new connection with this immense community might be fruitfull in the future.

Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on March 31, 2019, 11:44:11 AM
yeah option 4 seems to make the most sense at the moment, and I've disabled tiled rendering for the moment. Also found a neat example of a dialogue popup and implemented it to warn the user the model is too big to render ( figured it will come in handy anyway  :) ).   

And with that a first release of the addon can be downloaded here (http://github.com/GoldenQubicle/BAT4Blender/releases/download/alpha/BAT4Blender.zip). I'm curious if it will install and what issues may arise .. *fingers crossed*

Couple of notes;


As for install procedure;



To disable the Add-on simply uncheck the box. To uninstall go to the Add-ons tab in the user preferences, search for the addon and expand the info box with the little triangle, this will reveal the option to remove it (screen (https://cdn.discordapp.com/attachments/476300301872398349/561984823690788895/unknown.png))

As for how to procede, I'll have to play around with the .net exe and see if it can be called from Blender perhaps, in addition to some cleanup work code wise. .  ::) Also have a few options to explore regarding the lod slicing ordeal.   
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on March 31, 2019, 12:26:26 PM
Great Work @Vortex

I was able to install the current build however, I jumped the gun early and tried to install from a previous post.

If users have issues they will have to remove previous version. Just a heads up. Blender was getting cranky at me for having two installs.

You have done well so far. I will see If I can get some BATs out as a test.

Thanks again.

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on April 01, 2019, 09:31:47 AM
Thanks Stephen, hadn't thought about that scenario.  :thumbsup:

Also as has been noted, gmax uses multiple lights while in Blender there's a single source (i.e. the sun) which results in pitch black 'shadow' side of the model. In the example below gmax render on the left, blender render on the right (don't have a bigger picture at hand at the moment and too lazy to whip one up...   ::))

(https://cdn.discordapp.com/attachments/476300301872398349/562311102981996566/br-Jan._8_001552849346.png)

Perhaps this can be remedied with additional lights, perhaps with some skybox or environment texture or whatever else gives a good result. The main reason for release atm is that people can start tinkering with these kinda settings, and we can incorporate it into the add-on later. 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Jasoncw on April 01, 2019, 09:31:56 PM
I've been fiddling around with it, while trying to learn about Blender.

So first, for some reason my preview renders were exactly half as big as they should be. The renders are scaled 1:1, the object dimensions and units are all correct. I fixed this by scaling my model up. However, the preview render buttons as a whole work great, and the preview render correctly changes dimensions when your model is bigger or smaller. It's been very pleasant so far. :)

Also, the sun angle is incorrect although reading earlier this is temporary. While the sun angle does match in in-game sun angle, it doesn't match other BAT's sun angle. The in-game angle looks awful on BATs, the facades get drowned in shadow. For the purposes of testing, I've changed the sun angle to the BAT4Max sun angle, by first duplicating the sun, changing the strength of the original sun to 0, and then doing all work on the duplicated sun. This prevents the preview button from reverting my changes. The BAT4Max sun position is -474,-352,575, and it points towards the origin.


Learning about Blender's Cycles renderer has been... interesting, lol. On one hand some aspects are pretty solid, and other aspects are weird or behind the times.

For the material, in 3ds Max, I used an Arch & Design material with 50% grey, and reflectivity and glossiness of 0.2 each. In Blender, I used a Principled BSDF material with 50% grey and a roughness of 0.8. I used my Northern Trust Bank BAT because it has a lot of surface detail and a lot of deep crevices. Exporting messed up a tiny bit of the geometry. I don't have a Gmax comparison because there were too many polygons to import it. It's all grey right now in order to reduce variables (the materials and textures would never match anyway so you would never be able to use them as a direct comparison), and also because Blender is too confusing for me to go through and redo it yet, lol

I've tried to organize this so that it makes a little bit of sense, but please forgive me for this mess of a post.

(https://i.imgur.com/Wqskcx8.jpg)

Left: 3ds Max  Right: Blender

Lighting:

Normally in renderers that are trying to be physically accurate, lights are all given realistic brightnesses, and you use exposure control in order to adjust the ouput brightness. This happens with cameras in real life, and even your eyes, when you go between outdoors and indoors on a bright sunny day, you can't see anything for a few moments until your eyes adjust. This is important, because you need the light to be bright in order for it to bounce around and give proper indirect lighting. I was able to find this page (https://blendergrid.com/news/cycles-physically-correct-brightness) which does the work of figuring out how bright the sun and sky should be. So in line with the article, I added emissions nodes to the sun and sky, and have given the sun lamp and the sky strengths of 441 and 29, respectively.

I've given the sun the color from the main gmax sun (0.96 0.89, 0.74), and I've given the sky the color from the BAT4Max sky (0.56, 0.56, 1.00). I used the BAT4Max sky color because it was available, and that's what I'm aiming to match. I used the Gmax sun color because I don't know exactly what the sun color is in BAT4Max. imo the render looks very Gmaxish.

I also gave the sun a size of 0.1 m. "Size" gives the light source a dimension, and this is responsible for the effect of having sharp shadows that gradually become softer shadows the further out they are.

Since the lights are so bright, the render would normally be blown out, so you need to reduce the exposure. This is done in the "Scene" menu (idk exactly what all these different menus are called), above the BAT4Blender stuff. I used -6.75, but I think this is a number that may need adjustment.

Above the exposure is the color management area. Specifically, there's the "Render: View:" which can be changed from Default to Filmic. Switching between these two (or others) changes the way the internal color data is mapped to the final pixel colors in the renders. Filmic compresses a wider range of colors into output range, which is great. The disadvantage is that in order to make space for that wider range, the old range (the range that would typically be within visible range anyway) has less space, and so there is some compression and the image can look flat. This will take some further investigation.

From my current rendering, I'd like for the colors to be more vibrant, for the darkest areas to not be quite so pitch black, and for the forms to have a bit more definition (it looks flat). On the plus side I think the shadows are much better here, which is what you'd expect from a path tracer renderer.

Anitaliasing:

The next big issue is the antialiasing filter. These look at the samples in and around a particular pixel, and merge them, weighting them according to a curve. For some reason blender is really lacking in this department. There's Box, Gauss, the most ancient of filters, and Blackman-Harris, which is a modified Gauss. With BAT4Max we use Lanczos, which is a sharpening filter which is great for BATs. There are plenty of other common filters that would also work great but they're not in Blender. Blackman-Harris seems to be the best available. The default size of 1.5 is very fuzzy, and making that number smaller doesn't make the image sharper as much as it makes it less antialiased. I have it at 0.75 in my render.

In the sampling settings you can change "Path Tracing" to "Branched Path Tracing" which lets you manually choose how many samples each sampling type gets, so if you're having problems in your render (for example, glossy surfaces looking noisy), you can increase the samples for that alone. But since AA is the base sample type you can't individually increase it, you have to jack up the rendering settings as a whole, so branched path tracing is not helpful to us. But I've also found that increasing the samples doesn't fix it anyway.

This is a major problem and as far as I know it's unfixable. I'll keep looking into it though.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: fantozzi on April 02, 2019, 12:13:42 AM
Reading through Jason's report it crossed my mind that "AMD Radeon Pro Render for Blender (https://www.amd.com/en/technologies/radeon-prorender-downloads)" could be the better option for rendering bats. It should work on CPU (only) even if no Radeon GPU on bord. But I never used it and can't say nothing about it except they say it has much more anti-aliasing options including lanczos.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 09, 2019, 05:48:31 AM
Wow, Fantozzi, good find -- I never knew about that renderer. I might try it out.

Also, Jasoncw, fwiw, there are more anti-aliasing settings accessible if you select "Branched Path Tracing" as your sampling method, as described here (https://blender.stackexchange.com/questions/128066/rendering-options-missing). No idea if they'll improve matters, but maybe worth a try? 2.8 includes a new renderer, but it also seems to lack the anti-aliasing method you're after.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 09, 2019, 06:04:28 AM
Quote from: vortext on April 01, 2019, 09:31:47 AM
Perhaps this can be remedied with additional lights, perhaps with some skybox or environment texture or whatever else gives a good result. The main reason for release atm is that people can start tinkering with these kinda settings, and we can incorporate it into the add-on later.
Fwiw, it's pretty easy to use HDRs as environment textures for Blender, and there's also a pretty mature and well-supported (and free) dynamic sky plugin that could be of use here.

(Been busy w IRL stuff for ages, so I'm just going back through this thread now)
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 09, 2019, 06:40:42 AM
Quote from: vortext on March 31, 2019, 11:44:11 AM
And with that a first release of the addon can be downloaded here (http://github.com/GoldenQubicle/BAT4Blender/releases/download/alpha/BAT4Blender.zip). I'm curious if it will install and what issues may arise .. *fingers crossed*

AMAZING. Super excited to play around w this. One day we'll look back on this first release as the day we took our first step toward freeing ourselves from the shackles of Autodesk! <3   
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: vortext on May 09, 2019, 01:20:25 PM
Quote from: tomvsotis on May 09, 2019, 06:04:28 AM
(Been busy w IRL stuff for ages, so I'm just going back through this thread now)

yeah ditto, been looking around for new job over the past month or so which was rather taxing with all the jumping through hoops and what not. Got a couple of offers and accepted one of them just yesterday! Super exciting, and while I still have to actually sign they already send me some flight dates to visit the offices abroad as part of 'meet n greet'. Say what now! All this is wildly off topic but I'm still reeling tbh  :D

At any rate, if things pan out how they're shaping up atm it looks like I'll be inbetween jobs during July at least, i.e. lots of free time. :thumbsup: 
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 09, 2019, 11:21:12 PM
Quote from: vortext on May 09, 2019, 01:20:25 PM
Quote from: tomvsotis on May 09, 2019, 06:04:28 AM
(Been busy w IRL stuff for ages, so I'm just going back through this thread now)

yeah ditto, been looking around for new job over the past month or so which was rather taxing with all the jumping through hoops and what not. Got a couple of offers and accepted one of them just yesterday! Super exciting, and while I still have to actually sign they already send me some flight dates to visit the offices abroad as part of 'meet n greet'. Say what now! All this is wildly off topic but I'm still reeling tbh  :D

At any rate, if things pan out how they're shaping up atm it looks like I'll be inbetween jobs during July at least, i.e. lots of free time. :thumbsup:

Oh, congratulations! I also need a new job; fingers crossed things pan out as well for me as they do for you!

As far as Bat4Blender goes, I've just installed the AMD Renderer, and I'm gonna use it to render a new model that I had been planning to export to 3ds Max 2017. We'll see how it goes!
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 09, 2019, 11:34:14 PM
Hey, just FYI, can't seem to get the Bat4Blender package to install on my Mac. Here's the error i'm getting:

(https://i.imgur.com/KUj0fiv.jpg)

It installs OK on Windows. This seems to be a Python issue with pathing... I'll see if I can find a way around it.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on May 10, 2019, 12:12:18 AM
Lol OK I've found the problem: I'm not sure what you used to ZIP these files, but for some reason MacOS was seeing each filename as including the relative path from Windows (I guess because *nix systems use a forward slash instead of a backslash to signify path separations, so Linux users may run into this, too):

(https://i.imgur.com/YPZ55cD.jpg)

Getting rid of the "Bat4Blender\" bits out of each file name, then re-zipping and installing, worked fine. I think this might be a problem with including an internal path in the ZIP file? Maybe just ZIP the files themselves?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: tomvsotis on September 01, 2019, 03:55:20 AM
Now that the forum is back:
- Blender 2.8 is out! For real! And it's really great! There's a good video about it here (http://"https://www.youtube.com/watch?v=MyxWDHy4ppY").
- God I hate 3ds max with the blazing rage of a thousand suns. I honestly don't know how you guys have been able to use it for so long! People had justifiable complaints w Blender pre-2.8's UI, but dear lord, 3ds max's UI is just as horrible, its workflows make no sense, and it's full of the sort of shortcomings that tend to pile up in monolithic pieces of software as the years go by. (You want to preview a material with a composite diffuse map in the viewport? Sorry, you can't. Why? Just.... because.) Seriously, if you've never given Blender a proper try, this is a really good time to do so.
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Barroco Hispano on September 01, 2019, 04:21:21 PM
@vortext Does the script work in blender 2.80?
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: eggman121 on September 02, 2019, 05:24:49 PM
I also think the new Blender is a great step froward.... However...

It lacks the ability to Import/ Export .3ds files

I still have Blender 2.79b for compatibility with making Models. I don't have access to 3ds MAX and once an i/o for .3ds format makes an arrival (For Blender 2.8x) we should be in a position to make some sweet models and keep the BATing continuing as well as opening up new avenues for BATers to make New models.

Exciting times ahead.

@alejogc13 I think that the script would need to be adjusted for Blender 2.8x (.3ds compatibility pending)

-eggman121
Title: Re: Using Blender (open source modeling program) for content creation.
Post by: Barroco Hispano on September 02, 2019, 05:55:08 PM
I think that if we don't find a solution in 1 or 2 years we will have to resort to unorthodox methods.