Network Protocol

[croxis]

Draft:
The network protocol is the communication between the client(s) and the server.  Due to the unique nature of the genera it is probably best to develop our own protocol.  The tool used to develop this protocol is Protocol Buffers which have libraries for Java, C++, Python, as well as 3rd party libraries for C# and others.  ProtoBufs are designed for forward and backwards compatibility.  This gives wiggle room for different client and server versions and reduce version mismatch hell that can be common for open source games.  It also serializes to binary reducing network traffic.  My objective is to make the game playable on a 56k modem.

Protocols are defined in .proto files with syntax similar to C++ or java. These files are then compiled into classes native to the language in question.  The appropriate class attributes are defined, the protocol is serialized to a string using hex binary, and then is deserialized on the other end.

Example .proto file

Code Select Expand

message Container {
    optional Chat chat = 1;
}

message Chat {
    //Chat message
    required string message = 1;
    required string from = 2;
    optional string to = 3;
    extensions 100 to 199;
}

extend chat {
    optional int32 bar = 100;


Let us look at the chat message.  message Chat will be compiled into an object with different attributes. required string message =1: Required means the attribute is required for serialization.  Required attributes can not be removed from .proto files without breaking backward comparability. String defines the type, message is the variable name, and = 1 is the binary tag used for encoding.  The tag can also not be changed without breaking compatibility.  Optional arguments are exactly that, and will not take any space in the wire if not defined.  Extensions allow the possibility of extending a message.  This allows 3rd party people to extend the protocol with their own .proto files.  This allows the original to stay intact.  This will be useful for mods which may need additional communication.

As a serialized message is not self identifying there needs to be a message container to rule them all.  The Container message does just that.  Messages can be nested, so the Container will have optional message lines for every message that can be sent.  Extensions can be useful here for third party messages, as well as organizing messages into ranges based on their function (100-199 for building, 200-299 for stats updates, 300-399 for disasters, etc.  or 100-499 for client specific, 500-899 for server specific, or 900-999 for common messages). Protocol definitions will be developed in this thread.

Side note: Protobuf can also be used for save game format as well.

Quick FAQ:
Why not XML?
XML is good for only two things, human readability and data interoperability.  It doesn't translate well into object oriented programming and writing xml parsers are a pain.  They serve a function, such as saved games, but not for network communication.  All those extra <> and stuff needs to be communicated and adds network overhead.

Why not YAML then?
I love yaml!  It works very well with python and I was considering it for serialization.  In fact python objects can be serialized into yaml directly.  The problem is someone on the other side might not be using python so data structures native to python might not translate.  Also there is no native forward or backwards compatibility.  Because it serializes to an ascii string

JSON is out too then?
I have not personally used JSON but it was also a contender for serialization.  Same issues though.

[croxis]

Reserved for CityMania specs

[Sartoris]

Are you looking into using UDP?

Would WCF (windows communication foundation) be comparable to ProtoBufs?
Sorry for the questions, I am just curious as to how the mixed codebase (C# and Python) is going to interact. I have never done a project using more than one language at a time. (C# or C++ or Java)

[croxis]

I don't see the need for UDP as the data isn't time sensitive.

From what I am reading WCF isn't comparable.  Protobuf is just a data container which also serializes into a binary string.  How that is sent, stored, or used is up to the native code.  It is then deserialized into the native code.

[JoeST]

ProtoBuffs does look like an awesome way to go, I read about it a while ago, and it seems quite awesome

Joe

[Sartoris]

I understand now. ProtoBuffs is like Thrift.

[croxis]

Right, the difference is that Thrift is more for RPC