Network message solution for client/server architecture

Started by
3 comments, last by rpiller 11 years, 3 months ago

Hello,

What are most common approaches for solving network messaging issue for client/server architecture (more towards online RPG game than first person shooter, so amount of messages is much bigger because there is a lot more going on in the world)? By message I mean communication between client and server (back and forth).

First method - Message classes

So far I explored Ryzom and Planeshift MMOs sources and they seem to solve this by having a shared code between client and server that has a Message class that is then inherited by every message (ClientLoginMsg, CharacterSelectMsg, GetInventoryMsg). Obviously each message will be used in two ways - the sender will serialize message into bitstream thats sent over the network, and receiver will deserialize it into something usable (class members, data structures etc.). So it looks a bit like this in pseudo-code:


class Message
{
}

class LoginAuthorizeMessgae: Message
{
    string user;
    string pass;
    BitStream out;

    // Assume that BitStream has already MESSAGE_ID read from it, so all thats left is message content
    // Because we read MESSAGE_ID, dispatcher knows which message to instantiate and it can be sent to
    // proper handlers
    LoginAuthorizeMessage(BitStream* in)
    {
       in.read(&user);
       in.read(&pass);
    }

    LoginAuthorizeMessage(string username, string password): user(username), pass(password);
    {
       out.write(MSG_AUTHORIZE_ID);  // write message id
       out.write(user);
       out.write(pass);
    }
}

// Then, client side:

void SendAuthRequest(login, pass)
{
    LoginAuthorizeMessage msg(login, pass);
    dispatcher->Send(msg);
}

// Server side:

void ReceiveAuthRequestHandler(Message* msg)
{
    LoginAuthorizeMessage* msg = (LoginAuthorizeMessage*)msg;
    if (db->auth(msg->user, msg->pass)
    {
       // ...
    }
}


Problem with this approach seems to be insane number of classes that need to be maintained. Just creating a simple response message with one text field requires declaring class with 2 constructors, then its .cpp file etc. Pros of this method is that same messages can be used by both, receiver and sender, so the code can be shared between client and server. All the serialize/deserialize logic is in one place, so its easier to maintain and harder to make some mistake because you see code for both, serialization and deserialization.

I came to even more advanced solution than the one above - I have a factory class that can create an instance of a correct class based on MESSAGE_ID, then I have an event dispatcher that is able to send that exact instance to receivers. A bit of a code snippet to demonstrate:


// =================
// NetworkManager
// =================

NetworkManager()
{
   m_MsgFactory->Register<SampleMessage>(NetMessageType::MSG_SAMPLE);
}

NetworkManager::HandlePacket(Packet* packet)
{
   msgID = GetPacketMsgId(packet);
 
   // Create a class thats registered for a given msgID
   // - for example for MSG_SAMPLE it will instantiate SampleMessage class and pass
   //   bitstream from packet.data into its constructor

   NetMessage* msg = m_MsgFactory->Create(msgID, packet.data);

   // This call sends that message to methods that are registered to handle it
   // I register methods instead of msgID, so the method is called with exact
   // class that it needs (so SampleMessage, instead of Message). It doesn't 
   // require explicit casting in handler method)

   m_MsgDispatcher->DispatchEvent(msg);
}

// =================
// SampleMessage
// =================

class SampleMessage: public NetMessage
{
public:
   int a;
   float b; 
 
  SampleMessage(RakNet::BitStream& msg): SampleMessage()
   {
      msg.Read(a);
      msg.Read(b);
   }

   SampleMessage(): NetMessage(NetMessageType::MSG_SAMPLE)
   {
      // This sets MSG_SAMPLE id for this message, so we 
      // dont have to care about it later
   }
};

// ====================
// SampleMessageHandler
// ====================
bool Init(NetMessageDispatcher& dispatcher)
{
    dispatcher.Subscribe(this, &SampleMessageHandler::HandleSampleMsg);
    return true;
}

void HandleSampleMsg(const SampleMessage* msg)
{
    std::cout << "Received sample message: " << msg.a << "\n";
} 

This works really nicely, but its probably a bit slower than some direct method (that clever event dispatcher that sends exact instance instead of passing Message* which allows me to register methods that receive directly what they want is probably doing some casting that could be avoided, although these are static_casts). Also it doesn't help with a problem of enormous amount of very small classes that may be hard to keep in order.

Second method - sendXX methods to serialize some data to bitstream, and directly reading from message in receiver

This is method I've seen in some SWG emulator, but it was emulator and I only saw server code, so don't know how well it would work with client and if it was shared in any way or it was separate. In that code, there was a huge MsgLib.h file that defined a lot of methods like:



void sendPlayerPosition(Player*, Client*)
void sendServerTime(double time, Client*)
void sendInventory(Inventory*, Client*)
void sendChatMessage(message, channel, Client*)

and so on. Each of this methods serialized required data into a network message (msg.addUint32, msg.addString, msg.addFloat etc.) and sent it directly using provided Client.


void HandlePlayerPosition(Message* msg)
{
   msg.getFloat(&player.x);
   msg.getFloat(&player.y);
   msg.getFloat(&player.z);
}

This means that writing and reading happens in totally different places (server or client) so code is not really shared, all thats shared is opcodes or message ids because both sides need to be aware of them.

The question

So, the question is, are there other methods? Are methods I described good enough and what can be done better? Am I doing something really wrong here? I find it hard to gather any knowledge on this topic, there seem to be no books on this, and only help were source codes of these three MMOs I mentioned at the beginning.

Thanks for any input! It would be very interesting to see how others solve this.


Where are we and when are we and who are we?
How many people in how many places at how many times?
Advertisement

There are two different things here.

The first aspect is the serialized data you send across the wire. In both of the examples you sent the data across the wire is basically identical, in the form {ID, content}. It doesn't matter what the code looks like as long as it follows the protocol.

The second aspect is the code organization. Your first example calls the functions directly and the second example uses an event broadcaster/listener pattern. Those are the usual ways of doing it. Both patterns are widely used, and both patterns have their own benefits and drawbacks. You can even mix-and-match the two. Use whatever fits your design best.

Yeah, I'm interested more in the second aspect - code organization and design. The "internal" message format which is what's sent through the wire doesn't concern me - I use RakNet's BitStream for this, someone can use his own class or other lib. What I'm interested in is way to encapsulate this "raw" data into some higher level Message like the ones I shown above.

These benefits and drawbacks is what I want to discuss.


Where are we and when are we and who are we?
How many people in how many places at how many times?

First, note that I mainly use C# together with XNA and Unity3D, so maybe the rules for games and engines using C++ are different. Anyway, I strongly prefer the first option. While there are quite a few classes that needs to be created, I do not see this as a major drawback. It creates a very clear separation of "data that is put on the wire" and the other code which deals with this data. Everything gets encapsulated into one neat little package. And I suspect that even if you use something which is more akin to your second example (the SWG emulator one), you would still end up with some sort of encapsulation similar to an "Message" or equivalent to be able to do reliable sends, sequencing, etc.

Just as an example, this is a "Message" (my networking lib calls them "Events") which is used to send the click position and target object in a physics based puzzle game:


public class ShootBallEvent : NetEvent
{
    public static readonly ushort Id = 0;

    public NetObject Target;
    public Vector3 Position;

    public ShootBallEvent()
        : base(Id, NetEventDelivery.Unreliable, NetEventDirection.ClientToServer)
    {

    }

    public override void Pack(NetConnection conn, NetPacket packet)
    {
        packet.WriteVector3(Position);
        packet.WriteNetObject(Target);
    }

    public override void Unpack(NetConnection conn, NetPacket packet)
    {
        Position = packet.ReadVector3();
        Target = packet.ReadNetObject();
    }

    public override void Process(NetConnection conn)
    {
        if (Target == null)
            return;

        if (!ReferenceEquals(Target.Owner, conn))
            return;

        // ... do stuff to Target and Position
    }
}

I find this encapsulation to be very clear and easy to work with.

I use RakNet with BitStream's as well, and I fire "events" with messages, but I just go raw BitStream & Packet handling instead of wrapping them up in a formal message object. There are so many messages in a game that making an object for each one I just felt wasn't worth it. I'd rather just have a function be mapped to a message and then read the BitStream directly in there. The Event.h file I use is below. I do this so 1 message gets mapped to 1 function and I don't have to worry about messing with some switch control. Instead it's always just 1 mapping line of code.

Events are stored like:


map<MessageID, Event2<Packet&, BitStream&>> _events;

I have 1 area that captures packets coming in from the RakNet loop to dispatch to member functions that are registered to the message:


for(_packet = _peer->Receive(); _packet; _peer->DeallocatePacket(_packet), _packet = _peer->Receive())
{
RakNet::MessageID msg = _packet->data[0];


// make sure we mapped to the event
if(_events.find(msg) != _events.end())
{
// create bitstream and ignore the msg
RakNet::BitStream reader(_packet->data, _packet->length, false);
reader.IgnoreBytes(sizeof(RakNet::MessageID));


// raise the event
_events[msg].Raise((*_packet), reader);
}
}

And I register the functions like:


_events[(MessageID)ID_NEW_INCOMING_CONNECTION].Bind(this, &Server::OnNewIncomingConnection);

The function using the data would be defined as:


void Server::OnNewIncomingConnection(Packet& p, BitStream& reader)
{
    int age;


   reader.Read(age);


   _peer->Send(..., p.address);
}

To me the BitStream is basically the "message" object. It just isn't wrapped in a type. The plus is it doesn't get the overhead wrapping hit that a type does, if the message changes you don't have to go 2 places for that change (the message object, then in here to use new field), & you won't have potentially hundreds of little classes laying around (although if you have multiple people on a team this might help separate duties). The only negative is you don't get a pretty type that defines all the fields that the object has. You have to do that yourself. I'm not sure of the speed hit that would occur with the message object, but it'll be "slowER" than this, but it might not matter.

Event.h. This has EventX objects where X is 0 - 3 which is the number of parameters to the class function.


#pragma once
#include <list>


using namespace std;




class TFunctor0
{
public:
virtual void Call()=0;
};


template <class TClass>
class TSpecificFunctor0 : public TFunctor0
{
private:
void (TClass::*fpt)();
TClass* pt2Object;         
public:
TSpecificFunctor0(TClass* _pt2Object, void(TClass::*_fpt)())
{ 
pt2Object = _pt2Object;  
fpt=_fpt; 
}


virtual void Call()
{ (*pt2Object.*fpt)(); }
};


class Event0
{
public:
list<TFunctor0*>  mCaller;


template<class Target>
void Bind(Target* t, void (Target::*fnPtr)())
{ 
mCaller.push_back(new TSpecificFunctor0<Target>(t,fnPtr));
}


void Clear()
{ mCaller.clear(); }
       
void Raise()
{
list<TFunctor0*>::reverse_iterator iter;


for (iter = mCaller.rbegin(); iter!= mCaller.rend(); iter++)
{
(*iter)->Call();
}
}
};


//===============================================================================


template<class P>
class TFunctor1
{
public:
virtual void Call(P var1)=0;
};


template <class TClass, class param1>
class TSpecificFunctor1 : public TFunctor1<param1>
{
private:
void (TClass::*fpt)(param1);
TClass* pt2Object;         
public:
TSpecificFunctor1(TClass* _pt2Object, void(TClass::*_fpt)(param1))
{ pt2Object = _pt2Object;  fpt=_fpt; }


virtual void Call(param1 var1)
{ (*pt2Object.*fpt)(var1); }
};


template<class T1>
class Event1
{
public:
list<TFunctor1<T1>* >  mCaller;


template<class Target>
void Bind(Target* t, void (Target::*fnPtr)(T1))
{ mCaller.push_back(new TSpecificFunctor1<Target, T1>(t,fnPtr)); }
       
void Raise(T1 V1)
{
list<TFunctor1<T1>*>::reverse_iterator iter;




for (iter = mCaller.rbegin(); iter!= mCaller.rend(); iter++)
{
(*iter)->Call(V1);
}
}


void Clear()
{
mCaller.clear();
}
};


//===============================================================================


template<class P, class Q>
class TFunctor2
{
public:
virtual void Call(P var1, Q var2)=0;
};


template <class TClass, class param1, class param2>
class TSpecificFunctor2 : public TFunctor2<param1, param2>
{
private:
void (TClass::*fpt)(param1, param2);
TClass* pt2Object;         
public:
TSpecificFunctor2(TClass* _pt2Object, void(TClass::*_fpt)(param1, param2))
{ pt2Object = _pt2Object;  fpt=_fpt; }


virtual void Call(param1 var1, param2 var2)
{ (*pt2Object.*fpt)(var1, var2); }
};


template<class T1, class T2>
class Event2
{
public:
list<TFunctor2<T1, T2>* >  mCaller;


template<class Target>
Event2(Target* t, void (Target::*fnPtr)(T1, T2))
{ mCaller.push_back(new TSpecificFunctor2<Target, T1, T2>(t,fnPtr)); }


Event2(){}


template<class Target>
void Bind(Target* t, void (Target::*fnPtr)(T1, T2))
{ mCaller.push_back(new TSpecificFunctor2<Target, T1, T2>(t,fnPtr)); }
       
void Raise(T1 V1, T2 V2)
{
list<TFunctor2<T1, T2>*>::reverse_iterator iter;




for (iter = mCaller.rbegin(); iter!= mCaller.rend(); iter++)
{
(*iter)->Call(V1, V2);
}
}


void Clear()
{
mCaller.clear();
}
};




//===============================================================================


template<class P, class Q, class Y>
class TFunctor3
{
public:
virtual void Call(P var1, Q var2, Y var3)=0;
};


template <class TClass, class param1, class param2, class param3>
class TSpecificFunctor3 : public TFunctor3<param1, param2, param3>
{
private:
void (TClass::*fpt)(param1, param2, param3);
TClass* pt2Object;         
public:
TSpecificFunctor3(TClass* _pt2Object, void(TClass::*_fpt)(param1, param2, param3))
{ pt2Object = _pt2Object;  fpt=_fpt; }


virtual void Call(param1 var1, param2 var2, param3 var3)
{ (*pt2Object.*fpt)(var1, var2, var3); }
};


template<class T1, class T2, class T3>
class Event3
{
public:
list<TFunctor3<T1, T2, T3>* >  mCaller;


template<class Target>
void Bind(Target* t, void (Target::*fnPtr)(T1, T2, T3))
{ mCaller.push_back(new TSpecificFunctor3<Target, T1, T2, T3>(t,fnPtr)); }
       
void Raise(T1 V1, T2 V2, T3 V3)
{
list<TFunctor3<T1, T2, T3>*>::reverse_iterator iter;




for (iter = mCaller.rbegin(); iter!= mCaller.rend(); iter++)
{
(*iter)->Call(V1, V2, V3);
}
}


void Clear()
{
mCaller.clear();
}
};

This topic is closed to new replies.

Advertisement