Sign in to follow this  
SYJourney

Should I use multithreading for sending messages?

Recommended Posts

Hello gamedev.net,

I am developing a client for an MMORPG in c++. So far I have not used any multithreading in my networking components. I simply receive messages on the same thread that updates my world state and I am also sending messages on this same thread. I've been wondering wether I should be serializing my messages on a seperate thread, or even have one thread for every message? I should note that I don't have concrete performance problems, it's more a question of what a better practice is.

I have read through the FAQ and used the search function, but found nothing that matches the specifics of my case.

 

I only have control of the client, I cannot change specifics of the server. The server uses TCP. Each message to the server has to be encrypted, and the key for encryption changes with every message. So If I were to send each packet on it's own thread, I would still need atomicity for that. For the most part, all the messages are fairly simple, they only require somewhere from 1-20 calls to serialization methods to construct, but they are frequent and there are hundreds of different message types. There are also some messages that need to read data from stl containers. Currently I can just pass these as references but I'm assuming that I would need copying if serialization happens on a different thread.

 

Any help is appreciated, it's a private project done for learning and I'm still a beginner with game making.

Edited by SYJourney

Share this post


Link to post
Share on other sites

Hello gamedev.net,

I am developing a client for an MMORPG in c++. So far I have not used any multithreading in my networking components. I simply receive messages on the same thread that updates my world state and I am also sending messages on this same thread. I've been wondering wether I should be serializing my messages on a seperate thread, or even have one thread for every message? I should note that I don't have concrete performance problems, it's more a question of what a better practice is.

I have read through the FAQ and used the search function, but found nothing that matches the specifics of my case.

 

I only have control of the client, I cannot change specifics of the server. The server uses TCP. Each message to the server has to be encrypted, and the key for encryption changes with every message. So If I were to send each packet on it's own thread, I would still need atomicity for that. For the most part, all the messages are fairly simple, they only require somewhere from 1-20 calls to serialization methods to construct, but they are frequent and there are hundreds of different message types. There are also some messages that need to read data from stl containers. Currently I can just pass these as references but I'm assuming that I would need copying if serialization happens on a different thread.

 

Any help is appreciated, it's a private project done for learning and I'm still a beginner with game making.

 

How can you be working on a (M)MO and not be able to modify the server?

 

You should never create a new thread for every message that comes in, because you'll spend most of your resources creating threads.

 

Check the performance of components and break them off into new threads. Usually a safe bet is to work on getting your Audio calls into a seperate thread.

Share this post


Link to post
Share on other sites

How can you be working on a (M)MO and not be able to modify the server?

 

You should never create a new thread for every message that comes in, because you'll spend most of your resources creating threads.

 

Check the performance of components and break them off into new threads. Usually a safe bet is to work on getting your Audio calls into a seperate thread.

 

 

It's an old game of which the server already exists. There is open source code for the server which I could edit, but I want to focus on just the client. I am doing this so that I can focus on just learning c++, and don't have to make choices related to game design.

The point about costs of creation makes sense, what about creating one thread at startup which I pass all arguments for my messages to?

Edited by SYJourney

Share this post


Link to post
Share on other sites


what about creating one thread at startup which I pass all arguments for my messages to?

 

That would be good, most likely, but it's impossible to know without benchmarking your code.

 

Typically, the actual networking aspects should be fine on 1 thread.

 

Is there any specific reason why you want to multi thread the serialization/serialization, or is just to learn?

Share this post


Link to post
Share on other sites

That would be good, most likely, but it's impossible to know without benchmarking your code.

 

Typically, the actual networking aspects should be fine on 1 thread.

 

Is there any specific reason why you want to multi thread the serialization/serialization, or is just to learn?

 

 

My reason for wanting to multithread is that blocking the main thread for serialization, encryption and passing the message to the socket seems a bit awkward/clumsy. I feel like it should return as soon as the (high level) data needed for the message has been obtained.

For example: If I click on an npc, which causes a message with the npc's identifier to be sent, I feel like it should return after finding out the identifier. But currently, it will serialize the identifier from an int to 4 bytes, then encrypt those bytes, then add a header, then pass the header and body to the socket before returning. Of course this is only my feeling as someone who is relatively new to programming.

 

I checked with the diagnosis tool from my IDE (Visual Studio) and the cpu cost of sending packets is very low. It's surprising to me because alot of messages are sent per second, and each has to be encrypted with two different algorithms.

So I am assuming that even If I were to do it better, the performance gain will be neglegible. But If I do end up having a measurable performance gain, I will report it back here for future reference.

Share this post


Link to post
Share on other sites
In general, there is very little overhead on a modern computer to recv/send in the main thread. Even processing/serializing the messages is unlikely to ever show up on a profiler for a typical game client protocol.
Note that send() just copies data from your buffer into an "outgoing buffer" in the kernel, and recv() just copies already-received data from the kernel "incoming buffer" into your buffer.
Thus, a reasonable client implementation could look something like:

while (running) {
  int r;
  sockaddr_in addr;
  socklen_t asize = sizeof(addr);
  while ((r = recvfrom(sock, buf, sizeof(buf), MSG_DONTWAIT, (sockaddr*)&addr, &asize)) > 0) {
    process_incoming_packet(buf, r, addr);
  }
  read_user_input();
  perhaps_update_simulation();
  render_graphics();
  if (now() - last_sent_time >= send_interval) {
    last_sent_time = now();
    r = form_outgoing_packet(buf, sizeof(buf));
    if (r != sendto(sock, buf, sizeof(buf), 0, &serveraddr, sizeof(serveraddr))) {
      error();
    }
  }
}
This sketch assumes UDP. TCP is slightly different, as you need to separate out your own packet boundaries and retain what's left in the buffer between reads.

Anyway, in games, the physics simulation, graphics rendering, perhaps audio, and sometimes the AI, may benefit from separate threads, but networking generally doesn't.
If and when it does, you'll want one single thread for networking, and have a well-defined (and well tested!) communication mechanism between that thread and the threads that need the data.

Threads help in one of two cases:
- you do computation that has well-defined input and output dependencies to the rest of the program AND that computation takes "significant work" (milliseconds or more.)
- you have some event that needs absolutely lowest latency (such as VR scanline-chasing rendering, or audio generation) and run the thread at real-time priority

Threads are sometimes also added to work around cases where proper asynchronous I/O is not available, such as reading from files on Linux. You can build a worker thread that accepts "read file" requests, and posts results back to whoever started the request when done. This isn't needed for networking, because non-blocking or select-based or event-based socket I/O is sufficiently asynchronous.

Share this post


Link to post
Share on other sites

Threads help in one of two cases:

- you do computation that has well-defined input and output dependencies to the rest of the program AND that computation takes "significant work" (milliseconds or more.)
- you have some event that needs absolutely lowest latency (such as VR scanline-chasing rendering, or audio generation) and run the thread at real-time priority

Threads are sometimes also added to work around cases where proper asynchronous I/O is not available, such as reading from files on Linux. You can build a worker thread that accepts "read file" requests, and posts results back to whoever started the request when done. This isn't needed for networking, because non-blocking or select-based or event-based socket I/O is sufficiently asynchronous.

 

 

What would you recommend for a server that's running multiple standalone rooms/arenas (no transfer of users between rooms)? In C# I'm envisioning a system that runs each arena on its own thread and has a separate dedicated network I/O thread. C# provides a built-in thread-safe ConcurrentQueue collection that would be useful for passing messages back and forth.

 

I'm wondering if a single network I/O thread (handling UDP traffic for all of the arenas, and possibly a second TCP connection for RCON) would be a bottleneck, or if I should run each arena with its own port and have it do its own I/O. If I'm running on a 1-2 core VPS, threading might also be unnecessary overhead. I could run everything on the main thread (recv, update arena 1, update arena 2, update arena 3, ..., send) as well, but that seems a little brittle.

Edited by sufficientreason

Share this post


Link to post
Share on other sites

What would you recommend for a server that's running multiple standalone rooms/arenas (no transfer of users between rooms)? In C# I'm envisioning a system that runs each arena on its own thread and has a separate dedicated network I/O thread. C# provides a built-in thread-safe ConcurrentQueue collection that would be useful for passing messages back and forth.

 

I'm wondering if a single network I/O thread (handling UDP traffic for all of the arenas, and possibly a second TCP connection for RCON) would be a bottleneck, or if I should run each arena with its own port and have it do its own I/O. If I'm running on a 1-2 core VPS, threading might also be unnecessary overhead. I could run everything on the main thread (recv, update arena 1, update arena 2, update arena 3, ..., send) as well, but that seems a little brittle.

 

Generally you want the amount of threads you have match the amount of cores you have available, or cores minus one. In the case you describe you'd evenly divide the amount of arenas over the amount of threads you have. There's no need for thread safe queues if there's no communication taking place between the arenas. You can do away with any complexity by having a single socket per thread that's shared among all arenas on that thread.

Edited by Mussi

Share this post


Link to post
Share on other sites

What would you recommend for a server that's running multiple standalone rooms/arenas (no transfer of users between rooms)?


I'd build a single process that can deal with a single room, and some process-manager that can spawn these processes as needed when rooms get created.
That way, if one room crashes, it doesn't affect any of the other rooms on that same machine.

If that's somehow totally impossible to build, then one thread per room is fine. (Or at least fine-ish.) You don't need an I/O thread in C#, as C# has both good support for overlapped I/O coming from its Windows heritage, and language support for async/await.
If you run C# on Linux, the runtime will emulate the overlapped I/O for you using its own threading system.

Share this post


Link to post
Share on other sites

Generally you want the amount of threads you have match the amount of cores you have available, or cores minus one. In the case you describe you'd evenly divide the amount of arenas over the amount of threads you have. There's no need for thread safe queues if there's no communication taking place between the arenas. You can do away with any complexity by having a single socket per thread that's shared among all arenas on that thread.

 

Makes sense. RCON access complicates this a little since you might want to query different arenas specifically without disconnecting and reconnecting to different ports. Otherwise that seems like a good approach. Either my matchmaking service will have to know what port to send players to or I'll have to run a thread on a separate socket on the game server that redirects new connections to the appropriate arena port.

 

 

I'd build a single process that can deal with a single room, and some process-manager that can spawn these processes as needed when rooms get created.

That way, if one room crashes, it doesn't affect any of the other rooms on that same machine.

 

This is also a good alternative, though it suffers from the redirection and RCON issues above. The one downside to this as opposed to multiple threads is that in this case each process would need to load the static read-only game data individually.

Edited by sufficientreason

Share this post


Link to post
Share on other sites
If you have lots of static data, that problem can be solved.
If the rooms are all spawned by the same parent process, you'd load the read-only data in the parent process before forking the child rooms. The OS will then map that data as copy-on-write, so it's actually shared between the child processes.
(Or you could do it manually with shared memory and/or file mapping, if you're an experienced systems programming engineer :-)

Yes, each room needs to be somehow coordinated with the matchmaker.

Share this post


Link to post
Share on other sites

If the rooms are all spawned by the same parent process, you'd load the read-only data in the parent process before forking the child rooms. The OS will then map that data as copy-on-write, so it's actually shared between the child processes.

 

Something tells me I have some reading to do. Thanks though, I didn't realize this was possible. The independent process method is probably the way to go then for maximum safety once I figure out how to do this memory sharing in C#. That parent process also would provide a single point of contact for matchmaking and RCON.

Edited by sufficientreason

Share this post


Link to post
Share on other sites

It sounds like you're doing a lot of "empty work" -- you mentioned that you serialize an int into 4 bytes, but an int is already 4 bytes! Unless the protocol specifies some weird way of passing integers, or requires a different endianness than your machine, there's really no serialization necessary there.
 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this