Linux server architecture
Author
im currently porting my mud server code to linux and I have a couple of questions:
If I were to fork() every client would the server be just as ineffient as a winsock server which spawnes a thread for every client?
Should I rather stick to a process which polls select in one big loop using nonblocking sockets?
I would like to be able to handle a couple of hundred simultanious connections.
any feedback will be appreciated :-)
Don''t fork, it makes the debugging of a multiplayer game VERY, VERY hard.
Why not using SDL_net instead? That''s what I do, for my mmorpg (the server is hosted on a Linux machine, but it also works without any modifications on win32).
SDL_net is fast, simple, powerful and portable. And LGPL
Height Map Editor | Eternal Lands | Fast User Directory
Why not using SDL_net instead? That''s what I do, for my mmorpg (the server is hosted on a Linux machine, but it also works without any modifications on win32).
SDL_net is fast, simple, powerful and portable. And LGPL
Height Map Editor | Eternal Lands | Fast User Directory
I''m no Unix guru, but I don''t think you want to call fork() for every client that connects. Posix threads might be a little bit better, but still not an optimal solution given the number of connections you''re talking about. To the best of my knowledge, select() would probably be the way to go. There''s a certain amount of overhead in setting up the descriptor sets, but atleast it''s simple and straight forward to implement.
I suggest you combine select() and pthreads if you really want your app to run at full speed. I would set up several (3-10, depends on the machine configuration on which you run your linux server) processing threads + one receiving thread + one sending thread + one control thread.
Processing threads are to process user info, update the server databases and forward the reply to the sending thread.
The sending thread is to manage client connections and send the forwarded data.
The receiving thread is to manage client connections and forward incoming data to the free processing thread.
The control thread is to manage the work of all the threads - your ''admin tool and network interface to your app''.
I would set up incoming and outcoming data queues for receving and sending threads and manage them with mutexes or smth.
And each of the threads should contain queues of data to be processed, that will help to manage bandwidth and overall server load.
That''s it. You can even use processes instead of threads under linux, but i think it would be harder for them to intercommunicate (threads share dynamically allocated memory by default, processes don''t).
"I''ll be Bach!" (c) Johann Sebastian Schwarzenegger
Processing threads are to process user info, update the server databases and forward the reply to the sending thread.
The sending thread is to manage client connections and send the forwarded data.
The receiving thread is to manage client connections and forward incoming data to the free processing thread.
The control thread is to manage the work of all the threads - your ''admin tool and network interface to your app''.
I would set up incoming and outcoming data queues for receving and sending threads and manage them with mutexes or smth.
And each of the threads should contain queues of data to be processed, that will help to manage bandwidth and overall server load.
That''s it. You can even use processes instead of threads under linux, but i think it would be harder for them to intercommunicate (threads share dynamically allocated memory by default, processes don''t).
"I''ll be Bach!" (c) Johann Sebastian Schwarzenegger
Author
X33: thank very informative reply.. I sorry about the thing I said about processes (I was high that day). Of course using processes would be idiotic because they doesnt share the same memory space and a MUD heavily relies on common data.
Your "threaded" idea sounds very interesting but sounds a bit like synchronization hell.. what thread is to call select()? and both the receive and send thread must use the same socket set.
I need to time so that select is called then the send and receive threads does one loop.. then select is called again.. how do I do that?!? the smartest would be to place the call to select in the threads but it would be overhead to first call select with only the read set.. and then again with only the write set when you can call it with both sets..
Your "threaded" idea sounds very interesting but sounds a bit like synchronization hell.. what thread is to call select()? and both the receive and send thread must use the same socket set.
I need to time so that select is called then the send and receive threads does one loop.. then select is called again.. how do I do that?!? the smartest would be to place the call to select in the threads but it would be overhead to first call select with only the read set.. and then again with only the write set when you can call it with both sets..
I''ve been trying to implement what x33 described for some time now and youre right it is syncronization hell, but I think its one of the better setups you can do. I''ve not gotten it working yet so I cant really offer any advice on implementing it, but if and when i do, I think im gonna write a full blown tutorial on it. =)
-=[ Megahertz ]=-
-=[ Megahertz ]=-
Well, first of all, pthreads are implemented through processes under Linux... =)... Try running a threaded app and type 'ps' in another virtual console - you'll see several processes (the actual number of processes is equal to 'parent thread + child threads + control thread (it is spawned by the kernel)'... The control thread is managing processes and its work is absolutely transparent for the programmer. You don't have to worry about it, it's just for your convenience =). Actually processes and pthreads are implemented through the unified kernel 'clone()' method but that one is hard to manage and control =)... I would recommend you reading the manpages if you need more info about how things are organized...
You don't have to worry about using the same socket descriptors in both sending and receiving threads, because socket descriptors are similar to file descriptors (you can even cast your socket descriptor to a file descriptor and use write() and read() on it! =)), and those descriptors are also shared between the threads. Just start your sockets under the parent thread and then spawn as many child threads as you may need... they will inherit all the descriptors from the parent thread... You can even run the receiving and sending code in one thread, so that shouldn't be a problem...
So... if I were you, I would set up things in the following way:
I would set up three types of threads: parent (main) thread, processing threads and threads responsible for communication.
1. The parent thread should be responsible for setting up all the resources, running a socket, creating and managing child threads, processing admin commands, setting up logging and do other preprocessing things.
2. The communication thread should run select() in a loop and manage incoming and outcoming data + listen for new connections and handle other communication thing. It also has to be best 'secured and stable'. You need to log each connection, request, etc... So... For example, if some data is received, the connection thread should check whether it is a connection request or a user command. If it is a connection request then it should be accepted and the client descriptor should be added to the main set of descriptors and passed to the select() during the next turn of the main loop. If we get some connected user data we need to check for integrity, authorize and secure it and then place it in the incoming data queue... If we the select only returns with 'ready to write' descriptors in the descriptor set, then we should check the outcoming queue for data addressed to the proper client, check its actuality (compare its time stamp with the current server time, etc...), compile a packet and transmit it to the client...
You will also have to build your headers on top of the protocol your app utilizes and a messaging system depending on the kind of your app.
The queues should be protected with mutexes and condition variables so that there's no blocking, dead-locking or race conditions... Consider stable server work (not speed) your priority. As a gamer I would rather like to play on a slow but stable server than reconnecting every 5 seconds... =) But it would be great to play on a both stable and fast server ;].
It is not actually a synchronization hell, everything is clear and stable (works fine under my linux and freeBSD boxes =)). Depends on your design and structure. Mutexes really help to avoid problems with synchronization, and they are quite clear and easy-to-use.
3. The processing threads run in endless loops. They check for data in the incoming queue, process it according to some rules and then place the replies, time stamps, IDs, status or error statements and other info into the outcoming queues, so that clients on the other side are 'up-to-date'. You can assign each thread a region of your game world to be responsible for, or set up a quad tree to manage huge server load, or create something of your own... it is up to you to decide.
Try implementing a simple app of this structure, testing to see if everything is ok, and then develop and upgrade it to a full-powered server.
If you still have some questions on the design, post more info about your requirements and more about the app you're trying to design... i'll try and help. =)
"I'll be Bach!" (c) Johann Sebastian Schwarzenegger
[edited by - x33 on April 3, 2003 2:35:39 PM]
You don't have to worry about using the same socket descriptors in both sending and receiving threads, because socket descriptors are similar to file descriptors (you can even cast your socket descriptor to a file descriptor and use write() and read() on it! =)), and those descriptors are also shared between the threads. Just start your sockets under the parent thread and then spawn as many child threads as you may need... they will inherit all the descriptors from the parent thread... You can even run the receiving and sending code in one thread, so that shouldn't be a problem...
So... if I were you, I would set up things in the following way:
I would set up three types of threads: parent (main) thread, processing threads and threads responsible for communication.
1. The parent thread should be responsible for setting up all the resources, running a socket, creating and managing child threads, processing admin commands, setting up logging and do other preprocessing things.
2. The communication thread should run select() in a loop and manage incoming and outcoming data + listen for new connections and handle other communication thing. It also has to be best 'secured and stable'. You need to log each connection, request, etc... So... For example, if some data is received, the connection thread should check whether it is a connection request or a user command. If it is a connection request then it should be accepted and the client descriptor should be added to the main set of descriptors and passed to the select() during the next turn of the main loop. If we get some connected user data we need to check for integrity, authorize and secure it and then place it in the incoming data queue... If we the select only returns with 'ready to write' descriptors in the descriptor set, then we should check the outcoming queue for data addressed to the proper client, check its actuality (compare its time stamp with the current server time, etc...), compile a packet and transmit it to the client...
You will also have to build your headers on top of the protocol your app utilizes and a messaging system depending on the kind of your app.
The queues should be protected with mutexes and condition variables so that there's no blocking, dead-locking or race conditions... Consider stable server work (not speed) your priority. As a gamer I would rather like to play on a slow but stable server than reconnecting every 5 seconds... =) But it would be great to play on a both stable and fast server ;].
It is not actually a synchronization hell, everything is clear and stable (works fine under my linux and freeBSD boxes =)). Depends on your design and structure. Mutexes really help to avoid problems with synchronization, and they are quite clear and easy-to-use.
3. The processing threads run in endless loops. They check for data in the incoming queue, process it according to some rules and then place the replies, time stamps, IDs, status or error statements and other info into the outcoming queues, so that clients on the other side are 'up-to-date'. You can assign each thread a region of your game world to be responsible for, or set up a quad tree to manage huge server load, or create something of your own... it is up to you to decide.
Try implementing a simple app of this structure, testing to see if everything is ok, and then develop and upgrade it to a full-powered server.
If you still have some questions on the design, post more info about your requirements and more about the app you're trying to design... i'll try and help. =)
"I'll be Bach!" (c) Johann Sebastian Schwarzenegger
[edited by - x33 on April 3, 2003 2:35:39 PM]
Author
x33:
hmm makes sense.. the I would monitor the listening socket in my main thread.. accept connections and hand them of to one of the communication threads.. which fed the incoming data queues and sends the data in the outgoing queues.. the process thread would then process the data in the incoming queue and write to the outgoing.. brilliant.. but why have more than one process thread? and how many connections should I manage per communication thread?.
also the advantages to this architecture will only be that it will scale on SMP''s right? I mean threads causes overhead on a uni processor system so would it not always be most efficient to put it all in the mainthread on a uni processor system?
hmm makes sense.. the I would monitor the listening socket in my main thread.. accept connections and hand them of to one of the communication threads.. which fed the incoming data queues and sends the data in the outgoing queues.. the process thread would then process the data in the incoming queue and write to the outgoing.. brilliant.. but why have more than one process thread? and how many connections should I manage per communication thread?.
also the advantages to this architecture will only be that it will scale on SMP''s right? I mean threads causes overhead on a uni processor system so would it not always be most efficient to put it all in the mainthread on a uni processor system?
quote:Original post by x33
Well, first of all, pthreads are implemented through processes under Linux...
Again, I''m not really up to speed on Linux programming, but I thought more recent kernels included a native POSIX threads implementation. The latest version of Red Hat Linux makes reference to the NPTL in its release notes. Surely a native implementation wouldn''t be built on top of processes?
Once again... the basic structure of such an application is: the client connects to a server, then begins communication. For example, a gamer want to move his unit from point A to point B on the map. So, he just selects the proper unit and then points it to the destination. But the other clients need to see whether that unit has moved or not. So his client part sends data which contains fields similar to this: [UID][UNUMBER][COMMAND][ARGS]. UID - is a user (client) ID.
UNUMBER - the number of the object client is operating on.
COMMAND - is the action for the object to perform (0 == move, 1 == attack, 2 == take, etc...)
ARGS - is the field containing additional data for the user command to be properly processed by the server (if COMMAND == 0(move), then ARGS should contain coordinates of the destination point).
Depending on the type of data the client can wait for confirmation or rejection back from the server... For example, if the server couldn''t move a unit to the destination point, it sends a ''reject'' packet... and on the client side the unit says something like ''I can''t go there, Master!''... If it is a confirmation - the unit says ''As you wish, Master!'' and begins to move...
And on the server side the logic is more complex as it has to process and calculate everything except graphics (graphics rendering is the task of the client part): AI, game world and map changes, coordination and user interaction. etc... there''s a huge amount of work for the server.
Depending on the result of its calculations server needs to decide whether to send a ''confirm'' or a ''reject'' packet. Now imagine you have 800 clients connected to your server... As there''s only one client request being processed in the moment the other clients would have to wait until their commands are processed. For example imagine it takes 0.3 seconds for the server to process a certain command. And one of the clients has sent data. As long as the server is only capable to process one command at a time, every client has to wait (0.3 * 800 = 24 seconds) until his next command is processed... Well, I think a gamer wouldn''t be happy to play in such an environment... =) So the server needs to make its calculations as fast as possible and to process as many requests as possible at a time. That''s why we need several processing threads. So... there server needs to make an illusion of simultaneous processing. That''s it! We have to run several processing threads. They are not really simultaneous, but the gamer won''t see that... He will think the the server is only processing his input, and nobody else...
As for the connection threads, i would recommend using only one thread to manage connections and send/receive data, because the only thing it should do - is verify the data and copy the buffers to the proper queues.
Search for Beej''s Guide to Network Programming - it will tell you how to build a loop with a select() (see the ''multiplexing with select()'' section of the guide) and how to manage different types of incoming data (data from an already connected user, or a connection request), how to manage client disconnections, etc... You''ll just have to add queues and protect them with mutexes. You can find a link to the guide on this site in the network programming section.
As for the messaging system, I''m implementing it with the pluggable factories. Those are really cool! =) Check them out (there''s also an article on this site about what pluggable factories are and how to use them in a networking environment).
So the result is: one main thread, one connection thread and several processing threads (be aware of starting 800 threads... it takes 0.1 second to switch between the threads, so all the processor time could be eaten with the OS and the server won''t have access to the CPU to perform its tasks). Do some calculations and decide which number of threads suits your server most.
"I''ll be Bach!" (c) Johann Sebastian Schwarzenegger
UNUMBER - the number of the object client is operating on.
COMMAND - is the action for the object to perform (0 == move, 1 == attack, 2 == take, etc...)
ARGS - is the field containing additional data for the user command to be properly processed by the server (if COMMAND == 0(move), then ARGS should contain coordinates of the destination point).
Depending on the type of data the client can wait for confirmation or rejection back from the server... For example, if the server couldn''t move a unit to the destination point, it sends a ''reject'' packet... and on the client side the unit says something like ''I can''t go there, Master!''... If it is a confirmation - the unit says ''As you wish, Master!'' and begins to move...
And on the server side the logic is more complex as it has to process and calculate everything except graphics (graphics rendering is the task of the client part): AI, game world and map changes, coordination and user interaction. etc... there''s a huge amount of work for the server.
Depending on the result of its calculations server needs to decide whether to send a ''confirm'' or a ''reject'' packet. Now imagine you have 800 clients connected to your server... As there''s only one client request being processed in the moment the other clients would have to wait until their commands are processed. For example imagine it takes 0.3 seconds for the server to process a certain command. And one of the clients has sent data. As long as the server is only capable to process one command at a time, every client has to wait (0.3 * 800 = 24 seconds) until his next command is processed... Well, I think a gamer wouldn''t be happy to play in such an environment... =) So the server needs to make its calculations as fast as possible and to process as many requests as possible at a time. That''s why we need several processing threads. So... there server needs to make an illusion of simultaneous processing. That''s it! We have to run several processing threads. They are not really simultaneous, but the gamer won''t see that... He will think the the server is only processing his input, and nobody else...
As for the connection threads, i would recommend using only one thread to manage connections and send/receive data, because the only thing it should do - is verify the data and copy the buffers to the proper queues.
Search for Beej''s Guide to Network Programming - it will tell you how to build a loop with a select() (see the ''multiplexing with select()'' section of the guide) and how to manage different types of incoming data (data from an already connected user, or a connection request), how to manage client disconnections, etc... You''ll just have to add queues and protect them with mutexes. You can find a link to the guide on this site in the network programming section.
As for the messaging system, I''m implementing it with the pluggable factories. Those are really cool! =) Check them out (there''s also an article on this site about what pluggable factories are and how to use them in a networking environment).
So the result is: one main thread, one connection thread and several processing threads (be aware of starting 800 threads... it takes 0.1 second to switch between the threads, so all the processor time could be eaten with the OS and the server won''t have access to the CPU to perform its tasks). Do some calculations and decide which number of threads suits your server most.
"I''ll be Bach!" (c) Johann Sebastian Schwarzenegger
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
