Jump to content
  • Advertisement
Sign in to follow this  
Fallonsnest

MMO Network Design

This topic is 3314 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Im working on little MMO project and can use some advice on the networking part. First of all, im not really creating a game (yet), im more trying to make some base MMO design more or less to have a challange to see if i can do it and my interest in MMO design. so the idea behind the project: - Master Server (handle accounts/worlds) - Character Server (handling character info) - World Server (server running a small game world) People can login, choose a world server, create/select a character, travel around, chat, interact with npc, fight some mobs ... The way im building the networking is using TCP with select() method for multiple connections. now my FD_SETSIZE is set to 1024, so limiting me to 1024 connections at the same time. Now i have a few ideas in mind to solve to connection limit issue. 1) Limited zone users as the game world is split up in zones, it would be possible to have 1 fd_set / zone, so allowing 1024 players in same zone at same time. I dont know how acceptable this would be for real MMO games. 2) Linked fd_sets a second solution would be having a linked list containing a 1024 array fd_set so when the first fd_set contain 1024 sockets and a new socket is added, a new node to the linked list is added with a new 1024 array fd_set. When a connection drops a socket is moved from the last fd_set to the first available fd_set till the last one is empty and can be cleanup up again. 3) Single socket select Not sure if this would be a good idea and how it would affect server performance, network performance. It would be possible to let every socket have its own fd_set and so called socket->select() method on each single socket. now in case the 2the method would be the best method, i wonder about the fd_set size. Will having small fd_set (for example 64) have better performance as largde fg_sets (for example 1024). note: all networking is done in separated thread, when a socket receive data, the data is parsed, validated and pushed in the the MessageInQue. The main loop thread check for incomming message trough that same Que (mutex locks) and handle the message. Sending data is done same way, if main loop thread must send data, it pushed a new message in the MessageOutQue, so the network thread can deque the message and send it.

Share this post


Link to post
Share on other sites
Advertisement
you should look into epoll instead of select.
http://blog.kovyrin.net/2006/04/13/epoll-asynchronous-network-programming

Share this post


Link to post
Share on other sites
While _Kami_ makes a good point, it should be noted that epoll is specific to Linux. That means that it *cannot* be used under windows (or any other Unix-like system, for that matter) without wrappers.

Share this post


Link to post
Share on other sites
Setting FD_SETSIZE to something bigger than 64 on Windows is not recommended, because the Windows kernel doesn't like blocking on more than 64 handles at the same time. If you find that you need more than 64 handles in the same wait operation, it is recommended to go with IOCP.

If you want all that handled for you, I suggest using the boost::asio library, which provides asynchronous I/O for Linux and Windows (on sockets as well as files) using one cohesive API. Let someone else sweat the small stuff :-)

Share this post


Link to post
Share on other sites
Currently looking into IOCP but its bit hard to learn with limited info and sample being so different from eachother its hard to track down the main steps.

As for 3the party libraries such as boost, as i mentioned in the main topic, im doing this project to have a new challange, so prefer to write myself and learn from my mistakes.


Now about the IOCP i looked into some samples but all of them work so different.
I guess the way of implement depend a lot from application to application.

Now in my case it might get bit complicated. First of all i have more as 1 listener.

as for the master server i have the main master server port where clients connect, execute some message (login, select server, select/create char) and disconnect again.

then i have another connection where multiple char/world servers can connect to (connection stay alive all time)

After the little bit i learned about IOCP so far i could use some advice to see if im going the right directions.

1) Should all sockets from both listeners be put in same completion port?
2) Should i add my listeners to the completion port or run them in 2 extra threads with blocking accept?



now i have 1 more question about: CreateIoCompletionPort() method
When u call it the first time, u can set the last parameter 'NumberOfConcurrentThreads' for the number of threads
i notice most samples set this value to 0 and create theyre own set of worker threads calling 'GetQueuedCompletionStatus()'
now this confuses me a bit.

for example, u make 2 worker threads calling 'GetQueuedCompletionStatus()'
these are 2 visible threads. But if u set the thread parameter also to 2 when making the IOCP, does it mean there 2 more threads running on background (non visible threads)
If so ... what those threads do?

Share this post


Link to post
Share on other sites
Quote:
Original post by Fallonsnest
Currently looking into IOCP but its bit hard to learn with limited info and sample being so different from eachother its hard to track down the main steps.


There is a very recent discussion about just that.

Share this post


Link to post
Share on other sites
Wrote a first IOCP test today but wondering how good my code is.

the test is bit based on the IOCP sample of the windows SDK and some stuff i read about IOCP dont make sense with it.

1 thing i read is that an overlapped structure should never be used again and a new overlapped struct should be made for each operation. As u can see in code bellow (same as the sdk iocp sample) the overlapped structure is being re-used.

As u also notice in the code is that i create 2 overlapped structures, 1 for read, 1 for write, so both can happen same time. (im running dual core pc, so this code will use 2 IOCP threads and 4 worker threads)
Now my send code is bit dirty, its bit based on my first design where every frame the code check if there is pending data to send. I know i should only start write actions once there is data and stop send events as soon all data is send.

Question1: Can i re-use the overlapped structure like in code bellow or should i made a new one?
Worker thread code ...

Question2: Can i use a read/write IO operation at same time or can i only do send if a read operation completed?
I know the code below is bad as there should be no send event (without passing any data) all time if there is no data to send



//-------------------------------------------------------------------------------
// Included header files
//-------------------------------------------------------------------------------
#include "StdAfx.h"

#define DEFAULT_BUFFER_SIZE 1024

//-------------------------------------------------------------------------------
// Name:
// Desc:
//-------------------------------------------------------------------------------
struct IO_CONTEXT
{
WSAOVERLAPPED Overlapped;
BYTE Operation;
char Buffer[DEFAULT_BUFFER_SIZE];
WSABUF WSABuffer;
};

//-------------------------------------------------------------------------------
// Name:
// Desc:
//-------------------------------------------------------------------------------
DWORD WINAPI WorkerThread( LPVOID lpParam )
{
LPWSAOVERLAPPED lpOverlapped = NULL;
SOCKET hSocket = INVALID_SOCKET;
DWORD dwNumBytes = 0;
DWORD dwFlags = 0;
BOOL bResult = FALSE;

// Get the completion port handle
HANDLE hCompletionPort = (HANDLE)lpParam;

// Run the thread loop
while( true )
{
// Check the completion port for pending io operations
bResult = GetQueuedCompletionStatus( hCompletionPort, &dwNumBytes, (PULONG_PTR)&hSocket, (LPOVERLAPPED*)&lpOverlapped, INFINITE );

// Check if we have a valid socket
if( hSocket == 0 )
break;

// Get the context
IO_CONTEXT* pContext = CONTAINING_RECORD( lpOverlapped, IO_CONTEXT, Overlapped );

// Check what kinda io operation we dealing with
if( pContext->Operation == 0 )
{
// Check for dropped connetions
if( 0 == dwNumBytes )
{
OutputDebugString( L"Connection dropped ...\n" );
closesocket( hSocket );
}
else
{
//TODO: Process the buffer
OutputDebugStringA( pContext->WSABuffer.buf );

// Begin receiving the next buffer
ZeroMemory( pContext->WSABuffer.buf, pContext->WSABuffer.len );
WSARecv( hSocket, &pContext->WSABuffer, 1, NULL, &dwFlags, &pContext->Overlapped, NULL );
}
}
else if( pContext->Operation == 1 )
{
//TODO: Check for outgoing data and put in the WSA buffer

// Begin sending the next buffer
ZeroMemory( pContext->WSABuffer.buf, pContext->WSABuffer.len );
WSASend( hSocket, &pContext->WSABuffer, 1, NULL, dwFlags, &pContext->Overlapped, NULL );
}
}

// Exit the thread
return 0;
}

//-------------------------------------------------------------------------------
// Name:
// Desc:
//-------------------------------------------------------------------------------
int wmain( int argc, WCHAR* argv[] )
{
#if defined(DEBUG) | defined(_DEBUG)
// Enable run-time memory check for debug builds.
_CrtSetDbgFlag( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
#endif

// Initialize WinSock
WSAData wsaData;
WSAStartup( MAKEWORD( 2, 2 ), &wsaData );

//--> Begin IOCP code
// Get the system information
SYSTEM_INFO systemInfo;
GetSystemInfo(&systemInfo);

// Calculate the number of threads
DWORD dwThreadCount = systemInfo.dwNumberOfProcessors * 2;

// Create the completion port
HANDLE hCompletionPort = CreateIoCompletionPort( INVALID_HANDLE_VALUE, NULL, 0, systemInfo.dwNumberOfProcessors );
if( NULL == hCompletionPort )
goto App_Cleanup;

// Create the worker threads
HANDLE* hThread = new HANDLE[dwThreadCount];
for( UINT32 iThread = 0; iThread < (UINT32)dwThreadCount; iThread++ )
hThread[iThread] = CreateThread( NULL, 0, WorkerThread, hCompletionPort, 0, NULL );
//--> End IOCP code

//--> Begin socket code
// Create the listen socket
SOCKET hListener = WSASocket( AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED );
if( INVALID_SOCKET == hListener )
goto App_Cleanup;

// Bind the socket
sockaddr_in service;
service.sin_family = AF_INET;
service.sin_addr.s_addr = inet_addr( "127.0.0.1" );
service.sin_port = htons( 3000 );
if( SOCKET_ERROR == bind( hListener, (SOCKADDR*)&service, sizeof( service ) ) )
goto App_Cleanup;

// Start listening
if( SOCKET_ERROR == listen( hListener, 5 ) )
goto App_Cleanup;
//--> End socket code

// Run the loop
int i = 0;
while( true )
{
// Accept the next socket
SOCKET hAccept = accept( hListener, NULL, NULL );

OutputDebugString( L"Connection accepted ...\n" );

// Add the new socket to the completion port
CreateIoCompletionPort( (HANDLE)hAccept, hCompletionPort, (ULONG_PTR)hAccept, 0 );

// Begin receiving data
DWORD dwFlags = 0;
IO_CONTEXT* pContextRecv = (IO_CONTEXT*)HeapAlloc( GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof( IO_CONTEXT ) );
pContextRecv->Overlapped.Internal = 0;
pContextRecv->Overlapped.InternalHigh = 0;
pContextRecv->Overlapped.Offset = 0;
pContextRecv->Overlapped.OffsetHigh = 0;
pContextRecv->Overlapped.hEvent = NULL;
pContextRecv->Operation = 0;
pContextRecv->WSABuffer.buf = pContextRecv->Buffer;
pContextRecv->WSABuffer.len = sizeof( pContextRecv->Buffer );
ZeroMemory( pContextRecv->WSABuffer.buf, pContextRecv->WSABuffer.len );
WSARecv( hAccept, &pContextRecv->WSABuffer, 1, NULL, &dwFlags, &pContextRecv->Overlapped, NULL );

// Begin sending data
dwFlags = 0;
IO_CONTEXT* pContextSend = (IO_CONTEXT*)HeapAlloc( GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof( IO_CONTEXT ) );
pContextSend->Overlapped.Internal = 0;
pContextSend->Overlapped.InternalHigh = 0;
pContextSend->Overlapped.Offset = 0;
pContextSend->Overlapped.OffsetHigh = 0;
pContextSend->Overlapped.hEvent = NULL;
pContextSend->Operation = 1;
pContextSend->WSABuffer.buf = pContextSend->Buffer;
pContextSend->WSABuffer.len = sizeof( pContextSend->Buffer );
ZeroMemory( pContextSend->WSABuffer.buf, pContextSend->WSABuffer.len );
WSASend( hAccept, &pContextSend->WSABuffer, 1, NULL, dwFlags, &pContextSend->Overlapped, NULL );
}

App_Cleanup:
// Tell the worker threads to exit
for( UINT32 iThread = 0; iThread < (UINT32)dwThreadCount; iThread++ )
PostQueuedCompletionStatus( hCompletionPort, 0, 0, 0 );

// Wait for the worker threads to exit
WaitForMultipleObjects( dwThreadCount, hThread, TRUE, INFINITE );
delete[] hThread;

// Cleanup the listener
if( INVALID_SOCKET != hListener )
closesocket( hListener );

// Cleanup the completion port
if( NULL != hCompletionPort )
CloseHandle( hCompletionPort );

// Cleanup WinSock
WSACleanup();

// Exit the application
return 0;
}





[Edited by - Fallonsnest on May 22, 2009 3:11:56 PM]

Share this post


Link to post
Share on other sites
There is quite a lot of topics regarding IOCP, some recent as well, such as these or this. They should have some answers to various implementation details.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!