Archived

This topic is now archived and is closed to further replies.

clustered mmorpg questions

This topic is 5137 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am working on the design of a fantasy mmorpg that is set in Feudal Japan. Currently this is a "garage game" development. I plan to develop a good working demo for this game, and then seek investment to bring it to the masses. The ideal goal is to support 100K simultaneous players with 2.5K players per shard. I am concerned with making it very scaleable. I would ideally like to get a bare bones version of an expandable architecture running on a small test cluster of linux boxen. In order to meet this goal, I would like to break the game down into a set of processes (yes, in the unix sense). The list below indentifies the various processes of the game as I see it: * firewalling * authorization * patching * AI(s) * static game content database(s) * dynamic game database(s) * login lobby * landblock game/physics rules management processes * master control process * physical server control process * web portal process The basic client server flow would be: 1) client connects to the auth process and logs in (SSL) 2) a client patcher process starts, and commmunicates with the patcher process to update any out of date client files 3) the client is moved to the game login lobby, where they may choose a game shard to play 4) client''s last logout location/char profile is loaded from the dynamic game database and the character is spawned on the landblock process that is managing the login location 5) player plays, and is passed from landblock to landblock as he/she traverses the game world. Character data is backed up periodically to the dynamic game database. 6) player logs out, the character is persisted again at logout, and the player is moved to the login lobby 7) player can log out at this point or elect to join a shard and resume playing =================================================================== My first concern in tying all of this together is ip spoof proofing. I want to employ a challenge/auth technique. In other words, the very first packet from the client to a server machine will be to request a 32-bit anti-spoofing connection key. Every subsequent packet that contains this key must be from the initial IP. This verifies that the client really is at the IP they claim to be. My first question is would it be advisable to change this key periodically? If so, how often? Also, how do I maintain this key if it changes; in other words would I need another server just for the maintenance of this? Would it be advisable to build the spoofing protection into a "smart" firewall (but not very standard)? Is 32 bit enough? Perhaps I should use 64? =================================================================== A second question gets into how does the client ip data get routed to the right landblock server? Would it be inadvisable to tell the client the port:ip address of the landblock server directly? I am thinking I almost need a "smart" firewall/router process for this application too. As in, would this be advisable: * The landblock management process when it spawns a player object, registers with the smart router that gameplay packets for a particular connection key should now be routed to this landblock * during a handoff of a player from one landblock to another, one of the two landblocks will send a transaction message to the smart router to route gameplay packets to the new landblock My biggest concern is if I do follow this method, there is no easy way for the smart router to scale if it becomes a bottleneck. Maybe as part of the architecture I should include the ability to support multiple smart routers who autobalance the connection loads between them? =================================================================== Is any of this way off base? Please if you need more implementation details to address these questions, let me know. Any feedback is much appreciated!

Share this post


Link to post
Share on other sites
You''re ideas look good. Have you started coding yet or are you still just designing? I''m just curious since I''m working on something similar. 100k players? I like it. Very ambitious! I''ll be happy to get 50 people on at the same time.

You''re anti-spoofing technique will protect you from someone randomly pretending to be someone else, but you''ll still be susceptible to attacks from someone intercepting your message and then spoofing the whole packet (key and all). The only way around this is to come up with some kind of private/public key type encryption scheme where the server changes the key periodically in synchronization with the client. That''s probably more trouble than it''s worth and it''s still susceptible to reverse engineering.

As for the routing of messages to the landblock server. I am planning to just give the client the address/port when they sign in. That''s what most MMOG''s that I''ve seen do. The only reason I can think of to not do this is that it puts other open ports on the servers at risk of attack. I''m not planning on running anything else on my servers so it''s not an issue for me. If you have a single server setup as a router, it makes it easier to bring your entire system to its knees with a single DOS attack.

bpopp (bpopp.net)

Share this post


Link to post
Share on other sites
"You''re ideas look good. Have you started coding yet or are you still just designing? I''m just curious since I''m working on something similar. 100k players? I like it. Very ambitious! I''ll be happy to get 50 people on at the same time."
Heyas bpopp!
I''m still mostly in the design phase. I started this little project about a month ago. I''m working on the game design doc still (its VERY rough), but I wanted to verify my clustering architecture so I''m writing a test version of this in python.

I really want to get a working frame that goes from a client, through the login process, patching, all the way to firing up a simple engine that lets you move around a simple cube in the environment. From there it will be time to go back to working the design doc and adding the real content in this framework.

BTW, I live in California, but I noticed you''re from Tennessee. I am from TN originally, in the Murfreesboro area.


"You''re anti-spoofing technique will protect you from someone randomly pretending to be someone else, but you''ll still be susceptible to attacks from someone intercepting your message and then spoofing the whole packet (key and all). The only way around this is to come up with some kind of private/public key type encryption scheme where the server changes the key periodically in synchronization with the client. That''s probably more trouble than it''s worth and it''s still susceptible to reverse engineering."
I am just going for spoof protection to add a first line of security. Login/password and subscription information will be passed with a secure socket which I believe uses the dual-key system you mentioned.

"As for the routing of messages to the landblock server. I am planning to just give the client the address/port when they sign in. That''s what most MMOG''s that I''ve seen do. The only reason I can think of to not do this is that it puts other open ports on the servers at risk of attack. I''m not planning on running anything else on my servers so it''s not an issue for me. If you have a single server setup as a router, it makes it easier to bring your entire system to its knees with a single DOS attack."

I''ve since given this more thought, and here''s what I''m going to do:

There will be multiple smart routers. The client program will have a list of these addresses and pick one at random. If loading becomes an issue, I could perhaps have the routers communicate and dynamically hand off clients to maintain load balance.

The nice thing about this is it gives a good security control point; people outside will not be able to spam the landblock/patch/login directly.

Another good thing is it gives an access point for the spoofing key. The combination of the firewall ip address and an *internal* client connection id gives a unique way of identing each client connection. This connection key can be used on the insulated servers to associate game messages with a client. Also the firewall servers can periodically change the *external* anti-spoof key and this is transparent to the backend servers.

In other words, the firewall server will take the external key and associate it with an internal connection ID. It receives external packets, strips off the external key, then adds the internal key and forwards the data to the game servers. It can change the external keys at will and the internal servers will not have to deal with key changes.

Share this post


Link to post
Share on other sites
quote:
I really want to get a working frame that goes from a client, through the login process, patching, all the way to firing up a simple engine that lets you move around a simple cube in the environment.


That''s similar to me. My first goal for this prototype I''m working on is to have multiple users login, get connected, and to be able to walk their avatars around my simple terrain. I''ll also handle some basic messages such as chatting, emotes, etc. I''m already a month behind schedule, but if I can keep up my current pace I should finish before Thanksgiving (or shortly after). I have all of the client stuff built (MD2 models, terrain, skydome, console, etc.) and the server is handling three messages (disconnect, chat, and connect). Once I finish I''m planning on posting an invitation here on Gamedev to see if I can get some testers.

If you''re curious you can take a look at my project page (see my signature). Unfortunately, the CPU melted on my linux server this morning so my site is down. Hopefully I''ll have it back up tomorrow afternoon sometime.

quote:
I live in California, but I noticed you''re from Tennessee.


Yeah, I''m from Memphis. Not much happens here (especially compared to CA), but it''s ok.

quote:
I am just going for spoof protection to add a first line of security.


Yeah, I imagine that will work. It should at least prevent a majority of the kiddie stuff. I think the only way it would fail is if someone happened to be on the same subnet as one of your customers and they intercepted a message that was being sent using a packet sniffer. Even then, all they could do is spoof bogus messages to the server which the server would most likely reject. I''m sure you''ve read this, but just make sure you put ALL your logic on the server. The client should have absolutely no authority over its state (or anyone elses).


bpopp (bpopp.net)

Share this post


Link to post
Share on other sites
heh small world.

I used to live in TN, but now live in Cali as well and I too am working on a project similar to what you guys are doing.

I decided to mimic the server architechure that Everquest uses where theres a login server (possibly additional ones to support load), several different world servers, and each world server has numerous zone servers connected to it.

Most of the network framework is done and I''ve gotten the project to the point where the client can log into the login server via a username and pw and then get a list of worldservers presented to them along with their status. Users can then pick a world server and connect to it, at this point they are presented with a list of characters (or the option to create a new character) to choose from. Once they select a character the player connects the appropriate zone server (whatever one is hosting the zone the player was last in when they quit the game) and can move around.

Ive got a basic spawn system in place and mobs basically just spawn and despawn as if they were being killed while the player moves around. Other players can log in and see either other moving around. Although with only sending positional updates out to all the clients every 500ms the dead reckoning kinda sux and the movement can be jerky sometimes.

Data persistence is handled using MySQL DB. Characters and user accounts are stored here and each world that is created gets its own database for storing its relevant information. These can be hosted on either the same machine or their own machines.

I''m currently working on a GUI system so users can type in messages and commands to the server and getting the player to the point where he can zone into other servers hosting other areas of the game world.

Once I''ve added a few more things to the code, I''m going to start looking into creating some content creation tools and hopefully some content to go with it. =)


-=[ Megahertz ]=-

Share this post


Link to post
Share on other sites
Sounds awesome Megahertz. I envy you. I''ve come so far in the last 10 months, but I still feel like I have such a long way to go. Is this something you plan to try and market or are you just doing it to prove you can (like me). Any chance I could take a look at a demo?

bpopp (bpopp.net)

Share this post


Link to post
Share on other sites
quote:
Original post by Megahertz
Although with only sending positional updates out to all the clients every 500ms the dead reckoning kinda sux and the movement can be jerky sometimes.


Every 500 ms??? Doesn''t that cause a bunch of stress on your server? If you had on 1000 clients, you''d be sending out 500 updates per second per client. That''s 500,000 packets per second (presuming you don''t limit the updates to those close by, which of course you probably are But even so, you must be sending out a *ton* of position updates). That''s going to bring your server to it''s knees, isn''t it?

I''m only sending out updates if the last positional update was > 5 seconds ago on the client end. To smooth out the in-between time, I also send the velocity data along with it and let the client calculate where other avatars are.

I also plan on sending out an update when a client changes velocity to reduce the popping, but I haven''t gotten that far yet Once I do that, I figure the popping should be pretty minimal at that point, and I can correct an unsynched position over a few frames rather than all at once.

I''m also finding that my average player has between 200-1000 ms of round trip lag time (depending on their connection). This is another reason I went with a much larger update time.

So I''m curious to know how you''re managing with the 500 ms updates. Do you find that''s working okay?

Share this post


Link to post
Share on other sites
just curious MHz, how long did it take you to get to that point? That''s awesome you have that much working. I am also planning to use mysql for data persistence. Also I have a *bunch* of questions:
1) did you use udp or tcp?
2) do you do any spoof protection?
3) do you use SSL for the login/password traffic?
4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
6) what platform are your servers running on? Linux/X86?
7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
8) what languages are you leveraging for development? All c++ or mixing in python, etc?
9) what algorithm are you using for collision detection?
10) is your client directx or opengl or what?
11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
12) Are you using the observer/observable model for figuring out client updates?
13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
14) Is your client engine BSP based or outdoor roamish/portals based?



Here are my answers to the above questions, for what it is worth:

1) did you use udp or tcp?
I plan to use udp for almost everything, with some (TBD) amount of TCP used for the login/patch process

2) do you do any spoof protection?
Yes, as above. I''m writing my smart firewall now.

3) do you use SSL for the login/password traffic?
I plan to, but my smart firewall only handles UDP, and handing off connection IDs is not trivial, but maybe I will have this process manage a SSL socket in addition to the UDP.

4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
I do not plan to store credit card numbers as yet. I plan to pop up a resubscribe when the user closes in on the end of a subscription period.

5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
I''m planning on having the various processes be configured from a master control progam(MCR)at boot up, and the MCP persists the config in a mysql database.

6) what platform are your servers running on? Linux/X86?
Linux servers, windows dx9 client

7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
I haven''t yet but when I get into the backend game servers, I might use twisted (www.twistedmatrix.com)

8) what languages are you leveraging for development? All c++ or mixing in python, etc?
Planning to use python for almost everything for my alpha demo. Moving to the release I plan to rewrite the bottlenecks in c++.

9) what algorithm are you using for collision detection?
I am planning on using a mix of http://www.gamasutra.com/features/20020118/vandenhuevel_01.htm

and

a great little sphere tree algorthm from the game gems 2 book, and I think EQ is using sphere tree.

10) is your client directx or opengl or what?
planning on a DX9+ client. Realistically, by the time this is released (2-3 years), DX9+ pcs will be the norm, IMO.

11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
Planning on using the sphere trees as talked about in 9.

12) Are you using the observer/observable model for figuring out client updates?
I am going to try to use the observable/observer model.

13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
I am thinking client updates just have to occur every 200ms or so to do proper server side collision testing. Maybe I''m wrong here though? Perhaps if the client sent along a spline curve approximating its path, you could get away with fewer updates.

14) Is your client engine BSP based or outdoor roamish/portals based?
outdoor engine, no bsping planned. I''m still fuzzy on how this will all play out.


As an aside on updates, I''ve seen the following anti-lag measures:
1) change update frequencies of objects based on how "important" they are. If its hostile NPC it is important. If its a loot bag or friendly npc, it may not be as important
2) decrease update rate for more distant objects. I.e., the farther away it is, the less frequently you update it
3) prediction: instead of just positions, send velocities as well, so the client can approximate the object positions
4) Run the simulation some amount of time lagged so the future is to some degree already known (implies you can interpolate more rather than extrapolate)
5) prediction: instead of just putting the objects at positions, use the positions to plot out a smoother curve like a spline
6) Change update frequency based on how well your (probably linear) prediction model is handling the object. For example if someone is running in a dead straight line for a long period, the velocity prediction model works well and you can slow the update rate. If they are moving erratically, the update rate may need to increase


Share this post


Link to post
Share on other sites
Well, I'm not Megahertz, but if you care, here are my answers

>1) did you use udp or tcp?
UDP, exclusively. I understand the lure of using a hybrid UDP/TCP approach (for file updates and the like), but I had already implemented a reliability and ordering protocol for UDP that I could apply to selected packet types at will, so there was no need to use TCP in my program (besides which it would have vastly complicated my server network code, which is IOCP anyway so therefore pretty complex as it is )

>2) do you do any spoof protection?
Yes. My users are required to prove they are the person they say they are. This involves IP/Port checks as well as encoded ID values.

>3) do you use SSL for the login/password traffic?
No. I use MD5 encryption and the plain text version *never ever* leaves the client machine. Since MD5 is irreversable, I think this is secure (someone tell me if I'm wrong, heh).

>4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
Yes I do. I haven't yet looked into this, but my understanding is that PayPal has a system whereby they hold all that info and deal with the billing process (for a transaction fee) and in return I get something that plugs into my database allowing me to know at any given moment who does and doesn't have a current account. That's about all the details I kn0w about it, like I said, I've not researched it at *all*

>5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
Not applicable. I have one server

>6) what platform are your servers running on? Linux/X86?
WinXP Pro.

>7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
No.

>8) what languages are you leveraging for development? All c++ or mixing in python, etc?
C++, exclusivly. Of course, I talk to my database (MySQL) using the normal SQL language, but even then, in the server code, I use the mysql.h header so I can access it via C++.

>9) what algorithm are you using for collision detection?
Undecided. Collision detection hasn't been implemented in my game yet. I'm open to suggestions (I'll check out your links).

>10) is your client directx or opengl or what?
DirectX (D3D more to the point of you question, though I also use DInput and DSound).

>11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
I'm using a landblock based system. If an item is in the same or an adjacent landblock to the player, it's data gets transmitted. Yeah, it's a bit inefficient (at initialization, you are sending up to 9 landblocks worth of data, while during play at worst case you could send 5 landblocks if the player crosses a boundary just right), but any other way I've seen requires major computations on the server end that I don't want the server spending time on. My server is based on the idea of processing incoming packets and getting onto the next one as fast as possible. It's up to the client to decide what is viewable from the data given.

>12) Are you using the observer/observable model for figuring out client updates?
No, if I understand your question properly.

>13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
My clients send velocity and position change updates whenever a velocity changes (forward, sidestep, or angular, in other words, whenever the user stops or starts pressing a movement key), or every 2 seconds, whichever comes first. The server returns an ack to the client when it gets an update. Also all my clients send out 3rd party position update requests if the last one recieved for that avatar is more than 5 seconds old (as I said before). The server sends out a 3rd party position update upon request, or when that party changes one of their velocities (the second part of that hasn't yet been implemented, but that's the plan).

14) Is your client engine BSP based or outdoor roamish/portals based?
ROAM, portals indoor (though only ROAM is implemented as yet, there are no indoor sections to my game at this point ).

-Ron

[edited by - RonHiler on November 16, 2003 1:24:51 AM]

Share this post


Link to post
Share on other sites
Hey Ron,

Thanks for the informative response!

Your all UDP implementation inspires me to do the same. You''re right, as long as you can apply a message sequencing/resend on top of udp, why bother with tcp?



">2) do you do any spoof protection?
Yes. My users are required to prove they are the person they say they are. This involves IP/Port checks as well as encoded ID values."

That is exactly what I am doing. Just curious, how big is your ID value? 16, 32, 64 bit?


">3) do you use SSL for the login/password traffic?
No. I use MD5 encryption and the plain text version *never ever* leaves the client machine. Since MD5 is irreversable, I think this is secure (someone tell me if I''m wrong, heh)."

I don''t think this is secure. What prevents someone intercepting a packet and reusing the hash to gain someone''s login? True they won''t have the plaintext password, but just resending the hashed to the server will work.

The only way to really do this securely that I know of is with a public/private key encryption system. In these systems only the private key can unlock what the public key has encoded, and vice versa. The usage would be:

1) server sends the client the public key
2) then the client can encrypt with the public key and send data, and only the private key on the server can decode this data.

The above is probably secure enough for login/passwords for non-financial data. If credit cards numbers are being send you probably need a bit more...

A problem that still remains is if someone can intercept packets and impersonate the server. To prevent this, people use digital signatures from a trusted, guaranteed secure third party. I''m a little fuzzy on how you avoid this but I think people just do something like store the public key with the trusted 3rd party.


">4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
Yes I do. I haven''t yet looked into this, but my understanding is that PayPal has a system whereby they hold all that info and deal with the billing process (for a transaction fee) and in return I get something that plugs into my database allowing me to know at any given moment who does and doesn''t have a current account. That''s about all the details I kn0w about it, like I said, I''ve not researched it at *all*"
Paypal is a good idea, I need to look into how they operate.


">9) what algorithm are you using for collision detection?
Undecided. Collision detection hasn''t been implemented in my game yet. I''m open to suggestions (I''ll check out your links)."

The sphere tree algorithm shows up in the books Game Programming Gems 2 and Game Programming Gems 3, and it looks very powerful, but i haven''t tried implementing it yet.

">11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
I''m using a landblock based system. If an item is in the same or an adjacent landblock to the player, it''s data gets transmitted. Yeah, it''s a bit inefficient (at initialization, you are sending up to 9 landblocks worth of data, while during play at worst case you could send 5 landblocks if the player crosses a boundary just right), but any other way I''ve seen requires major computations on the server end that I don''t want the server spending time on. My server is based on the idea of processing incoming packets and getting onto the next one as fast as possible. It''s up to the client to decide what is viewable from the data given."

Just curious, how big are your landblocks, and how is the scale setup with respect to your player?

">13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
My clients send velocity and position change updates whenever a velocity changes (forward, sidestep, or angular, in other words, whenever the user stops or starts pressing a movement key), or every 2 seconds, whichever comes first. The server returns an ack to the client when it gets an update. Also all my clients send out 3rd party position update requests if the last one recieved for that avatar is more than 5 seconds old (as I said before). The server sends out a 3rd party position update upon request, or when that party changes one of their velocities (the second part of that hasn''t yet been implemented, but that''s the plan)."
hmm client driven updates. Thats interesting; it seems like a good idea.

Share this post


Link to post
Share on other sites
Good questions. I'm not very far along, but answers are as follows:

1) did you use udp or tcp?
So far UDP only. My server code, so far, is pretty flexible though so I may look at adding TCP for certain messages (file transfers, etc.).

2) do you do any spoof protection?
I use IP/port to identify connections initially. I also check this against the clients unique ID. At some point I'll probably add a key of some sort to prevent simple spoofing.

3) do you use SSL for the login/password traffic?
No, I'll most likely use the MD5 method that RH is using. Currently it's just plain text over TCP.

4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?
I'm planning on making my project free. If it ever goes commercial, I'll store CC# in a database (probably a different one from the primary database). On some of my web projects, I have encrypted them using simple encryption. This is worthless if your web server gets compromised, but if that happens, you're in trouble anyway.

5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?
Currently just one server. I'm not planning for a "massive" MORPG yet. I may once I get further along.

6) what platform are your servers running on? Linux/X86?
I'm using SDL_Net on Linux for the servers. The clients are DirectX 8 and also use SDLNet. The server code is all abstracted so if I want to use something other than NetSDL, I can easily replace it.

7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
No. I've read a little about beyond2 in Massively Multiplayer Game Development, but it sounded complex and if anyone's going to complicate my code, it's going to be me!

8) what languages are you leveraging for development? All c++ or mixing in python, etc?
C++ exclusively. I may look at embedding python for some basic scripting. That sounds interesting.

9) what algorithm are you using for collision detection?
My collision is all very basic AABB collision at this point. Honestly, this may be sufficient for what I want to do. I'm going to focus on gameplay vs. realistic graphics/physics.

10) is your client directx or opengl or what?
DirectX 8. I'll most likely use 9 when I finish the prototype.

11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
Currently I'm not. My "map" is 2048 by 2048 so I'm just broadcasting everything to everything. This is obviously not going to hold up well under load, but it will be easy to add some simple range-based occlusion.

12) Are you using the observer/observable model for figuring out client updates?
Sort of. I think the pattern I am actually using is called "Multicast". Each message registers itself with a handler that extends an abstract IHandler. As the messages enter the queue, they are then passed off to the appropriate derived handler.

13) does the client send update positions once every 500 ms too? If so, how does this figure into determining if someone is hacking the client to walk through walls?
I'm working on this now so I can't really say. I'm hoping I won't have to send updates that frequently, but we'll see.

14) Is your client engine BSP based or outdoor roamish/portals based?
Strictly outdoor at this point. I'll most likely use portals when I start adding indoor environments.

My journal is back online if anyone is curious. It's goes back to the very beginning with screenshots all along the way.

http://www.bpopp.net/articles/view.php?id=403

[edited by - bpopp on November 16, 2003 3:57:54 PM]

Share this post


Link to post
Share on other sites
quote:
Original post by red_mage
Your all UDP implementation inspires me to do the same. You're right, as long as you can apply a message sequencing/resend on top of udp, why bother with tcp?

Well, let me qualify that by saying you need to code it in such a way that you can apply it selectively to particular packet types. Also you should be able to send ordered OR reliable OR both depending on the need. If you don't do those things, you might just as well use TCP

quote:

">2) do you do any spoof protection?
Yes. My users are required to prove they are the person they say they are. This involves IP/Port checks as well as encoded ID values."
That is exactly what I am doing. Just curious, how big is your ID value? 16, 32, 64 bit?

32 bit. One somewhat unfortunate side effect of my security procedure is that if a dialup user (with a dynamic IP) loses thier connection and redials, they can't actually get back into the game, because they now have a different IP address than they logged into it with. They actually have to wait for the server to decide they are stale and auto-log off the account before they can get back in, which takes a few minutes. But, eh, what are you gonna do? I don't see any way around this without introducing more security holes than I have already, heh.

quote:

">3) do you use SSL for the login/password traffic?
No. I use MD5 encryption and the plain text version *never ever* leaves the client machine. Since MD5 is irreversable, I think this is secure (someone tell me if I'm wrong, heh)."
I don't think this is secure. What prevents someone intercepting a packet and reusing the hash to gain someone's login? True they won't have the plaintext password, but just resending the hashed to the server will work.


Yeah, that's a good point, and of course you're right. I'm actually in the process of adding a public/private key encryption system on top of protecting the plaintext password as well.

quote:

">11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?
I'm using a landblock based system[...]
Just curious, how big are your landblocks, and how is the scale setup with respect to your player?

Of course, the scale is arbitrary. For various reasons, I've set a scale of 1 landblock = 1 sq. kilometer.

To be honest, I don't think my landblock system as it is right now will hold up for long. There could be literally hundreds of items in each landblock that would need to be transmitted every time a player crossed a landblock boundery. I have every item in my game indexed by landblock, so I can get at them very quickly, but even so, there is some overhead in putting together the packets and sending them.

On the other hand, what I *don't* want to do is check items against player for distance every time they take a step. That would be equally disasterous.

The problem is speed. Like someone said (I think it was KalvinB), if you spend an average of 1 ms processing each packet that comes into the server and you have 1000 players, you've just added 1 second of software induced lag to your game on top of the internet lag.

Probably what I'll do is some sort of hybrid between the two systems. I'll maybe to a check to see what landblocks are visible to the player, then check only those items in those landblocks (or subdivisions thereof) against the player to see if they are in range for sending their data. I may also split my landblocks into 4x4 grids for the purpose of item data checking/sending. I don't know, I'll have to see how it works out

quote:

hmm client driven updates. Thats interesting; it seems like a good idea.


Only way to do it, as far as I'm concerned. As far as packets go, the server should only respond, never initiate (as much as possible, sometimes you have to break that rule (chat packets for instance)).

bpopp, cool journal. It's always interesting to me to read about development progress!

-Ron



[edited by - RonHiler on November 17, 2003 2:23:26 PM]

Share this post


Link to post
Share on other sites
Here's an update from my world...

Basically I'm thinking that maybe routing should be left to routing hardware. Real router/firewalls are going to give me the better price/performance than any software "smart router solution" IMO. They will *not* be as secure as my smart firewall/routers I described before. There are other problems as well.

The smart routers I had described required you to move ahead in the game login state before you could access the next server. In other words you could not physically get packets to the game landblock servers without first moving through the login and patching states. And once your game packets were being routed to a game landblock you still couldn't talk to the other game landblocks till you were handed off and rerouted to the next landblock server.

Another nice thing about them is they provided a single access control point for generating connection IDs, and only challenge/response for spoofing would need to be done once, instead of once for each socket that the client connects to.

Without the central connection ID generation of smart firewall/router method, the login servers will have to generate connection IDs for the client and broadcast the association to all backend servers. This is ugly.

The other real thing I have a problem with for just letting the client connect directly to the backend server is the client becomes responsible for addressing each landblock via its socket. I really do not like the possibilities that opens up.

I do like the smart firewall approach but I worry about all this smart firewalling being a network bottleneck. For each client the routers would have to store their external connection key, the internal connection ID and the accounting server/patch server/landblock server/whatever other server they are being routed to.

In the client=>server direction, the smart fw/r would need to translate external connection key to a client ID, then route the message to the listeners for this client id.

In the server=>client direction, the smart fw/r would have to translate a client ID to an (external connection key, client ip, client port.

For security, I would need to allow a max of say 10 connections from any one IP. This is to prevent flooders from taking my servers down by moving through the challenge/response to create the association over and over and eating up all my server memory. So I would need to manage this.

Plus I would need to manage a duplicate request for a connection from a particular IP/port. As in if the client didn't realize it succeded and tried rerequesting auth again. I wouldn't want multiple connection IDs mapping to a single client/port.

I would also need to time out the association for the client if they don't cleanly terminate thier connection.

What do you guys think?

Also I have given more thought to my class that will wrap around sockets. Its described in some detail here:
http://www.gamedev.net/community/forums/topic.asp?topic_id=191145

[edited by - red_mage on November 18, 2003 6:43:51 PM]

Share this post


Link to post
Share on other sites
1) did you use udp or tcp?

- Both, I''m using pretty much the same design that Everquest uses, a central login server with multiple worldservers that are accessable from the login. Each "world" is comprised of multiple zone servers that all authenticate and register with the world server. Server to server communication is strictly TCP while the client to server communication uses both TCP and UDP depending on what server the client is talking to.

The client to server communication uses TCP if the client is talking to the login or world servers. Theres a breif overlap period where the client talks to the world server via TCP and the zone server via UDP, but only between the period where the client requests a connection with the zone server and the time the client gets back a response.


2) do you do any spoof protection?
- Not at this time. We''ll at least no intentional protection. MMORPG''s typically do not give out the clients IP address to other clients, so it would be hard for somebody to tamper with another''s connection outside of the local network. Inside the network, traffic could be potentially intercepted and modified but for the general user this shouldnt be an issue and a rare case.

3) do you use SSL for the login/password traffic?

- Nope. Right now the passwords are plaintext, but they will eventually be encrypted via a public key supplied by the server. RSA is fairly secure and should offer adequete protection.

4) Do you plan to store credit card numbers, and if so, how would you securely store it in your mysql database?

- Of course, the CC#''s will probabbly also be encrypted using some sort of asymmetric encryption.

5) how do all the various servers know which server is at what address? Is there a database for this or is it handled in startup scripts, or what?

- Basically the whole architecture is a tree. All the zone servers connect and register with their appropriate worldserver and the worldservers do the same with the login server. When the client wishes to connect to a certain server the client is sent the ipaddressort of where to connect.



6) what platform are your servers running on? Linux/X86?

- Currently developing on WinXP, once stable and theres no more design changes I''ll prolly port it to *nix.

7) are you leveraging beyond2, nebula, nevrax nel, or twisted?
- Nope, so far everything is homegrown aside from things like OpenGL and DirectX.

8) what languages are you leveraging for development? All c++ or mixing in python, etc?

- Its all C++. Scripting will prolly be handled by a custom scripting language.


9) what algorithm are you using for collision detection?
- None atm, low on the priority list atm while i finish up the network code and stuff. Its a toss up between dynamic plane shifting and elipsoid/sphere collision detection. Things are heavily leaning towards DPS since players of different races will be different heights, sizes, etc. It''ll prolly end up some combination of the two.

10) is your client directx or opengl or what?

- Both, OGL for graphics, DX for sound/input.

11) (Related to collision detection) How are you determining viewability for a client? As in, how do you decide what things the client can see from the server perspective?

- Right now its simply distance based. For the most part I''m keeping things simple my first go around.

12) Are you using the observer/observable model for figuring out client updates?

- I''m not familliar with how the observer design patter works. Positional updates are sent out every 500ms or so for close entities and right now its at 3 seconds for entities beyond the FOV of the player.

13) does the client send update positions once every 500 ms too?

- The client sends updates a little faster, usually when a key is pressed/released or when the player rotates his avatar. Idle rate is once every second.

If so, how does this figure into determining if someone is hacking the client to walk through walls?

- Haven''t adressed that issue yet, but its on my list.

14) Is your client engine BSP based or outdoor roamish/portals based?

- just a simple octree atm. Project hasnt progressed far enough along to worry about the rendering engine too much. In fact its last on my list of priorities. I''m trying to get as much of the other parts of the engine done before working on the graphics engine. I''ve got a simple renderer up but nothing elaborate.


-=[ Megahertz ]=-

Share this post


Link to post
Share on other sites