Jump to content
  • Advertisement
Sign in to follow this  
cjmrdc

Help me theorize about bandwidth needs.

This topic is 4166 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Howdy, I've been doing some crude math to help refine what I have to know about bandwidth constraints, unfortunately what I don't have access to are the average bandwidth consumption of clients of popular games out right now. So I've been using some theoretic numbers with the help of some other brain stormers to guesstimate what other games are using. We've come up with the average bandwidth usage per user in World of Warcraft to be about 5mb/hr, and amongst us were unable to establish a good number for higher use games such as CounterStrike, so we're using both 20mb/hr and 40mb/hr per user for those games. I'm basing the rest of my math upon the bandwidth available to us, which is 48,000gb a month per rack currently. or 48million mb a month, doing some crude math (such as uing 1000 for my math, as opposed to 1024). The reason for this way to assume a steady number of connections, using a steady amount of bandwidth, breaks down such as this: 5mb an hour, 13,333 constant users 20mb an hour, 3,333 constant users 40mb an hour, 1,666 constant users Our goal is 5,000 users connected, which works out to 13.3mb/hr max per user, or 3.69kb/s. This is again using the bandwidth that's currently available to the rack (without any other adjustments). The only data I've been able to find suggests that current implementations of CS (source) use a total of about 7kb/s, (up and down), so it's logical for me to assume that, if 3.69kb/s is the max for our upstream (this is not the max of our pipe, just of our metered upstream bandwidth), assuming some clever networking, we could in theory keep our bandwidth under 3.69kb/s per client. Here's where my question gets theoretical (well, more theortical). All this math is done assuming constant connections, we all know that bandwidth is not a constant thing, crowded areas do more and open areas do less. Also, this is assuming a steady number of people connected all the time, which we also know is not true. So, I'd like some assumptions, or somewhere to find some data about the average curve of time people are connected (for primarily US audiences, US/UK is fine), I expect that it also depends on the day of the week, i'm assuming less people are on from 2-6am monday through friday then they are on the weekend when folks can stay up late. Any suggestions on, if the system is designed around 5000 simultaneously connected users per rack, where's my margin for total users that can be supported? I'm hoping some folks here with a bit more hands on knowledge will have a good idea or theory to sort it out. Thanks for your time.

Share this post


Link to post
Share on other sites
Advertisement
You won't know anything for sure until you test it, or run the numbers on your protocol and world design.

How many state updates, how large are they, what is the player distribution, and so on.

WoW and CS are very different games, that achieve such rates for completely different reasons - CS generates more traffic because it updates a small number of objects frequently, and WoW updates a large number of objects infrequently.

I've read various papers on bandwidth, although not much has been published since this data is mostly confidential. SOE did publish some numbers, and some observations, but that was still from EQ era when things were probably differnt.

Overall, all that I did find was publications on empirical estimation of bandwidth, not the numbers themself. Going with 5Mb/hour as average, 15 mb/hour peak would probably be a reasonable estimate for usual aproaches. For MUDs or different games it would obviously be much lower, and anything real-time increases that.

But without specifying how and what your game transfers, it's impossible to say anything - you could be like SecondLife, which pushes 100 kb/sec or like a mud that sends 50 bytes/second.

Share this post


Link to post
Share on other sites
It would help to use the right unit qualifiers, too. I doubt you're talking about five millibits :-)

5 mb == 0.005 bits
5 Mb == 5,000,000 bits
5 Mib == 5,242,880 bits

Now, if you're talking about bytes (which, from context, it seems that you might be doing?) then you should use uppercase B:

5 mB == 0.005 bytes
5 MB == 5,000,000 bytes
5 MiB == 5,242,880 bytes

When calculating bandwidth, realize that load is dependent on time-of-day. When graphed, it will do a sine wave, with the difference between peaks and valleys depending on how international your client base is. There is also a difference between week days and weekends.

Without knowing more about your game, it's impossible to say what would be a reasonable bandwidth. However, you might want to go about your design the other way around: If you want to sustain X players peak, then you might conservatively budget for X*0.7 players on average. If you then have Y amount of bandwidth per month, then the amount of bandwidth used by a connection would be Y/(X*0.7). Design your protocol and game such that you are below that target.

Share this post


Link to post
Share on other sites
I use a very accurate binary class for data serialization so my method for determining bandwidth is complex.

knowing that a packet will probably have 1 state/event ID ( a short), 2 shorts for x and y, and 2 floats for velocity x and y (or shorts, tons of ways to do it). that's 2 + 2 + 2 + 8 + 8 = 22 bytes for a single player. Then add say a few state changes, 1 per packet, a simple event ID with a byte value should suffice (like change weapon then a weapon ID or something :/ ) so 22 + 2 + 1 = 25 bytes

Then for say an MMO zone or one lobby server you have 32 players that are in such close proximity that their positions need to be updated 5 times a second (not sure how accurate that is for a game, but whatever). Then say they are changing weapons at mad speeds (5 times a second I guess)

32*25 = 800 bytes per player packet. (kind of big :( lol)

So lets say these 32 players are addicts to your new game and never stop playing (or players get on and off of the game or something... chinese gold farmers or something).

32 players * 800 bytes * 5 seconds = 128,000 bytes per second or 125 kB per second

Okay these guys play for a full day straight (86400 seconds)
86400 seconds * 128000 bytes = 11059200000 bytes or 10.2996826 GB

okay then

10.2996826 GB * 30 days in a month = 308.990478 GB going out from the server

rofl, I hope I made a mistake... Okay so what we learned it it would take a lot of bandwidth for 32 players updating constantly 24/7. By breaking a large game into zones you significantly reduce the bandwidth.

READ THIS: These 5,000 users will be doing what? Is it an FPS? How large is the map or place?

You also have to think about how much CPU your servers have. If this is like some cool MMOFPS with server-side calculations then you might have problems.

Share this post


Link to post
Share on other sites
I apologize for not giving a more thorough description of the type of data we're talking about here. If you're not interested in reading more about the specifics of this question, the rest of this post won't mean much to you.

First, I apologize for improper identifiers, my math was presuming the following. 48,000GB a month is what I started from, the current available upstream transfer (note again, this is not doing math based on the connection being maxed, it's based on bandwidth that is paid for, anything over 48,000 is overage, while this number can be raised, I'm attempting to base my math on staying under that limit). From there, I go from 48,000 GB to simplify my math, I've moved to 48,000,000 MB per month, divided by 30 (to approximate, I realize a real month is not always 30 days, work with me here). So 48m/30 is 1,600,000MB a day, 66,666MB an hour. I think that math is close to being correct. So, assuming I want 5,000 users connected as my goal, I divide that 66,666 by 5,000 - giving me 13.333, or 13.3MB/hr per user. diving that down to the second, I get 3.69kb/s.

The overall packetsize has not yet been finalized, we are currently dealing with 3 different schedulings for packet creation and dispatch, from the game server itself, the exploit detection system, and the weather system - they all operate through seperate packets, of varying size (and the latter two, exploit and weather, are not involved in this math, we're referring solely to vital game communications.) What should be kept in mind here is largely FPS style updates, ranging from 20-30 packets from the client to the server a second. The return from the server is based on what is required for the client to operate. We minimize this by the very well known tactic of putting clients into "spheres" whereby they only receive data from things nearby that effect them. Anything that is not static is also updated when a player comes into range. Further, our topology seperates player load based on perceived load, which is weighted by, in order, a) cpu usage, b) amount of bandwidth being used on current server vs. specified max, and c) number of packets being processed, both outgoing and incoming. If any particular group exceeds itss max weight, or if the overall weight exceeds the overall max, the master server will choose one of two things to do, based on available resources. It will either activate a dormant server as a mirror, to split the load between the two, or will dynamically redraw the zones to add a dormant server to the pool, these splits and redraws occur in the most populated areas, the effect here results in some servers having control over large, sparsely populated areas, and several servers controlling desnsely populated areas. I'm getting off topic here though.

The only reason I'm still theorizing is much more effort is going into server-side creation than the client, and at this time stress testing with multiple clients isn't feasible, 'nor has our packet structure been finalized so I'm not able to approximate our current sizes. That's why I'm trying to base my thoughts and discussion off of theoretical data, although theoretical - hopefully based on fact enough to be close to realistic.

@Sirisian
Actually, as far as CPU vs. bandwidth goes, due to some miscalcuations solely on my part, we've ended up with MUCH more processing power than bandwidth. Basically, we've got a rack of 24 identical servers, (2.8ghz opteron, 2gb ram, 250gb storage per server), although overkill - the point was to have dormant servers as they fit into our scheme for dynamically creating and scaling zones as to assure a low lag, even in the most congested areas.

Share this post


Link to post
Share on other sites
Quote:
I get 3.69kb/s.


I think you mean 3.69 kB/s :-) That's certainly enough to run a "normal" MMO on, as long as it's not terribly kinetic.

If you do server-side physics, then the number of players you can put on the same physical server machine (assuming they have weapons, vehicles, etc, and assuming you're using a single, SMP-style memory bus), may vary from 100 to 500 or so, derived by looking at apparent capacity limits on systems like Second Life, City of Heroes and Planetside. Although Planetside with 500 people on an island will give you a tick rate of like once per second...
NUMA architectures may get you a lot more per server "machine," if you write for it, but at a significant escalation in cost per user served.

Anyway, you can optimize a lot. If people are standing still, send updates for them less often. If some quantities don't change (velocity, say), don't send those (i e, send a bit mask of what you'll send). If dead reckoning based on previous packets would get you close enough to where the player actually is, send packets less often. And, last, realize that peak simultaneous online population only happens for a few hours a day.

Share this post


Link to post
Share on other sites
Quote:
Original post by hplus0603
Quote:
I get 3.69kb/s.


I think you mean 3.69 kB/s :-) That's certainly enough to run a "normal" MMO on, as long as it's not terribly kinetic.


Maybe I should just start typing out exactly what I meant instead of using abbreviations incorrectly, thanks for powering through it though.

The issue is that "normal MMO" traffic is too slow, we've got to do better than that. Our goal is for the updates to be closer to what you'd expect from an FPS. We'll definitely have to do some squeezing, 100 users per server is definitely not cost effective. However, 500 would be arguably acceptable, however we were originally aiming for 1000 per server. All physics are done serverside, the client does approximately the same calculations for visualization, but the servers math is the final say so. I suppose as far as CPU utilization goes that's going to be left up to stress testing, there's no real way I can envision to get accurate simulations to base that off of, but I'm honestly not as concerned about CPU usage as I am bandwidth. I'll probably be eating those words once I see how much real users impact servers.

I'm wondering if with the way usage goes, assuming that the entire gaming audience is US (or possibly US/UK), that there are several hours of the day that will be more or less completely dead, that if 3.69kB/s is the max average for 5k users, that, assuming the 5k users is true, and I can assume that all of them will not be logged in simultaneously, if we can aim for something more like 5kB/s or 5.5kB/s. Although I do know if the design is based on over capacity, that I'll end up overshooting.

Also at this point all servers are single processor, although I might need to consider multi-processor and taking the NUMA approach, it wouldn't really effect the cost of the rack too terribly much, and might make the number of users per server more comfortable, and more cost effective.

Thanks your replies so far, although I look forward to more input on the subject.

Share this post


Link to post
Share on other sites
Well if you have a lot of excess CPU, you can sacrifice CPU for bandwidth. For instance a really good way to save bandwidth might be to use bit packets that optimized per bit. Also using the spatial partitioning on the server side map to find distances and occlusions is very effective.

My tutorial doesn't cover it very much, but:
http://gpwiki.org/index.php/Binary_Packet
(the c++ one optimized only for booleans to use 1 bit, but can be edited easily for any number of bits, feel free to use it's open source).

You know, use UDP if you want to save bandwidth. Having to resend packet might for one be useless with an FPS. However using a UDP/TCP system might be beneficiary depending on the complexity.

If you're the programmer then for each variable that might ever be send for the player on the server and client place a boolean variable that detects if it ever changes. This will greatly reduce bandwidth used.

Another thing to note is portal occlusion. You can also sacrifice upload for bandwidth. You can have the server only send the velocities of objects that aren't visible. Then every 2 seconds have the client send out a position packet for all of it's visible objects and allow those to be corrected. For objects not in view this will keep them accurate on a large scale map. For instance planes or vehicles and other players while the players character is inside of a building. So for instance in planetside (since they haven't released any info on this) a bomber may be flying and drop bombs. The bombs data would be send to almost everyone in a large area of the map since players in buildings can hear it and might run outside. The bombs are completely separate from the bomber.

On a side note, Planetside 2 would be nice to see.

Share this post


Link to post
Share on other sites
Dont forget to add the 22 byte UDP overhead.
It is quite much, so you have to send many things in one packet to be efficient. Like shooting (multiple bullets per packet), weapon changing, state changing, etc.
(22 byte data with 22 byte overhead... 50% of the packet is useless... hmmmm
100 byte data with 22 byte overhead is much, much better)

Share this post


Link to post
Share on other sites
Last I checked, the IP packet header size was 20 bytes, and the UDP header size was 8 bytes, for a total of 28 bytes. I could be wrong, thugh -- this is from memory.

Quote:
I might need to consider multi-processor and taking the NUMA approach


I wouldn't count on it. My feeling is that the leasing alone on a 16- or 32-CPU NUMA machine would cost more than the cost of an additional rack full of 1U pizza boxes.

For kinetic play, the best way to go is to use an input-synchronous system, such as used in City of Heroes (by Cryptic Studios) or There.com (which is based on our OLIVE platform, btw). Send only input, either as changes in input, or as a RLE encoded trail of the last N seconds of input, and calculate forward based on that. The easiest way to know you won't get too much out of sync if packets are lost is to send a traditional state record, say, every 5 seconds.

If you quantize the mouse to 4096 steps yaw (0 - 360 degrees) and 2048 steps pitch (-90 - +90 degrees), and send input commands as 8 bits, RLE encoded over the last N steps, you'll get something like:

2 bytes -- entity identifier
3 bytes -- current yaw/pitch
1 byte len + 1 byte data -- previous control state
1 byte len + 1 byte data -- current control state

Or about 9 bytes per entity per update (assuming you get a change in command rate, on average, of 1 per state update "trail" size -- you can enforce this by varying the size of the trail, too). This assumes 3D coordinates -- if you are 2D (like it seems in your initial packet description), you can reduce it further, getting rid of pitch, and compact entity ID and yaw into 3 bytes total.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!