Howdy, I've been doing some crude math to help refine what I have to know about bandwidth constraints, unfortunately what I don't have access to are the average bandwidth consumption of clients of popular games out right now. So I've been using some theoretic numbers with the help of some other brain stormers to guesstimate what other games are using. We've come up with the average bandwidth usage per user in World of Warcraft to be about 5mb/hr, and amongst us were unable to establish a good number for higher use games such as CounterStrike, so we're using both 20mb/hr and 40mb/hr per user for those games. I'm basing the rest of my math upon the bandwidth available to us, which is 48,000gb a month per rack currently. or 48million mb a month, doing some crude math (such as uing 1000 for my math, as opposed to 1024). The reason for this way to assume a steady number of connections, using a steady amount of bandwidth, breaks down such as this: 5mb an hour, 13,333 constant users 20mb an hour, 3,333 constant users 40mb an hour, 1,666 constant users Our goal is 5,000 users connected, which works out to 13.3mb/hr max per user, or 3.69kb/s. This is again using the bandwidth that's currently available to the rack (without any other adjustments). The only data I've been able to find suggests that current implementations of CS (source) use a total of about 7kb/s, (up and down), so it's logical for me to assume that, if 3.69kb/s is the max for our upstream (this is not the max of our pipe, just of our metered upstream bandwidth), assuming some clever networking, we could in theory keep our bandwidth under 3.69kb/s per client. Here's where my question gets theoretical (well, more theortical). All this math is done assuming constant connections, we all know that bandwidth is not a constant thing, crowded areas do more and open areas do less. Also, this is assuming a steady number of people connected all the time, which we also know is not true. So, I'd like some assumptions, or somewhere to find some data about the average curve of time people are connected (for primarily US audiences, US/UK is fine), I expect that it also depends on the day of the week, i'm assuming less people are on from 2-6am monday through friday then they are on the weekend when folks can stay up late. Any suggestions on, if the system is designed around 5000 simultaneously connected users per rack, where's my margin for total users that can be supported? I'm hoping some folks here with a bit more hands on knowledge will have a good idea or theory to sort it out. Thanks for your time.

You won't know anything for sure until you test it, or run the numbers on your protocol and world design.

How many state updates, how large are they, what is the player distribution, and so on.

WoW and CS are very different games, that achieve such rates for completely different reasons - CS generates more traffic because it updates a small number of objects frequently, and WoW updates a large number of objects infrequently.

I've read various papers on bandwidth, although not much has been published since this data is mostly confidential. SOE did publish some numbers, and some observations, but that was still from EQ era when things were probably differnt.

Overall, all that I did find was publications on empirical estimation of bandwidth, not the numbers themself. Going with 5Mb/hour as average, 15 mb/hour peak would probably be a reasonable estimate for usual aproaches. For MUDs or different games it would obviously be much lower, and anything real-time increases that.

But without specifying how and what your game transfers, it's impossible to say anything - you could be like SecondLife, which pushes 100 kb/sec or like a mud that sends 50 bytes/second.

I use a very accurate binary class for data serialization so my method for determining bandwidth is complex.

knowing that a packet will probably have 1 state/event ID ( a short), 2 shorts for x and y, and 2 floats for velocity x and y (or shorts, tons of ways to do it). that's 2 + 2 + 2 + 8 + 8 = 22 bytes for a single player. Then add say a few state changes, 1 per packet, a simple event ID with a byte value should suffice (like change weapon then a weapon ID or something :/ ) so 22 + 2 + 1 = 25 bytes

Then for say an MMO zone or one lobby server you have 32 players that are in such close proximity that their positions need to be updated 5 times a second (not sure how accurate that is for a game, but whatever). Then say they are changing weapons at mad speeds (5 times a second I guess)

32*25 = 800 bytes per player packet. (kind of big :( lol)

So lets say these 32 players are addicts to your new game and never stop playing (or players get on and off of the game or something... chinese gold farmers or something).

32 players * 800 bytes * 5 seconds = 128,000 bytes per second or 125 kB per second

Okay these guys play for a full day straight (86400 seconds)
86400 seconds * 128000 bytes = 11059200000 bytes or 10.2996826 GB

okay then

10.2996826 GB * 30 days in a month = 308.990478 GB going out from the server

rofl, I hope I made a mistake... Okay so what we learned it it would take a lot of bandwidth for 32 players updating constantly 24/7. By breaking a large game into zones you significantly reduce the bandwidth.

READ THIS: These 5,000 users will be doing what? Is it an FPS? How large is the map or place?

You also have to think about how much CPU your servers have. If this is like some cool MMOFPS with server-side calculations then you might have problems.

I apologize for not giving a more thorough description of the type of data we're talking about here. If you're not interested in reading more about the specifics of this question, the rest of this post won't mean much to you.

First, I apologize for improper identifiers, my math was presuming the following. 48,000GB a month is what I started from, the current available upstream transfer (note again, this is not doing math based on the connection being maxed, it's based on bandwidth that is paid for, anything over 48,000 is overage, while this number can be raised, I'm attempting to base my math on staying under that limit). From there, I go from 48,000 GB to simplify my math, I've moved to 48,000,000 MB per month, divided by 30 (to approximate, I realize a real month is not always 30 days, work with me here). So 48m/30 is 1,600,000MB a day, 66,666MB an hour. I think that math is close to being correct. So, assuming I want 5,000 users connected as my goal, I divide that 66,666 by 5,000 - giving me 13.333, or 13.3MB/hr per user. diving that down to the second, I get 3.69kb/s.

The overall packetsize has not yet been finalized, we are currently dealing with 3 different schedulings for packet creation and dispatch, from the game server itself, the exploit detection system, and the weather system - they all operate through seperate packets, of varying size (and the latter two, exploit and weather, are not involved in this math, we're referring solely to vital game communications.) What should be kept in mind here is largely FPS style updates, ranging from 20-30 packets from the client to the server a second. The return from the server is based on what is required for the client to operate. We minimize this by the very well known tactic of putting clients into "spheres" whereby they only receive data from things nearby that effect them. Anything that is not static is also updated when a player comes into range. Further, our topology seperates player load based on perceived load, which is weighted by, in order, a) cpu usage, b) amount of bandwidth being used on current server vs. specified max, and c) number of packets being processed, both outgoing and incoming. If any particular group exceeds itss max weight, or if the overall weight exceeds the overall max, the master server will choose one of two things to do, based on available resources. It will either activate a dormant server as a mirror, to split the load between the two, or will dynamically redraw the zones to add a dormant server to the pool, these splits and redraws occur in the most populated areas, the effect here results in some servers having control over large, sparsely populated areas, and several servers controlling desnsely populated areas. I'm getting off topic here though.

The only reason I'm still theorizing is much more effort is going into server-side creation than the client, and at this time stress testing with multiple clients isn't feasible, 'nor has our packet structure been finalized so I'm not able to approximate our current sizes. That's why I'm trying to base my thoughts and discussion off of theoretical data, although theoretical - hopefully based on fact enough to be close to realistic.

@Sirisian
Actually, as far as CPU vs. bandwidth goes, due to some miscalcuations solely on my part, we've ended up with MUCH more processing power than bandwidth. Basically, we've got a rack of 24 identical servers, (2.8ghz opteron, 2gb ram, 250gb storage per server), although overkill - the point was to have dormant servers as they fit into our scheme for dynamically creating and scaling zones as to assure a low lag, even in the most congested areas.

Quote:Original post by hplus0603

Quote:I get 3.69kb/s.

I think you mean 3.69 kB/s :-) That's certainly enough to run a "normal" MMO on, as long as it's not terribly kinetic.

Maybe I should just start typing out exactly what I meant instead of using abbreviations incorrectly, thanks for powering through it though.

The issue is that "normal MMO" traffic is too slow, we've got to do better than that. Our goal is for the updates to be closer to what you'd expect from an FPS. We'll definitely have to do some squeezing, 100 users per server is definitely not cost effective. However, 500 would be arguably acceptable, however we were originally aiming for 1000 per server. All physics are done serverside, the client does approximately the same calculations for visualization, but the servers math is the final say so. I suppose as far as CPU utilization goes that's going to be left up to stress testing, there's no real way I can envision to get accurate simulations to base that off of, but I'm honestly not as concerned about CPU usage as I am bandwidth. I'll probably be eating those words once I see how much real users impact servers.

I'm wondering if with the way usage goes, assuming that the entire gaming audience is US (or possibly US/UK), that there are several hours of the day that will be more or less completely dead, that if 3.69kB/s is the max average for 5k users, that, assuming the 5k users is true, and I can assume that all of them will not be logged in simultaneously, if we can aim for something more like 5kB/s or 5.5kB/s. Although I do know if the design is based on over capacity, that I'll end up overshooting.

Also at this point all servers are single processor, although I might need to consider multi-processor and taking the NUMA approach, it wouldn't really effect the cost of the rack too terribly much, and might make the number of users per server more comfortable, and more cost effective.

Thanks your replies so far, although I look forward to more input on the subject.

Well if you have a lot of excess CPU, you can sacrifice CPU for bandwidth. For instance a really good way to save bandwidth might be to use bit packets that optimized per bit. Also using the spatial partitioning on the server side map to find distances and occlusions is very effective.

My tutorial doesn't cover it very much, but:
http://gpwiki.org/index.php/Binary_Packet
(the c++ one optimized only for booleans to use 1 bit, but can be edited easily for any number of bits, feel free to use it's open source).

You know, use UDP if you want to save bandwidth. Having to resend packet might for one be useless with an FPS. However using a UDP/TCP system might be beneficiary depending on the complexity.

If you're the programmer then for each variable that might ever be send for the player on the server and client place a boolean variable that detects if it ever changes. This will greatly reduce bandwidth used.

Another thing to note is portal occlusion. You can also sacrifice upload for bandwidth. You can have the server only send the velocities of objects that aren't visible. Then every 2 seconds have the client send out a position packet for all of it's visible objects and allow those to be corrected. For objects not in view this will keep them accurate on a large scale map. For instance planes or vehicles and other players while the players character is inside of a building. So for instance in planetside (since they haven't released any info on this) a bomber may be flying and drop bombs. The bombs data would be send to almost everyone in a large area of the map since players in buildings can hear it and might run outside. The bombs are completely separate from the bomber.

On a side note, Planetside 2 would be nice to see.