Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 03 Jun 2003
Online Last Active Today, 02:57 PM

#5160734 Multithread server design

Posted by hplus0603 on 15 June 2014 - 06:36 PM

Put in print statements in the session constructors (main constructor, copy constructor) as well as the session destructor.
Then put in print statements in the code that causes the problems.
Check out the sequence of events.
If you think you need more information, then start with the debugger and put a breakpoint in the destructor, and see what's calling it.

Are you using boost::shared_ptr<> for the objects you pass in to the io_service? With boost::bind?

#5160316 MMOs and modern scaling techniques

Posted by hplus0603 on 13 June 2014 - 09:30 AM

i mean when ping from A city to B city, and ping gives 50 ms it means that it really is avaliable in b after a 50 ms delay? (ping gives one way travel time?

Ping gives the two-way travel time. The time from A to B is half that of Ping.

If the sending application and the receiving application are written properly, and are running on servers that are properly provisioned (not overloaded,) then the application-to-application time will be very similar to the ping time. However, if there are problems in the implementation of the application, or the management of the server, application-to-application time may be a lot longer.

#5160159 Client loading entities

Posted by hplus0603 on 12 June 2014 - 04:43 PM

you don't want to waste network bandwidth by constantly sending everyone "create X entity" messages when they already have those entities loaded

The google term you want is "interest management."

In general, this is a server-side mechanism that in some cases take almost as much work as the server-side simulation itself (because it, too, is n-squared in complexity.)

#5159907 MMOs and modern scaling techniques

Posted by hplus0603 on 11 June 2014 - 06:49 PM

First: Try writing a robust real-time physics engine that can support even a small number like 200 players with vehicles on top of the web architecture that @snacktime and @VFe describe. I don't think that's the right tool for the job.


You'd think that, given how much has been said on the matter, that there would be at least one instance of people talking about using different methods, but I've not seen one.

Personally, I've actually talked a lot about this in this very forum for the last ten-fifteen years. For reference, the first one I worked on was There.com, which was a full-on single-instance physically-based virtual world. It supported full client- and server-side physics rewind; a procedural-and-customized plane the size of Earth; fully customizable/composable avatars with user-generated content commerce; voice chat in world; vehicles ridable by multiple players, and a lot of other things; timeframe about 2001. The second one (where I still work) is IMVU.com where we eschew physics in the current "room based" experience because it's so messy. IMVU.com is written almost entirely on top of web architecture for all the transactional stuff, and on top of a custom low-latency ephemeral message queue (written in Erlang) for the real-time stuff. Most of that's sporadically documented in the engineering blog: http://engineering.imvu.com/

#5159811 MMOs and modern scaling techniques

Posted by hplus0603 on 11 June 2014 - 11:17 AM

Distributed archtiectures are the right tool for the job. What I've seen is that the game industry is just not where most of the innovation with server side architectures is happening, so they are a bit behind.

You are absolutely correct that game developers often don't even know that they don't know how to do business back ends effectively. Game technologies in general, and real-time simulations in particular, are like race cars running highly tuned engines on a highly controlled race track. Business systems are like trucks carrying freight over a large network of roads. Game developers, too often, try to carry freight on race cars.

Race cars are great at what they do, though. That's the bit that is fundamentally different from the kind of business object, web architectures that you are talking about. (And I extend that to not-just HTTP; things like Storm or JMS or Scalding also fits in that mold.) When you say that "the optimization of a single server doesn't matter," it tells me that you've never tried to run a real physics simulation of thousands of entities, that can all interact, in real time, at high frame rates. There are entire classes of game play and game feel that are available now, that were not available ten years ago, explicitly because you can cram more processing power and higher memory throughput into a single box. A network is, by comparison, a high-latency, low-bandwidth channel. If you're not interested in those kinds of games, then you wouldn't need to know about this, but the challenge we discuss here is games where this matters just as much as the ability to provide consistent long-term data services to lots of users.

I see, equally often, business-level engineers, and web engineers, who are very good at what they do, failing to realize the demands of game-type realtime interactive simulation. (Or, similarly, multimedia like audio -- check out the web audio spec for a real train wreck some time.)

A really successful, large scale company, has both truckers and race car drivers, and get them to talk to each other and understand each other's unique challenges.

#5159644 MMOs and modern scaling techniques

Posted by hplus0603 on 10 June 2014 - 08:05 PM

modern online games are already using web scaling approaches

There really are two kinds of things going on in most games.
One is the real-time component, where many other players see my real-time movements and my real-time actions.
The other is the persistent game state component, which is not that different from any other web services application.
The trick is that, if you try to apply web services approaches to the real-time stuff, you will fail, and if you try to apply the real-time approach to the web services stuff, you will pay way too much to build whatever you want to build.

Some projects have tried to unify these two worlds in one way or another (for example, Project Darkstar) but it's never really worked out well. In my opinion, doing that is a bit like trying to unify real-time voice communication with one-way streamed HD video delivery -- the requirements are so drastically different, it just doesn't make any sense to do that.
That analogy isn't entirely bad: One-way streamed video is a lot more like "web apps" than real-time voice communication, which is more like interactive games, except without the N-squared interactions and physics rules.

So, one way of approaching the scalability problem is to dumb down the gameplay until it's no longer a real-time, interactive, shared world. That doesn't make for fun games for those people who want those things. The other way is to be careful about only building the bits that you really do need for your desired gameplay, and optimizing those parts to the best of the state of the art.
Getting the game developers, and the web developers, to understand each other, and work under the same roof, AND making sure that the right tool is used for the right problem, while delivering the needed infrastructure for gameplay development, is really hard to pull off; I've seen many projects die because it misses one of those "legs" and doesn't even know what it's missing. (A bit of the Dunning-Kruger effect, coupled with "I've got a great hammer!")

#5159530 game's protocol between client/server

Posted by hplus0603 on 10 June 2014 - 10:14 AM

In general, you want your code to be as table-driven and data-driven as possible. If you find yourself writing the same code multiple times, find a way to compress it into a table lookup or similar.

Thus, the creation of various Actions from string might look like:


class Action { /* abstract base class */ };
class MoveAction : public Action { /* concrete class */ };

class KillAction : public Action { /* concrete class */ };

class DieAction : public Action { /* concrete class */ };

struct NamedAction {
  char const *name;
  Action (*makeAction)(JSON *, Context *);
g_allActions[] = {
  { "move", &MakeMoveAction },
  { "kill", &MakeKillAction },
  { "die", &MakeDieAction },
Action *ActionFromJSON(JSON *json, Context *ctx) {
  char const *type = json->KeyAsString("type");
  if (!type) return nullptr;
  for (size_t i = 0; i != countof(g_allActions); ++i) {
    if (!strcmp(g_allActions[i].name, type)) {
      return (*g_allActions[i].makeAction)(json, ctx);
  return nullptr;


In a larger system, you'll likely use a hash table instead of a linear table scan, and each action kind will register itself in this table on start-up in some way.

#5159528 MMOs and modern scaling techniques

Posted by hplus0603 on 10 June 2014 - 10:08 AM

If I remember correctly, City of Heroes actually would spin up additional instances of "the same area" when player counts got too high. We also did that in There.com, to support large parties. This is good from a player point of view, too; you'd rather see all the players you can interact with, than having 1000 players in the same area and you can only see the nearest three meters...


When it comes to distributed simulation with scalable player density (not just player count,) that's a much tougher nut to crack. Web applications do not have nearly the level of coupling between different business objects that games do. When researching this back in the day, the best option I could come up with would be one where objects are only allowed to affect other objects one tick into the future, and there would be a big cross-bar of messaging of "these are all the interactions I detected this turn" that would be exchanged between the simulation servers between ticks. There would still be a n-squared scaling problem between servers, but with 40 or even 100 Gbps local networking, you can get pretty high in numbers before that becomes a bottleneck. This architecture lets objects be homed on a particular server, and then roam the entire world (assuming the server can keep the static world in RAM) and interact with all other objects with a one-tick delay. This is also very similar to the original DIS distribution model, except DIS typically had a single entitly (plane, tank, etc) per network node, rather than, say, 1000 entities per node.


The other problem is that dense simulation IS an N-squared problem. If everyone piles in a big dogpile, you will potentially have interactions between all players to all other players. There's no way around that. Same thing as collision detection -- in the worst case, every object really DOES interact every other object.

#5159178 Peer to Peer Online Game

Posted by hplus0603 on 08 June 2014 - 09:49 PM

I just don't want to have to send position updates for all monsters to a lot of players from one machine, so I'd have to come up with a deterministic movement simulation.

What problem are you actually trying to solve?

I e, can you provide X here? "Right now, PvE multiplayer games suffer because X."
And can you provide Y here? "My solution solves X by doing Y."

#5158774 How to keep track of sessions in HTTP requests

Posted by hplus0603 on 06 June 2014 - 02:03 PM

Actually, I split this to a separate topic, because it's a different question!


Generally, when dealing with sessions, you use a data store with timeout or time-to-live of some sort. Examples include memcached, Redis, or Cassandra.

For HTTP, when a user logs in, you create a new session and identify it with a session ID. Use a strong random number and verify that it's not already existing. Store information about the session in your data store, and store the session ID in a cookie in the browser.

When you receive HTTP requests with a session ID cookie, look up that ID in your data store; if it's still there, the session is valid.

The session ID needs to be hard to guess, and you need to have billions more session IDs than you have active sessions to defend against guessing attacks, but that's easy with a 128 or even 256 bit strong random number as your session ID, coupled with not allowing more than a dozen bad logins or bad sessions from the same source IP in some amount of time (say, 5 minutes.)


PHP specifically has some session management built in. By default, it just stores the data in a local file, so it only works on a single machine; you can extend it to use memcached or whatever once your service outgrows a single server. But it's also pretty reasonable to build your own as described above.

#5158420 Preventing cheating on game's server side

Posted by hplus0603 on 05 June 2014 - 09:37 AM

If you process the next message in the input queue as soon as it is there, and discard messages that are against the rules, then the "cleaning" will happen by itself, because if someone sends a second message, that will immediately be discarded (because it's against the rules.)

#5157243 multi-player games and physics engines

Posted by hplus0603 on 31 May 2014 - 06:30 PM

Yes, there is extra cost to re-playing physics.


Sometimes, you can store the state of each player at each step in the past for some amount of replay log, and only replay the entities that got invalidated. If entities interact (collide, weapon hit, etc,) then those sets of entities need to be rolled forward from the time when they interacted.


Often, physics simulation isn't particularly expensive, so the cost to doing this is acceptable, especially for fast-paced shooters with less than a hundred players per level.

Once the player count goes up, it starts becoming more important to hide latency in other ways, either by simulating commands when-received, or forcing players to lead shots, or something else.

#5155751 Packet combining

Posted by hplus0603 on 24 May 2014 - 05:35 PM

How about the "some packets to player A, some to player B" issue? How is that dealt with?

1. Set a network tick rate, such as 10 Hz or 20 Hz or 60 Hz
2. Each time a network tick comes up, iterate over all connected players
3. For each connected player, glom together all the updates that that particular player needs to see, and send as one packet to that player

#5155131 Multiplayer game architecture question

Posted by hplus0603 on 21 May 2014 - 04:19 PM

It's impossible to tell based on your description. As long as you solve synchronization such that you never get deadlocks, you're probably OK. Also, you don't want to be spawning a new thread each time you have anything to send to the server, but rather re-use a worker thread that pulls data to send from a queue.

Also, you generally don't want to be notifying more than one piece of the application about incoming data. Instead, it's common to use a model-view-controller system where the network acts as controller, and affects the model; the view then gets notified by the model when data changes. One set of data might be "whose turn is it?" and when that changes, the view changes how it displays itself.

#5154657 Using weak server side hash's.

Posted by hplus0603 on 19 May 2014 - 11:16 AM

The first question is: What kind of attack are you trying to defend against?
The second question is: What methods already exist for defending against that attack?
The third question is: How is this method different, for both good and bad purposes?

For example, if you calculate sha1 on the client, then that means that knowing the sha1 of the password is sufficient to log in -- you don't need to recover the original password, because the only thing the server sees is the sha1. You have effectively just changed the password to be a known 160-bit pattern.

Similarly, if you use hash on the server, then you do that to defend against the risk that the server database leaks. The two kinds of exposure that leads to is making it possible to log in to your service, and using the same password to log in to another service as the same user. Weak hashes allows the attacker to construct a duplicate for logging into your service much easier.