Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 03 Jun 2003
Offline Last Active Yesterday, 10:10 PM

#5188362 Discussion about technologies used in MMO video games through browsers and mo...

Posted by hplus0603 on 21 October 2014 - 01:01 PM

It's kind of outside the scope, because it's not a MMO

I'd like to understand how you define MMO. We certainly use approximately all the same approaches as I've seen at other MMO companies. The main difference between us and, say, WoW, is that we use even less physical simulation than they do (and they mostly fake it on the client side) and our stats are more tied to real actions, rather than simulated-world actions. Not a big difference on the back-end, all in all. (That being said, the previous company I worked at, There.com, spent an insane amount of work on trying to get the physics engine to be deterministic, despite supporting late joiners and targeting very narrow bandwidth -- I don't think that's actually necessary these days!)

Most of our challenges come around having to move over 20 million user-generated pieces of 3D content into a web browser. It turns out, our users/creators have been very creative with pushing the C++/Python render engine into all kinds of unexpected corner cases, and now expect the same behavior in a new render engine. (We couldn't re-use the one from the PC, unfortunately -- C++ is only a small part of the compatibility equation; GL versions, host OS support, threading-versus-not, and a host of other issues are actually bigger obstacles.)

#5187699 Discussion about technologies used in MMO video games through browsers and mo...

Posted by hplus0603 on 17 October 2014 - 12:52 PM

FWIW, we're building a version of IMVU that runs in a browser, because PC download is not a mass-market vehicle going forward for us. (We're not the kind of thing you'd expect on Steam, for example.) We're building it on WebGL and Websockets using zero plug-ins.

Given that we already have a back-end that does > 100k concurrent users, our main challenge is actually running at good frame rates in a majority of web browsers, as well as getting from "95%" to "99.5%" web connectivity. Sure, a MacBook Pro with NVIDIA graphics running Chrome will probably work fine. Meanwhile, an Intel GMA 950 running on little Timmy's laptop with Windows XP is a different matter... and those are like 20% of the web surfing public.
Similarly, the Samsung Android browser will happily create a Websockets object -- that doesn't actually do anything. It's a dummy, dead, object.
Those kinds of real-world problems end up taking 80% of development time. The "hard" scaling and tech problems are really just the 20% of the iceberg.

We've talked a bit publicly about various technologies we use.
My talk from GDC 2011: http://www.slideshare.net/JonWatte/message-queuing-on-a-large-scale-imvus-stateful-realtime-message-queue
Chad Austin's talk from CppConf 2014: https://dl.dropboxusercontent.com/u/1602057/Presentations/Connecting%20C%2B%2B%20and%20JavaScript%20on%20the%20Web%20with%20Embind.pdf
And some of the crazy stuff we have to do to actually make it work: http://engineering.imvu.com/2014/09/26/optimizing-webgl-shaders-by-reading-d3d-shader-assembly/

#5187219 What Causes Port Mismatches ?

Posted by hplus0603 on 15 October 2014 - 01:40 PM

null pointer ...

Then you're doing something wrong.

You should only need to create the socket once, and you should only need to create one output and one input stream.
Also, if you use TCP, the socket needs to be either connected, or accepted, before you can communicate on it.

#5187059 What Causes Port Mismatches ?

Posted by hplus0603 on 14 October 2014 - 07:20 PM

I think Washu's explanation is a little to over-simplified. It is actually the case that, if a server is listening on port 1234, and accepts a connection, packets through that connection do arrive on port 1234 (although at the API level, they will be routed to a socket other than the listening socket.)

In TCP, a connection is identified by a four-tuple: Destination IP, Destination Port, Source IP, Source Port.

When a client connects to a destination IP and port (like www.google.com:80,) the source IP is determined by the network address of the client, but the source port is generally randomly chosen by the kernel/network implementation on the client.

When one host extracts the "remote address" for a connection, it gets the IP/port of the other box. Thus, if "Destination" == "server," then the server will see Source IP, Source Port, and that port will be randomly allocated by the source. UNLESS the source binds the socket to a port before attempting to connect -- this is POSSIBLE, but it is pretty much always a bad idea. (For example, there's no guarantee that that port will be available on the host.)

There is another gnarl in the story: If the client (initiator of the ocnnection) is behind a NAT router, such as a typical cable modem or WiFi router, then the Source IP, Source Port that the client provides will be substituted by the router to some other Source IP, Source Port. However, the substitution will be made "back again" for returning traffic, so the client/server communication will still work.

#5186931 Sharing violations and "Network optomisers"

Posted by hplus0603 on 14 October 2014 - 09:25 AM

a QoS tool (that generally should only do stuff when bandwidth is scarce) is unlikely to be the cause of unreliable handshaking

Unless it's written by the lowest bidder on freelancer.com, based on a specification provided by a marketing department.
Not that that would EVER happen for ANY name-brand networking or system tools.
(Actually, it happens all the time, because nobody wants to pay for quality.)

#5185593 OpenSSL tutorials?

Posted by hplus0603 on 07 October 2014 - 12:55 PM

Also, there's a few blog posts from people trying to do this, and swearing at the terrible and inconsistent design of the API.
Reading through it, I would tend to agree.
If you can use a higher-level library like libcurl, that's probably better for you.

#5185415 MMORPG separated servers (world, login) and the client to it

Posted by hplus0603 on 06 October 2014 - 06:23 PM

I still don't understand what the problem is.

The simplest, and most robust, way to spin up instances of maps is to have a mapping, server-side, of instance-id to ip-address:port, and give this out to each player that wants to join that instance. If you have the CPU/RAM, you could have thousands of instances on a physical host, and thus have thousands of ports active on that host.

A perhaps slightly more efficient variant, if your application is perfect in threading and memory management, is to spin up a single game server, and have that load each map static data once, and create a separate "game world instance" per instance with dynamic data. That will save some RAM (through static map and code sharing,) but is significantly more complex.

In this second case, you would have to give some token to each user, which would identify the instance ID when the user connects back to the server.

The simplicity of option 1) is super great -- if a server process crashes, all other instances stay up. If you need more servers, just add more possible IP addresses that the instance could be hosted on. This also scales easily to cloud hosted solution.

#5185206 MMORPG separated servers (world, login) and the client to it

Posted by hplus0603 on 05 October 2014 - 07:01 PM

I think, because i cannot open a port each time I add a new map. I want to make separate " servers" for dungeons as well and I cannot open as much port as I have to.

Why not? Will you have more than 16,000 instances of maps/dungeons on a single server?

#5185135 MMORPG separated servers (world, login) and the client to it

Posted by hplus0603 on 05 October 2014 - 10:08 AM

The problem is, if i have 10-20 maps, I will have to open 10-20 +1 port on the server computer.

Why is this a problem?

#5185055 MMORPG separated servers (world, login) and the client to it

Posted by hplus0603 on 04 October 2014 - 09:10 PM

The right answer depends on how you intend to build your world.

If your world will be made up of multiple, separate shards (areas, instances, levels, ...) then it makes sense to have the login server tell the client "your character's current area is available at foo.bar.com:12343" and have the client make a world connection to that address.

If your world view is a single, large, seamless world, then perhaps you're better off creating a "gateway" server which handles all the connections to clients -- spin up as many of these as you need, and load balance with HAProxy or an F5 or whatever. The gateway server would then make/break connections to the different world and login and whatever-else servers, and funnel requests from all its clients across each of those connections.

In general, though, the best you can do is to make sure that you're well defined in what your protocols look like. What kinds of data does the client send, and when, and what kinds of data does each server send, and when? The more you can make this "stateless" (as in "I just stream out information X on interval Y") rather than "RPC" (node X sends a request and then waits for node Y to send a response) the better your system will likely perform in reality.

#5184814 Why do i get lag?

Posted by hplus0603 on 03 October 2014 - 01:25 PM

I think Raknet uses UDP, and thus doesn't need TCP_NODELAY. I missed the mention of raknet in the initial question.

Does Raknet do packet scheduling? Does it actually send packets at the same frequency as your simulation time steps?

In general, you should assume that full round-trip time will be bigger than your time step, and schedule all commands for some future tick, rather than requiring each client to get the data to you each time you take a step.

#5184418 Why do i get lag?

Posted by hplus0603 on 01 October 2014 - 03:22 PM

In typical lockstep simulations, you will send commands for X steps into the future. This means that the commands will actually be queued, and take effect later.

If you are using TCP, are you turning on TCP_NODELAY on your sockets?

#5183770 rate/hate/appreciate my mog server update model

Posted by hplus0603 on 29 September 2014 - 09:33 AM

I was asking for input on the whole [everything is-a record has-a record_session updates-a session] programming model.

That's one way of expressing the data-flow plumbing. I'm sure it can be made to work fine.

I find that what ends up being interesting in networked games is not the plumbing (important as it might be) but instead the game rules and how they collaborate with UI/display to give a particular "feel" to the game.

#5181270 What can be scripted and what should not in a server?

Posted by hplus0603 on 18 September 2014 - 09:04 AM

That depends entirely on your specific needs.

If you're trying to get thousands of players interacting on a single piece of server hardware with advanced physical interaction, being able to optimize using a profiler and staying away from GC pauses will be important.

If you're putting less than a hundred players on a single server and the main things the server does are enforcing rules on a per-skill basis and verifying/storing data, you could write the entire thing in Lua and it would run fine.

Separately, trying to debug a system that runs partially in a scripting language and partially in a native language can be challenging, because there are very few debugging systems that can "step into" across the language boundary.

#5180465 sqlite as db backend for mmorpg?

Posted by hplus0603 on 15 September 2014 - 09:15 AM

Not really. Anytime I've thrown real, heavy, multi-process workloads at sqlite, it's choked. It's not what it's good at.