Jump to content

  • Log In with Google      Sign In   
  • Create Account


Member Since 03 Jun 2003
Offline Last Active Yesterday, 10:00 PM

#5315959 Commiting persistent server data

Posted by on Yesterday, 09:47 AM

there's no good reason to keep data in memory that you're not using, and may never use again (e.g. if a player never comes back).

Shh! Don't tell the Redis folks!

(FWIW: We run the biggest single Redis instance I know of, with 768 GB of RAM. This turned out to be a mistake, because the entire kernel locks up for 10 seconds each time it forks to checkpoint the data.)

How would I go about implementing

At that point, you're building the lowest-level component of a database (or, for that matter, file system) which is "indirect block allocation and management."
A very simple way to do that is to split your file into chunks, say 1 MB each, and have each chunk link to the next chunk when it gets full. To read all the data, you follow the chain of links and concatenate all the data.
A slightly more sophisticated way is to make the first chunk be an array of chunk offset, and each time you need another 1 MB chunk, add a new offset to the table, and when the table runs out, you either say "file is full," or you apply the linked-list of table chunks, or you add a second layer of indirection.
(Chunk size varies by application -- 1 MB may be way too big or not big enough, depending on what you're doing.)
An even more sophisticated way of doing this is to structure your data in an ordered index -- at this point, you'll want to read up on B-trees, B*-trees, and other such structures, because you're well on your way to building your own database!

Simple math example:
Let's assume 1 MB chunks. Let's assume 64 bit file offset.
1 MB can fit 128K of file offset pointers. Each pointer references a 1 MB chunk of file data.
Maximum size of data stored in file: 128K * 1M == 128 GB of data.

#5315887 Commiting persistent server data

Posted by on 19 October 2016 - 06:41 PM

if I were to keep the data in a file in disk and if the data is unsorted, wouldn't it take too long to query what I need from that file?

If you don't save/load everything, but keep everything in an unstructured file on disk, then yes, that is kind-of the slowest of both worlds :-)

If you save/load pieces, then you typically either put each piece in some kind of database. This can be something heavyweight like MySQL or DB/2, or something simple like Berekeley DB or just a file with a table of contents index at the front, or something super-simple like a file per user/entity (with some recursive directory hierarchy to avoid 100,000 files in one directory.)

Also if the memory usage isn't a concern, is it still a bad idea to keep all data in memory?

If the performance is fine (maybe you can save the data asynchronously?) and you have no better use for the RAM, then that's fine.

Another option you may want to consider is memory mapping the file, which makes the file be automatically saved all the time without you having to worry about it.
mmap() on Linux; CreateFileMapping() on Windows.

#5315843 Commiting persistent server data

Posted by on 19 October 2016 - 11:15 AM

Some random points:

On writing binary data:
If you have a big struct of "everything" without any pointers in it, a single call to write() will write that out just fine.
Typically, you'll want to write() to a new file (foo.data.tmp) and then use rename() to replace the old copy with the new in an atomic operation (rename(foo.data.tmp, foo.data))
Typically, you also sync() and fdatasync() after closing the file before renaming, and then after renaming.
If you find that you need to be selective about what you write to disk, writing by appending to a big memory buffer, and then writing that big buffer to disk all at once, is another way to get good I/O performance.

On keeping data in memory:
You need to keep all ACTIVE data in memory, so that you don't block gameplay on accessing the database or disk.
However, when a user logs out, it's totally fine to purge that data from memory. When a user logs in again, re-load the data.

On arranging data for querying:
Whenever you find that you have to "look for data," you typically need to use a hash table (such as std::unordered_map<>).
Linear scans through arrays is fine for small arrays that fit in a couple of cache lines, but will suck fiercely once the array gets big.
If you need the data to be sorted, use a tree of some sort instead (such as std::map<>).

#5315202 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 14 October 2016 - 09:44 AM

I hope there aren't any floating point errors that can accumulate and mess up my timers with time.

If you use "float" then, yes, after hours of uptime, you may accumulate drift.
If you use "double," it will happen after years of uptime, which probalby is an OK trade-off.
An alternative is to count time in quanta (either "microseconds" or "simulation ticks" or whatever) and use an int32 or int64.

Separately: It's more likely that the rate of time progress on your PC will be slightly different from the rate of time progress on the server.
Thus, it is good to include timing information in each packet, and if you find that the PC is "too different" from the server, adjust the PC.
(This also happens when network conditions change and latency increases/decreases.)
Typically, you will never allow the PC to be ahead of the server, so if it is, immediately adjust the clock backward to match.
If the PC is too far behind the server, adjust the clock forwards by 1/10th of the delta. That way, it will be reasonably smooth when catching up, without making a large jump.

#5314881 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 12 October 2016 - 12:08 PM

I need to time perfectly when I press 'enter' and start the client app

Your "what time is it" (read the clock) function should include an "adjustment" parameter.
The clock you return to the game should be SystemClock() + Offset.
Then, your server should send "the time is now X" data in each packet it sends to the clients.
The clients can then compare the timestamp they get from the server, to the timestamp they have locally, and adjust the Offset to make them line up.

#5314753 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 11 October 2016 - 04:40 PM

The problem might not be that the laptop is "slow," but that scheduling of threads happens. If the code puts some data to the network library, then renders a frame, and then tells the network library to flush data to network if any is available, for example, then the time for rendering will be included in your measured latency.
If the three or four clients each render, and have to wait for the others to finish rendering before they can start their frame, then you may also be bound on the speed of your GPU's ability to work on many different things.

I believe that the internals of RakNet do not send a message immediately when you write some data, but instead collects possibly multiple sends, until it actually "flushes" the data into a packet.
Thus, your latency will always show whatever internal scheduling latency and batching happens in both your client, and the networking library. 20 milliseconds is not a terribly bad amount of latency.

#5314737 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 11 October 2016 - 03:15 PM

It depends. What else is the computer doing at the same time?
Is it rendering three different windows?
Is the graphics loop involved?
Given that you're using RakNet, and RakNet schedules its own packet sends (a send isn't necessarily generated immediately when you write some data,) I would expect that kind of latency even for local round-trips.

#5314727 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 11 October 2016 - 02:26 PM

I can't filter packets by processes or programs, how do you even use this stuff

You can filter packets by protocol type (TCP, UDP, etc) and port number.
Presumably your game uses a known port, so filtering by that should be easy!
Then, each separate process will have a separate source port for the connection (in addition to the server port.)

#5314542 Running both server and client on one PC causes lag despite ping < 1 ms.

Posted by on 10 October 2016 - 09:29 AM

I agree with "Kylotan," "lagging" is not a sufficiently deep description to draw any conclusions from.

A few pieces of data you should log/plot from each of your server/clients:
- Average number of ticks/steps/frames per second, every 5 seconds or so
- Maximum time between successive ticks/steps/frames per 5 seconds or so
- Received and sent number of packets per second
- Received and sent client/server clock values (you should have each packet include the current local clock, and the last-received remote clock)

Making sure that the log starts each line with date/time and milliseconds-since-game-start, will make it easier to match the log files up later.

Another great tool for debugging is Wireshark. You should absolutely get a copy of Wireshark and install it (if you don't have it,) and use it to capture all the traffic to verify that the data and packets are what you think they are!

#5313927 Need a simple way to store all received packets and then pass them to the cor...

Posted by on 04 October 2016 - 04:01 PM

Typically, you will have a hash table (or map, or dictionary) that uses the remote IP address/port as key, and uses the player ID (or player object) as value.
Then, you can look up which player it is, from the source of the incoming data.

There are some session hijacking risks involved, so using a secondary method to authenticate the connection (using encryption keys and signatures) is useful for bigger games with more risk, but you don't need to worry about that for now.

#5313926 Create player accounts on custom server (for steam users)

Posted by on 04 October 2016 - 04:00 PM

Is there any kind of critical user-data that is stored on the server (behind the scenes) of which i'm not aware of?

I can't read your mind, so I don't know what you're aware of :-)

Is the serversoftware (as an example Apache) storing the IPs of incoming traffic somewhere on the system?

This typically goes into access.log. There may be other Apache logging, too, such as Referer headers. Finally, once you scale the system, you will start getting logging from your proxies and firewalls as well.
That being said, a list of IP addresses without email addresses or names is not particularly sensitive for most use cases (especially for games.)

what to do if A) the player (somehow) loses his URL or B) someone gets hold of this URL

There is no solution that solves for all of the things you want at once. Engineering is all about making the correct trade-offs for the problem at hand!

#5313268 What are the typical requirements of browser-based MMORPG architectures?

Posted by on 29 September 2016 - 10:38 AM

But they also got an interesting GDC talk out of the effort, so it wasn't a total loss :-)

#5313047 What are the typical requirements of browser-based MMORPG architectures?

Posted by on 28 September 2016 - 09:47 AM

If you have data that you want to persist across sessions, there's really no reason to not put it in a SQL database.
Worst case, "id + JSON-as-TEXT" works as a key/value store. Adding metadata (like "what zone did the player last checkpoint in" or "what level has the player achieved" or "what account ID is this data associated with") to indexed fields makes certain kinds of gameplay management easier.

Redis is great for data that needs the data structures it provides, and data that you can always put a finite lifetime on.
Data without an expiration time in Redis is bad.
Also, data sets that need to be transactional and grow are not great matches, because once you have a 512 GB Redis server, Linux will take 5-10 seconds just to fork() each time you want to checkpoint, and while doing that, the entire machine is unavailable.

The reason I recommend against Mongo is that they still haven't really solved some important long-term operational issues, such as consistent backups while keeping availability.
The recommendation is literally "run a replica in another data center and hope nobody finds a way to inject bad delete commands."
(Although their paid, hosted, trust-them-with-all-your-backup-data solutions claim to solve this -- I have no data points on how well that works or what it costs.)

at the high level of box-ticking they all did similar things but practically they were all architected differently

Kylotan, I think you're adding great examples of the main caveat that I wanted to convey.
I'd like to push back a little bit against this, though. Yes, you DID architect them differently, but did you HAVE to?
I've worked on games and projects with several different graphics/rendering and physics engines, and they are all architected differently.
But, looking at them, and squinting a bit -- they could probably all have saved time by starting with something like Unreal Engine 4, had that existed at the time. (The available offerings were less complete at the times, and thus different decisions were made at the times.)

#5312950 What are the typical requirements of browser-based MMORPG architectures?

Posted by on 27 September 2016 - 08:45 PM

plus MongoDB for the game data

Where I come from, friends don't let friends use Mongo. You should look really hard at alternatives, such as Cassandra, or Redis, or SimpleDB, before you actually go down that route.

However when you say "game data," do you mean things like "current hitpoints" and "location of NPC X"?
Generally, putting that kind of data into any kind of network-attached memory is too expensive.
Game servers will keep that kind of data in-RAM, and make sure that clients are connected/routed to the appropriate server instance for the level/area/zone they are currently observing/interacting with.
When something important happens (pick up loot, etc) the game server will fire off an update message to a back-end database; other than that, if the game server crashes, players get kicked off and objects reset to initial locations.
Given that well-debugged servers crash very seldom, this is usually the right trade-off, because trying to persist ephemeral data has an unreasonably high cost.

#5312899 What are the typical requirements of browser-based MMORPG architectures?

Posted by on 27 September 2016 - 02:15 PM

First, there is a big caveat here: On the continuum between "extremely general," and "game specific," it's really hard to find a viable, useful, middle ground.
All MMORPGs need servers, so general server-management infrastructure is probably useful -- think "docker" or "elastic beanstalk" or such. Totally general, not game specific.
Similarly for storage infrastructure -- pick something general and well known, either relational (MySQL or Postgres) or NoSQL (Redis or Cassandra, or the cloud-based BigTable or SimpleDB.)
All MMORPGs need some kind of graphics, and user input, and audio. There are APIs in browsers for these things, and those APIs have a bunch of warts, so compatibility helpers are useful, as are 2D and 3D renderers, as ar even full-fledged game engines. That's a super-big, hundreds-of-man-years area.
Regarding quests, that's also kind-of generic, but also kind-of genre specific. Quests are found in most games -- campaign mode of RTS-es, single player FPS, single-player RPG, action-adventure, and a bunch of other game kinds need this just as much as "MMO" anything.

Now, once you start getting into any more details, you end up with genre specific requirements.
There are services around the periphery that can be useful to all modes (game matchmaking, server selection, account management, etc) but the gameplay does dictate the optimal architecture, and if you don't choose the optimal architecture, you will be at a cost disadvantage compared to your competitors.
Do you need server-side simulation? Is that simulation turn based or real time? Does it include physics simulation or just game rules? The approach you take matters. Travian is different from World of Warcraft is different from Planetside.
Calculate-forward-on-request (like Travian, etc) is very different from ticksd-simulation with physics (like Planetside.) Even the way that you represent state and marshal to/from network data may be different, based on requirements.