Blobs are not necessarily a good storage system when put into a database; their implementations are often not very well performing, simply because traditional databases aren't designed to have arbitrary sized data. You also end up having to write tools to pack/unpack the blob and in addition you cannot trivially search the blob data for debugging/maintenance purposes.
Rather than worrying about this all now, why don't you just write the server to use the extant arbitrary data storage system? Just dump the data into text files on a filesystem. If you need it backed up, there are tools for that already. If you need it distributing, there are tools for that. ACID is usually pretty easy to implement on the FS (you write to new files and switch the names).
A real-time video delivery environment I worked on did exactly this -- metadata is stored in a replicated database, so it can be queried as normal. The actual blob data is simply a regular linux filesystem. I was quite surprised by this -- I was expecting it to be a custom system. But it turns out that the linux filesystems have had so much optimisation work done on them that they're fast enough. If users request objects that the servers near to them don't have, an internal bittorrent network just transmits the files from datacentre to datacentre. There's a degree of "prepositioning" logic applied on top of this to reduce the "misses" if object popularity can be predicted.
The only "trick" involved is in giving each object in the system a unique name (easy enough), and then partitioning that name up, so that the storage forms a deeper, narrower tree structure. It turns out that the main limitation of filesystems in this use is that searching directories is a linear process and it's better to use several layers of nesting.
This has a number of advantages, particularly if you're storing something as plain text -- your server can now be grepped, or sedded to maintain the data. You can hand edit the files if you need to.
Above all, you've just saved a ton of development time. Put a facade onto this process and later on you can always switch to a DB/blob mechanism or using a data provision service (eg; AppEngine or Amazon Elastic Block Store).
MMO Architecture doubts
1 gateway handling all connections redistributing messages to "sub-servers". Would this solution be "scallable"? I mean you are limited by 1 gateway server.
You're not limited to 1 gateway server, for many reasons:
Currently, I'm running an instance of HAProxy that has on the order of 100,000 simultaneous connections, load-balancing TCP streams. The machine can do a lot more than that, too. Shuffling TCP packets back and forth is pretty simple and well optimized in kernels these days.
You can also use a hardware load balancer, which will do the same thing using dedicated hardware. Those boxes are designed to do many millions of simultaneous connections, spreading them across thousands of back-end machines based on various algorithms (random, sticky, port-based, round-robin, header-based, etc).
Finally, you can do DNS load balancing -- a single name can map to more than one IP address, and each client will essentially pick one random out of the list of addresses (or, more likely, pick the first, and the server returns them in random/rotating order).
Both hardware and software load balancers will also typically do availability checking. When a particular server does not respond, no new sessions are forwarded to that server, so if a single server crashes, it won't leave new connections stranded.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
