Sign in to follow this  
redmilamber

Efficient HTTP upload?

Recommended Posts

I want to add HTTP-upload to a game server. We have an HTTP server built-in to our game server for testing and for runnning websites directly linked to the game-data (it's handy for a lot of things to have direct access when you want it, without having to setup separate login servers, authorization servers, etc). It's quite a good demo on how to make new servers for arbitrary protocols, too, but it's never been our primary concern to sell websrevers, so a few esoteric features (like HTTP upload) are missing. Here's the problem: I'm doing this because I want to accept uploads of 25Mb or more, frequently (at peak, lots of times every day). These are complete games; the biggest upload I've seen in the past was almost 100Mb, and there've been quite a few over 50Mb before. For obvious reasons, I'd like to do this efficiently! And it has to be using java. We have a method call that lets you transfer a set number of bytes *directly* from the network card to the hard-disk, without going via RAM and CPU. Wonderful, except... AFAICS the designers of RFC 1867 made a huge mistake: they failed to include a content-length descriptor, so there is *no way* you can tell how many bytes are going to be uploaded for a given file without actually reading every single one of them. ?? Running an internet server on a high-bandwidth link, we attract a *lot* of unwelcome attention (e.g. over 100 attacks per week is usual "background noise" even if there's no weblinks to the server itself...once it goes live, we expect a lot more). So, for uploads, we are forced to stream in very small chunks because otherwise it becomes far too easy for some bastard to kill the server by transferring some massive files in parallel and using up all the RAM. If we were able to stream directly to disk, this wouldn't matter, because we would be using 0 RAM. Worse, the whole HTTP server is built on an architecture which assumes you can read the entire body before doing the HTTP parse. I could very easily "divert" a set number of bytes to a different channel (nb: easy re-routing of streams of bytes from one process to another is a feature we have built-in), so if I *knew* how big the files were I could just "divert XX bytes to file-writer YY", but if I have to read them all manually first then I actually need to re-write the whole HTTP parser :( :( :(. So...any thoughts on how to get decent performance, or just how to get away with not rewriting the whole parser?

Share this post


Link to post
Share on other sites
The performance of your upload is very unlikely to be limited on the specific read mechanism. If your netlink is 10 Mbit/s, they'll have a hard time getting through more than 1 MB/s, which even Java reading 10 bytes at a time and writing to disk using fputc() should be able to handle.

Assuming you have disk space, when receiving uploads, I would open a temporary file, read everything from the connection to the file, and when I'm done, I know how big the thing is. Either I'll just read the first 64k for the header, and hand off the file to whomever needs the entity, or if I need all the data in memory, at that point I'll allocate a large enough byte array and read it all back in in one fell swoop.

You probably want to do that read-back in a thread or using asynchronous I/O, though, unless it's OK for your server to block for a few seconds when an upload completes.

Regarding DoS attacks, bandwidth based attacks aren't that common (we see them less than 10% of the time), because bandwidth itself usually costs for the attackers, too. You can somewhat throttle this by only allowing 1/Nth of your available bandwidth to any one class C subnet, and throttle the rate at which you read (which, by TCP windowing, will throttle the rate at which they send). N should be pretty small -- something like 10, unless you have truly impressive inbound pipes, to avoid penalizing regular uploaders. Settings N to 2 is probably sufficient to mitigate the effects of all but the most determined attackers.

Also, I'm not so sure you'll get all that many more attacks when you announce and go live. The internet is LARGE, and even if you buy a Superbowl ad, only some percentage of the surfers out there will even hear about you. The attackers don't read the web (or news) to find targets; they run port scanners. Port scanners don't particularly care whether you have announced your service or not :-) The only exception is if you get slashdotted -- that can shut down a small site quite unexpectedly.

Share this post


Link to post
Share on other sites
Python's http library has the ability to parse chunks of html files, so you might want to look at how they do it.
Quote:

The internet is LARGE, and even if you buy a Superbowl ad, only some percentage of the surfers out there will even hear about you. The attackers don't read the web (or news) to find targets; they run port scanners. Port scanners don't particularly care whether you have announced your service or not :-) The only exception is if you get slashdotted -- that can shut down a small site quite unexpectedly.

Generally, only lower-level crackers scan for sites with vulnerabilities; therefore, as long as your server is all patched up you'll be pretty much immune to these types. Higher level crackers almost always have a target already selected via some other means and don't stop until they've cracked into it; these are the people you should be worrying about.

[Edited by - bytecoder on December 1, 2004 6:17:45 PM]

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Thanks for the thoughts.

Have gone with a chunked stream, with a set of filters that switch each chunk (i.e. one buffer-worth of data) to different destinations (stream to file, stream to other buffer for in-memory manipulation, dump, etc) as if they were packets. I agree now that I was being paranoid about upload performance - this seems to work well, although I'm going to stress test it first and see what happens.

I like the idea of limiting by subnets and using backpressure to slow things down. Do you have any details/pointers to info on what particular network cards actually do? I've enough experience of NIC hardware to know that they often don't do what you'd expect them to :( and have unpleasant "quirks" in how they deal with some situations...

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Oh, sure. (incidentally, I appear to have some 100Mbit PCI cards that do support this (if I read the manuals right) - some ultra high-end Compaq and 3Com stuff designed for servers; they've got other funky features like: multiple ports per card, on-card programmable memory for filtering and stuff, etc. Never used them myself except for compatability testing in ordinary situations...)

What I meant was: If I do this in software, will the standard NIC's bugger it up by being too smart / too stupid? In my experience you don't have to get all that out of the ordinary before your discovering unfixed bugs in your NIC's/drivers >:(.

Share this post


Link to post
Share on other sites
Quote:

will the standard NIC's bugger it up by being too smart / too stupid?


No, not really. If you send a 200 byte TCP packet (with the TCP_NODELAY option set), then the card has two choices:

1) send it as you requested
2) hold it, waiting for more data

Doing 1) is the typical path. Doing 2) would violate what you specifically requested, but, more importantly, would also hurt badly in certain RPC performance benchmarks. Not to mention it would be significantly harder to implement.

Thus, I think you'll be fine. After all, as far as the CARD is concerned, you're just sending a little bit of data every so often -- how can it tell what you're doing higher up in the stack?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this