I want to add HTTP-upload to a game server. We have an HTTP server built-in to our game server for testing and for runnning websites directly linked to the game-data (it's handy for a lot of things to have direct access when you want it, without having to setup separate login servers, authorization servers, etc). It's quite a good demo on how to make new servers for arbitrary protocols, too, but it's never been our primary concern to sell websrevers, so a few esoteric features (like HTTP upload) are missing. Here's the problem: I'm doing this because I want to accept uploads of 25Mb or more, frequently (at peak, lots of times every day). These are complete games; the biggest upload I've seen in the past was almost 100Mb, and there've been quite a few over 50Mb before. For obvious reasons, I'd like to do this efficiently! And it has to be using java. We have a method call that lets you transfer a set number of bytes *directly* from the network card to the hard-disk, without going via RAM and CPU. Wonderful, except... AFAICS the designers of RFC 1867 made a huge mistake: they failed to include a content-length descriptor, so there is *no way* you can tell how many bytes are going to be uploaded for a given file without actually reading every single one of them. ?? Running an internet server on a high-bandwidth link, we attract a *lot* of unwelcome attention (e.g. over 100 attacks per week is usual "background noise" even if there's no weblinks to the server itself...once it goes live, we expect a lot more). So, for uploads, we are forced to stream in very small chunks because otherwise it becomes far too easy for some bastard to kill the server by transferring some massive files in parallel and using up all the RAM. If we were able to stream directly to disk, this wouldn't matter, because we would be using 0 RAM. Worse, the whole HTTP server is built on an architecture which assumes you can read the entire body before doing the HTTP parse. I could very easily "divert" a set number of bytes to a different channel (nb: easy re-routing of streams of bytes from one process to another is a feature we have built-in), so if I *knew* how big the files were I could just "divert XX bytes to file-writer YY", but if I have to read them all manually first then I actually need to re-write the whole HTTP parser :( :( :(. So...any thoughts on how to get decent performance, or just how to get away with not rewriting the whole parser?

Python's http library has the ability to parse chunks of html files, so you might want to look at how they do it.

Quote:
The internet is LARGE, and even if you buy a Superbowl ad, only some percentage of the surfers out there will even hear about you. The attackers don't read the web (or news) to find targets; they run port scanners. Port scanners don't particularly care whether you have announced your service or not :-) The only exception is if you get slashdotted -- that can shut down a small site quite unexpectedly.

Generally, only lower-level crackers scan for sites with vulnerabilities; therefore, as long as your server is all patched up you'll be pretty much immune to these types. Higher level crackers almost always have a target already selected via some other means and don't stop until they've cracked into it; these are the people you should be worrying about.

[Edited by - bytecoder on December 1, 2004 6:17:45 PM]

Thanks for the thoughts.

Have gone with a chunked stream, with a set of filters that switch each chunk (i.e. one buffer-worth of data) to different destinations (stream to file, stream to other buffer for in-memory manipulation, dump, etc) as if they were packets. I agree now that I was being paranoid about upload performance - this seems to work well, although I'm going to stress test it first and see what happens.

I like the idea of limiting by subnets and using backpressure to slow things down. Do you have any details/pointers to info on what particular network cards actually do? I've enough experience of NIC hardware to know that they often don't do what you'd expect them to :( and have unpleasant "quirks" in how they deal with some situations...

Oh, sure. (incidentally, I appear to have some 100Mbit PCI cards that do support this (if I read the manuals right) - some ultra high-end Compaq and 3Com stuff designed for servers; they've got other funky features like: multiple ports per card, on-card programmable memory for filtering and stuff, etc. Never used them myself except for compatability testing in ordinary situations...)

What I meant was: If I do this in software, will the standard NIC's bugger it up by being too smart / too stupid? In my experience you don't have to get all that out of the ordinary before your discovering unfixed bugs in your NIC's/drivers >:(.