Quote:Original post by samoth

Quote:Do you remember if the nr_events parameter to io_setup affect this behavior?

No, the only thing that I remember is that it happened even with as few as a dozen or so requests if each request was supposed to read a few dozen megabytes of data. I can only guess why that happened, probably because one big request was broken up into several smaller bits that the hardware can chew, or something, and this was probably limited as well?

Actually, I just figured out when the kernel returns with -EAGAIN. The size of the queue is the smaller number between nr_events passed during io_setup and the number stored in /sys/block/sdX/queue/nr_requests.

I'm using fairly small blocks now (512B), so the behavior you've observed might still persist. BTW, io_submit is only asynchronous with O_DIRECT, so this might have been the issue.

The page cache does a few things in suboptimal ways (well, it's really really good for general purpose stuff, but you can do much better with specialized stuff). MYISAM uses the page cache for example, but it's possible to do much better without it in a specialized situation.

One of the core problems in Linux async I/O is that the syscall model was initially synchronous. open/close/ioctl/read/write are all expected to return when done. The only way for the kernel to send asynchronous information back to the application is through a signal -- there is no concept of a callback function.

The real way to implement async I/O in a kernel that will give you the best peformance, is to make the kernel interface be inherently asynchronous, and to have a kernel-defined message queue per process (or per thread, or per user-defined token, or whatever). The higher-level blocking calls would then just be built as a layer on top of the async stuff.

This is similar to, for I/O, you really want to define a block-level interface, and a buffered character-level interface can be built on top. At least that part most implementations get it mostly right :-)

Unfortunately, so much UNIX software is so rooted in the synchronous model, that it's really quite hard to attempt to move the entire stack (applications, C library, kernel, drivers) to an asynchronous model.

On Windows, the window message loop and I/O completion ports and APCs all serve as well-defined system-to-application messaging models. In fact, one of Windows problems is that it has too many return mechanisms :-) Thus, writing asynchronous code on Windows (at any layer of the stack) is generally more straightforward than UNIX. It's by no means ideal, though.

Another problem is the way that a GUI is a bolt-on on top of UNIX. While that's great for running head-less servers, it's a real problem when it comes to interaction between the GUI system and the kernel.

Quote:Original post by hplus0603
there is no concept of a callback function.

Well, there's io_getevents. Also the AIO mechanism can communicate with the process through an fdevent - this is ideal because you can interleave network IO and disk IO in one thread. There are also real time signals, which are essentially queues similar to Windows message queues. I'd say lack of interfaces isn't the problem - it's lack of a single well defined standard. That's the downside of bazaar development (of course there are upsides as well).

Quote:Original post by Antheus
Is Grand Central Dispatch for user mode tasks only, or does it apply to syscalls as well? It claims to work with file descriptors, but is the asynchronous processing emulated or actually implemented in kernel?

There certainly is kernel support involved, because when Apple released Grand Central as open-source, they mentioned that it can run *without* kernel support.

If you want to look deeper, the source to grand central itself is available here, the source for the blocks runtime it depends upon is here, and you can pull the kernel support from the XNU project...

Quote:there's io_getevents

As others have said, there are a number of possible mechanisms. That's not the point I was trying to make, though.

Here's the point: None of them is *the model* that most software was written to expect. UNIX has run for 40 years on the blocking-I/O, read()/write() model. It's pervasive throughout the ecosystem. The closest you can get to that in the form of less simplistic I/O is select(), which isn't really helpful for asynchronous operation, only for non-blocking operation (and it's not even non-blocking in the sense of non-stalling on disk I/O).

Changing the UNIX ecosystem is hard, and takes time. In fact, very hard, and takes lots of time (as in "decades").