• Content count

  • Joined

  • Last visited

Community Reputation

852 Good

About CoffeeMug

  • Rank
  1. So who is still around these days?

    I (CoffeeMug/kill - man that was a long time ago) haven't signed in in years, though I've been occasionally typing into the address bar recently. Hi!
  2. I usually waste a lot of my time on things that don't matter and as I'm getting older it's becoming a bigger priority for me to spend my time wisely. This year I resolved to get this part of my life under control. I think the best way to do it is to have a daily routine. When I tried to force a routine on myself, I realized there are three problems I need to solve: - Sometimes I just don't want to do things in the scheduled order - Sometimes other things take priority so I can't stick to schedule - I cannot force myself to have a regular sleep schedule I solved the first two problems in the following way. I divide the day into four hour chunks. The rule is, I can rearrange the tasks within a chunk any way I want, but they must be done within the designated chunk. If something prevents me from doing a task, when I get back on schedule I can drop any of the items within the chunk I'm currently in, but not in any other chunks. This way I have enough flexibility to do the things I want (or can), but enough structure that I don't lose track of things. But I can't solve the third problem. My natural sleep schedule is very irregular. If I go to sleep at a given time, I can't fall asleep, and can't wake up in the morning. If I sleep when I want, my schedule becomes very irregular (I end up going to sleep on different times every day, usually my sleep time shifting later and later into the night until I flip over). Every time I was forced to keep a sleeping schedule (for a full-time job), I have been very unproductive and miserable. This has been the biggest stumbling block to implementing a daily routine. Does anyone have a similar problem? How do you solve it? All the standard remedies (exercise, etc.) don't help me go to sleep at a regular time, and I don't want to force myself to sleep via OTC medication. Perhaps I can work around it? Or force a schedule somehow? How do you do it?
  3. Scheme and Common Lisp

    Quote:Original post by cache_hit Honestly, just use Haskell. Haskell and Common Lisp are totally different beasts, and IMO both are well worth learning. I don't use either in my daily work anymore, but each one has made me a better programmer, in different ways. Haskell forces you to learn what functional programming really is about at its core (the whole lambda calculus and typed lambda calculus business, along with currying, higher order functions, and all that good stuff). It also forces you to learn how to deal with state properly, and how to think of code in terms of data and operations on that data (instead of objects). Common Lisp teaches totally different things. It teaches you how to think of code and data interchangeably, and how to think of and perform compile time programming for really useful things. It teaches you what truly dynamic programs are like (arguably beyond any other language available today). It also teaches you amazing things about software development with the meta-object protocol stuff, and how to think of OOP in terms of multimethods. I don't have specific advice, just meta-advice. It's really not that hard - it's only hard because it's foreign. Keep at it, and after a while it will become second nature. Remember that if something seems "stupid", it's because of the unfamiliarity. Some really smart people have spent many years to make this stuff happen, so don't just brush it off. The time investment is well worth it but ultimately these aren't "holy" topics, it's just knowledge. In retrospect I'm not sure if this is a better investment of time than learning, say, statistics, or *really* understanding calculus (as in reading Newton's and Leibniz's original papers), or working your way through The Road To Reality (I still haven't done these three things, they're on my New Year's resolutions list for the next three years). But hey, you gotta start somewhere...
  4. I'm a bit of a lisper at heart, so I was hoping to use C++ templates to feed my lisp macro frenzy - I wanted the compiler to generate some code for me. I ended up doing something similar, but not exactly what I described here - I still get compile time resolution, but the syntax for using it is a bit ugly. It's not worth going over the solution. I was implementing a small object allocator system, and I wanted a class that would create a pool for each small object type I need (instead of creating the pools manually). So saying something like allocator.malloc<foo_t>() would add the code for a pool of foo_t objects. I didn't want to do it at runtime for performance reasons. Thanks to everyone for your help!
  5. Well, in my current implementation I don't use any RTTI or string comparisons (it's all hashed integers), and that's what gives me poor performance. Could you give an example of what you mean by using macros? I'm not quite sure what you're referring to (I thought of using macros, but couldn't figure out a good way).
  6. I've already done this with a hash table, but unfortunately a runtime lookup decreases performance by a factor of three, and this is a performance critical piece. I could force people to code these things manually, but it would be really really nice to find an automatic solution.
  7. Suppose I have the following piece of code: class foo_t { public: template<typename T> void bar() { // ... } }; What I would like to do is to have every instance of the bar function to have its own member variable within foo_t (or something similar, like an indexed array). It seems that at compile time I can know how many different bar() functions were instantiated, so I should be able to have a member array in foo_t, and for every bar() function I could index into that array. Unforunately I don't know how to accomplish this - I'm new to C++ template trickery. An easy way I can think of is to place a static variable within bar, but I need this to be stored per runtime instance of foo_t. Is there a good way to do this?
  8. Quote:Original post by hplus0603 But for each time the buffer cache misses, you will stall the thread One way around this is to create multiple threads per CPU with decreasing priorities. This way if one thread does stall, the others will take over outstanding requests. This is probably a good practice to use anyway because even with fully asynchronous I/O you might occasionally stall *somewhere* (memory paged out, etc.) IOCP does it automatically, but with Linux it takes a bit of creativity to get the same functionality [smile] Of course complete asynchronous IO is much better than a blocking version for high performance software. I was just pointing out that the situation isn't that bad.
  9. Quote:Original post by samoth I still don't get it why kernel developers seem to think that "asynchronous" and "buffer cache" have to be mutually exclusive. Well, I think the idea might be that if the buffer cache performs well enough (> 90% hit rate), then you have no need for an async call, since the operation is non-blocking anyway. I hear you, though. You can always emulate it with thread pools, but it would be nice if the kernel offered a "real" solution. Quote:I think a block I/O API entirely based on allocating buffers and requesting operations on those buffers would be able to perform very well. To be fair, Linux has sendfile and splice and friends...
  10. Quote:Original post by samoth do you have any *usable* documentation regarding this? Unfortunately there isn't any good documentation. I had to poke around the kernel and libaio source code to figure out how to connect eventfd to aio. It's very simple once you get it working, but the fact that there is no documentation is pretty frustrating. BTW, libaio man pages refer to *kernel* data structures, not libaio data structures. They're very similar but the names aren't correct. Anyway, you create a notification event by calling eventfd(0, 0). After that, when you issue an AIO request (via io_submit), you connect it to the event fd via an undocumented libaio function io_set_eventfd. Once you do that, it's smooth sailing - you can epoll the event fd and when the poll wakes up to handle the event, io_getevents is guranteed to return some events for you. This way you can interleave network events and disk io events within epoll in a single thread. You get a single thread handling both networking and disk IO which hugely simplifies synchronization aspects of your code (you effectively delegate all synchronization to the kernel). You then build a state machine that controls state transitions based on network and IO events and voilà - you get a highly scalable network server [smile]. If you have any specific questions, I'd be happy to answer them here.
  11. Quote:Original post by hplus0603 there is no concept of a callback function. Well, there's io_getevents. Also the AIO mechanism can communicate with the process through an fdevent - this is ideal because you can interleave network IO and disk IO in one thread. There are also real time signals, which are essentially queues similar to Windows message queues. I'd say lack of interfaces isn't the problem - it's lack of a single well defined standard. That's the downside of bazaar development (of course there are upsides as well).
  12. Quote:Original post by samoth Quote:Do you remember if the nr_events parameter to io_setup affect this behavior?No, the only thing that I remember is that it happened even with as few as a dozen or so requests if each request was supposed to read a few dozen megabytes of data. I can only guess why that happened, probably because one big request was broken up into several smaller bits that the hardware can chew, or something, and this was probably limited as well? Actually, I just figured out when the kernel returns with -EAGAIN. The size of the queue is the smaller number between nr_events passed during io_setup and the number stored in /sys/block/sdX/queue/nr_requests. I'm using fairly small blocks now (512B), so the behavior you've observed might still persist. BTW, io_submit is only asynchronous with O_DIRECT, so this might have been the issue. The page cache does a few things in suboptimal ways (well, it's really really good for general purpose stuff, but you can do much better with specialized stuff). MYISAM uses the page cache for example, but it's possible to do much better without it in a specialized situation.
  13. Ok, just did some tests. Despite all the fear mongering out there, I found that io_submit works rather well (for open source software [grin]). It takes about .8 microseconds to submit (as opposed to milliseconds when I run the same timing code on regular preads). If I submit more requests than the kernel can handle, I get -EAGAIN return value without a delay. One open question is how many requests the kernel can handle without returning -EAGAIN. I found that even if I set nevents (when creating the io context) to a large number (larger than the total number of requests), in many cases I get -EAGAIN. This happens largely on slower drives - on a faster drive this is less of a problem. I suppose the kernel doesn't allow the outstanding queue of requests to get too big, no matter what you set nevents to. I'm digging through the kernel aio code now (linux/fs/aio.c if you're interested), but it'll take me a while to ramp up. I'll post here if I find something.
  14. Quote:Original post by samoth Using io_submit, I was unpleasantly surprised to find that once you exceed some threshold (either request-wise or size-wise), your "asynchronous" requests suddenly start taking dozens of milliseconds to submit. That actually doesn't surprise me. If the kernel didn't do this, it would be very easy to execute a DoS attack (just submit spurious IO requests in an infinite loop). Do you remember if the nr_events parameter to io_setup affect this behavior? According to the docs you should be able to submit at least nr_events without blocking. I'm implementing a high performance database, so I can't use the page cache anyway. Since I have to implement my own caching, having to use O_DIRECT to make io_submit work isn't a problem. Having it block before reaching nr_events, however, would be a problem. I'll test this today and post the results.
  15. Well, I'm using "the other one" (io_submit and friends) :) I haven't gotten around to testing its behavior yet, but from what I understand it always works on files opened with O_DIRECT, and works on XFS even without O_DIRECT (this took some digging through the kernel code). Hopefully I'll get to do some benchmarks soon, so I can post the results. Linux support for this kind of stuff is surprisingly bad - AIO is crucial to high performance networking. It might be that the OS supports these types of workloads well, but documentation is terrible, and Google returns obsolete results, with almost no credible benchmarks. Sorry, but ranting helps me blow off some steam :)