Jump to content

  • Log In with Google      Sign In   
  • Create Account

#Actualachild

Posted 16 July 2013 - 11:04 PM

If it's going to be entirely random and your computer has less than 20 GB of RAM, this is going to be painful no matter what. Does this at least target SSD drives? Or is it targeting normal hard disk drives? The best flags for performance will be quite different between these two...

 

Also if it is so random, can you do this on multiple threads at once? Or does the algorithm/whatever require serial processing?

 

[edit] Why not just say it all huh?

Okay, so worst case scenario is that you are using a normal hdd. In this case, you have two options.

  1. Use default options, hint that it will use random access. Typically, windows is going to keep caching as more and more pages of memory get read (since you are not releasing anything and have to map it all at once). It is possible the random access flag will help this somewhat, but don't count on much here (I'd love to be wrong). You will typically get decent performance until memory fills up, followed by a very long time of "frozen" computer. Windows is simply paging a bunch of stuff in and out of memory. No need to pull the plug usually. Just go eat some lunch. Possibly take a nap. Seriously.
  2. For consistent time without the crazy hiccups, use the write-through flag. Now you can skip the cache. There may be another flag... perhaps direct buffer when you open the file or something. I can look tomorrow at our code and see. Single threaded for both of these options is the way to go. Perhaps it is best to put all your I/O on its own single thread though...

On the other end of the spectrum would be a computer with 20+ GB RAM, and 2-4 SSDs in a RAID 0 configuration (sure it's slightly risky but not nearly as flimsy as the internet would have you believe)

  1. No need for write-through, it will actually slow you down. 2-4 threads writing is the sweet spot. Test this though. All our tests were for maximum sequential throughput in high speed video acquisition. Your usage is very different. There's really not much more to say here... you'll notice 2 different limits at different stages of testing - 1 your RAM bus speed, and 2 your total SSD(s) speed. Again - writing such small bits randomly will probably prove this entirely wrong. Remember you'll actually be reading/writing chunks based on 1 memory page size and 2 sector size - not much difference between an 8 byte and 4096 byte transaction.

I'm guessing your configuration is with a normal hdd given your frozen computer. I'd say shoot for something in between these 2 situations. With any luck you can at least switch it with a known stable SSD. You'll see a dramatic difference.

 

Is it okay to ask what the application for this is?


#1achild

Posted 16 July 2013 - 10:46 PM

If it's going to be entirely random and your computer has less than 20 GB of RAM, this is going to be painful no matter what. Does this at least target SSD drives? Or is it targeting normal hard disk drives? The best flags for performance will be quite different between these two...

 

Also if it is so random, can you do this on multiple threads at once? Or does the algorithm/whatever require serial processing?


PARTNERS