I need a FAST way of reading files
Author
Ok, here''s the problem:
We''re coding a game using OpenGL and large amounts of data which have to be loaded on-demand. Currently, we use plan vanilla
fread
for disk reading, reading the whole file in a single call. Is there any way out there to read faster than that? like DMA or something? WE are using C++ with Win32
thanks in advance guys,
Well ...
File mapping might be faster, but I don''t know how one would go about using it.
But if you read the whole file into memory, isn''t your limiting factor going to be memory access time, not disk access (and of course, the structures you use to represent it)
--
File mapping might be faster, but I don''t know how one would go about using it.
But if you read the whole file into memory, isn''t your limiting factor going to be memory access time, not disk access (and of course, the structures you use to represent it)
--
Meet Bunny luv'' and the girls, all ready to perform on your desktop just for you (adult content).
Win32 filemapping doesn''t load all the file in memory always, it''s load in blocks and when those blocks are loaded when you use them... Small files are mostly loaded entirely in memory, but as soon as Windows needs that memory for anything else, it put that back on the disk to be loaded later as soon as is needed again.
For more info in Win32 File handling, look at my filemapping tutorial.
--DK
--H. Hernán Moraldo
http://www.hhm.com.ar/
For more info in Win32 File handling, look at my filemapping tutorial.
--DK
--H. Hernán Moraldo
http://www.hhm.com.ar/
Native OS file I/O methods are slightly faster than fread or cout, which is even slower.
If you want details search for Paul Pedriana''s homepage. He''s got a document called HPCPP (High Performance...)or something close to that I reckon.
"This album was written, recorded and edited at Gröndal, Stockholm in the year of 2000. At this point in time money still ruled the world. Capitalistic thoughts were wide spread. From the sky filled with the fumes of a billionarie''s cigar to the deepest abyss drenched in nuclear waste. A rich kid was a happy kid, oh..dirty, filthy times. Let this be a reminder."
- Fireside, taken from back of the Elite album
If you want details search for Paul Pedriana''s homepage. He''s got a document called HPCPP (High Performance...)or something close to that I reckon.
"This album was written, recorded and edited at Gröndal, Stockholm in the year of 2000. At this point in time money still ruled the world. Capitalistic thoughts were wide spread. From the sky filled with the fumes of a billionarie''s cigar to the deepest abyss drenched in nuclear waste. A rich kid was a happy kid, oh..dirty, filthy times. Let this be a reminder."
- Fireside, taken from back of the Elite album
If you know the order of loading, could you merge all the ''little'' files into a single file which you load in a single read code - something like .WAD files.
As for actually reading the file, fread() is not the quickest way of reading the file, since it has overhead for buffering the input, so you might try _open/_read/_close() commands which go striahgt to DOS. Or, if memory serves, Windows has come file handling functions as part of its API - try those.
But personally I doubt if getting the information off the disk is real the problem - see what you can do the pre-process the files into what ever your internal format is, and manually optimise the loading functions.
As for actually reading the file, fread() is not the quickest way of reading the file, since it has overhead for buffering the input, so you might try _open/_read/_close() commands which go striahgt to DOS. Or, if memory serves, Windows has come file handling functions as part of its API - try those.
But personally I doubt if getting the information off the disk is real the problem - see what you can do the pre-process the files into what ever your internal format is, and manually optimise the loading functions.
A PAK file system or somethign similar might help out allot too. Check out the Quake1 source, or if you want to be cooler use zip files, try zlib, I am using it and it works pretty well.
If you want to maximize the throughput on a windows system, use asycronous IO and completion ports. As soon as you have enough data to process something, process it while you wait for the next chuck of data to come through. This way the cpu is kept busy while waiting for the data to stream in (and it will wait alot).
If you want to have functional progresss bars, cancel buttons etc... you can use a seperate worker thread to do the loading and parsing of the data. That way the application message pump won''t stall during the lengthy read. You can also make a DoEvents type function that takes care of messages in your file loading code and call it once per pass.
Magmai Kai Holmlor
- Not For Rent
If you want to have functional progresss bars, cancel buttons etc... you can use a seperate worker thread to do the loading and parsing of the data. That way the application message pump won''t stall during the lengthy read. You can also make a DoEvents type function that takes care of messages in your file loading code and call it once per pass.
Magmai Kai Holmlor
- Not For Rent
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
