Jump to content
  • Advertisement
Sign in to follow this  
Lutz

Async IO and file size

This topic is 2377 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey,

I'm using OVERLAPPED IO for asynchronous file IO (Win32, C++) to read files to a buffer in one go and that part works fine. The problem I'm having is to know how big the buffer has to be, i.e. I need to know the file size. I'm using GetFileAttributesExA right now because it's my understanding that this is the fastest way to do it, but it's still a blocking command. Especially when loading data from a network drive, that's a total killer as it can block for a long time for remote access.

Is there any way to get the file size asynchronously?

Thanks,

- Lutz

Share this post


Link to post
Share on other sites
Advertisement
yes put file size in the header of the file. so that first bits you get from the file tell you some infos bout the file. Else you have to stream with unkown size.

Share this post


Link to post
Share on other sites
You could try GetFileSize, but it's probably just the same... You could use a background thread to make these blocking calls, but that's not ideal...

I actually use one file as an 'index' for all my other files - I keep this 'index' loaded at all times, and can read file-sizes out of it myself.

Share this post


Link to post
Share on other sites
I second Hodgman's approach. This is also useful if you happen to use a pack-file type structure where all your data is in one physical file; you can then just memory-map chunks of it as necessary to get the data you need.

Share this post


Link to post
Share on other sites
A colleague of mine actually suggested the same thing, use a pak file. I think that's the way to go. Thanks, guys!

Share this post


Link to post
Share on other sites
Also wanted to mention, I used GetFileSize before, but that seems to be equally slow if not slower (some people say it's slower, I didn't get a big difference). My thread that calls [color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

[background=rgb(250, 251, 252)]GetFileAttributesExA is already a background thread, so it's not stalling the game or anything, but it stalls the whole async pipeline. I could put [/background]

[/font][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

[background=rgb(250, 251, 252)]GetFileAttributesExA into its own thread, that's an option, but it seems like using a pak file is a much more elegant solution.[/background]

[/font]

Share this post


Link to post
Share on other sites
A pack file also lets you store your own metadata for all the files held within it, and read that once on startup. It also lets you do other nice things like ordering the files to minimize seeking overhead. That just gives you one or two potentially blocking file read calls on startup.

However, I'd also go with making a dedicated thread to handle all file reading. That lets you do other things like transparently decompress data from the file, and prioritize time sensitive file reads. Also doing multiple simultaneous reads from different threads will hit performance as it can cause extra seeks.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!