Jump to content
  • Advertisement
Sign in to follow this  
Headkaze

[.net] Dynamic text file loading (fastest)

This topic is 4016 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I have a software project that has a few large text files (~7 MB) that I was loading in using File.ReadAllLines(). Then I search through the file which has entries that I read into a Dictionary. I decided to add dynamic data loading so it only read the entry it needs at the time to reduce memory usage. I'm still using File.ReadAllLines() but this time only storing the entries needed at the time into the Dictionary. The problem is there is still a large amount of memory used which I assume is when I read the entire file using File.ReadAllLines(). I guess the memory will eventually be freed up when GC comes around to it. Maybe I should do a GC.Collect() after doing this to retrive the memory, or is that not recommended? The reason I read all the data at once is because I find it to be alot quicker to scan through a file that way. But it sort of defeats the purpose of saving memory when you still read the whole file in at once. Okay the memory will eventually come back, but I would like to know of any ways to scan a file quickly to extract the info I need without using a large chunck of memory to read the entire file. What would be the fastest way, TextReader.ReadLine()? File streams seem slow to me. Any suggestions?

Share this post


Link to post
Share on other sites
Advertisement
The problem with ReadLine is that it reads a byte (i think) at a time, checking for when it hits 0x13 or 0x10, using a string builder, then returning tostring() when it hits one, you could write your own using ReadBlock using a good size buffer size, then parsing yourself and refilling your buffer when needed.

Share this post


Link to post
Share on other sites
All .NET Stream class derivitates implementas a 1024 bytes buffer by default. You can supply this value during construction but you don't need to create your own class. You can use StreamReader which implements TextReader to get more control if you want to. ~7 MiB of data should be no problem at all.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!