Archived

This topic is now archived and is closed to further replies.

file io functionality (...speed)

This topic is 5152 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey guys, gals, etc. My Problem: - I am using the C Low Level IO routines . These functions invoke the operating system directly for lower-level operation than that provided by stream I/O. Low-level input and output calls do not buffer or format data. - I have to parse text files of undetermined size. - I have both singleton struct and virtual class implementations (both are equally excruciatingly slow ). The Fact: - It more than likely that the speed is because of poor search routines on my part, but.... My Question: - Can anyone recommend some fast text searching algorithms or web sites devoted to such as I''ve found none. - Also, are the cpp iostream classes noticably faster? What about FILE *p''s? -Any help would be appreciated!

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Hello,

My suggestion would be to load the whole file into memory then parse it from there. This is what I have done. You might want to use linked lists for this. I heard std::list is good although I have never made a jump to it since I wrote my own linked list implementation a couple of years before STL was the standard.

Hope that helps

Share this post


Link to post
Share on other sites
The fastest way is indeed to first fully load the file into memory, no matter how large it is, and then use that memory. Iostream etc will be slower, it introduces overhead, so if you really need speed, use a FILE and read it in fully in one swift fread

Share this post


Link to post
Share on other sites
quote:
Original post by m_e_my_self
The fastest way is indeed to first fully load the file into memory, no matter how large it is, and then use that memory. Iostream etc will be slower, it introduces overhead, so if you really need speed, use a FILE and read it in fully in one swift fread


iostream use at least the same low-level routines I presume...probably you forget read() method in ifstream
std and stl are better !!!

If you want to process BIG text file I suggest you to use ''ropes'' or list of string.


list<string> text_file_lines;


Then you can read lines and put them in your list.

If you want to load at max speed you can use a strstream and load it with your ifstream or create your own buffer.
But you need probably to split your file in lines to process them efficiently with std::string methods.
But I dont know what you want to do...

PS: I wrote tons of template list,array,string...when I learned STL I simply forget about them..and FILE* too !!!

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
" My suggestion would be to load the whole
file into memory then parse it from there"

Well, not quite but close. Use memory mapping instead of loading the file into virtual space (otherwise you run the risk of having it getting swapped out to disk again if it is too big. DOH!).

Share this post


Link to post
Share on other sites