Sign in to follow this  

fopen+fscanf is there any better way?

This topic is 4020 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

hi i have to load large data files (like code below) and i want to know if this fopen+fscanf method is the only way or if there is any better way (its part of a landscape loader.. normally i got to load 25 time with a code like this below..takes 8-10seconds...i was doing a simple test with an opengl texture loader and then grab data from it.. this method was only a bit over an second..so there must be anything better) simple example what iam using at moment greetings and thx for any hint :)
[source lang="cpp"}

FILE *filenamex,*stream;

...
...
...

	stream =fopen (filename, "r"); 


	if( stream != NULL )
	{
	fseek( stream, 0L, SEEK_SET );

	for (i=0;i<160000;i++)
        {
	fscanf( stream, "%d", &example[i] );
	}
	fclose(stream);



Share this post


Link to post
Share on other sites
Use stream:


#include <ifstream>
#include <ofstream>

std::ifstream & std::ofstream for input and output filestreams

|------------------

#include <iostream>

std::cout and std::cin for character input and output


Google yields more information on all of those.

Hope that helps,

Dave

Share this post


Link to post
Share on other sites
Guest Anonymous Poster

Instead of your 160000 iteration for-loop on fscanf, try this:

fread(example, sizeof(int), 160000, stream);

Share this post


Link to post
Share on other sites
C++ streams usually aren't faster than cstdio.

If you want to do it really fast, the best way probably is memory-mapping the file and using a special purpose parser.

Share this post


Link to post
Share on other sites
And remember, it's more efficient to read binary data than numerical data from a file, so a texture loader will run faster, because it can read in a very large buffer at once, without the need to process and convert data types. You might want to look into writing your data files out to a binary format and then youc an load it much simpler with fread or similar functionality.


FILE *stream;
stream =fopen (filename, "r");
if( stream == NULL )
return;
fseek( stream, 0L, SEEK_SET );
for (i=0;i<160000;i++)
{
fscanf( stream, "%d", &example[i] );
}
fclose(stream);
//Now lets save this to a binary format!
stream = fopen(outfilename, "wb");
if (stream== NULL)
return;
//I'm at work, check if i have these in the correct order!
fwrite(example,sizeof(example_struct),160000,stream);
fclose(stream);
//Now we have a binary representation of our file, which will load MUCH faster!!

/**To Load **/
FILE *stream;
stream =fopen (filename, "r");
if( stream == NULL )
return;
fseek( stream, 0L, SEEK_SET );
fread(example,sizeof(example_struct),160000,stream); //Load it all in one shot!
fclose(stream);

Share this post


Link to post
Share on other sites
Quote:
Original post by Ready4Dis
And remember, it's more efficient to read binary data than numerical data from a file, so a texture loader will run faster, because it can read in a very large buffer at once, without the need to process and convert data types.

Binary files are more compact than text representations, so loading reading them is quicker, yeah. Converting data doesn't need to be a bottleneck, a harddisc tops at 100 MB/s, so you have around 20 cycles per byte to convert your data, that's more than enough for most reasonable ways of encoding data.

Bulk reading is quicker in most circumstances, so you should try it even if you still need to convert the data you read.

Share this post


Link to post
Share on other sites
wow

thats perfect

i got much much more help here that i ever think to get


the "loader" is a part of an landscape terrain loader
including height(1xchar) and color data(1xchar)
(i put that into an landscape texture,only 256different colors the
terrain engine calculates the landscape soft)
my landscape got size 2000x2000 and its divided
in 25 pieces (each file holds 400x400) .. every time the position gets over
middel+-400 the offset moves and new landscape parts
are reloaded so i got a seamless landscape running
(walking or with horse it takes a little time to reach this border..
but when flying a dragon the data reloader is often needed )

much thx to all here

Share this post


Link to post
Share on other sites

This topic is 4020 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this