Sign in to follow this  

fread/fwrite (a deeper question)

This topic is 4839 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I know that fread/fwrite buffer file data. For example when you fread a byte of data, behind the scene a chunk of the file has actually been read in, a lot more than just that single byte you need. I know this is done to minimize context switching, but how much actually gets buffered in?? Also- do fseek and ftell require a context switch? If anyone can point me to some resouces that would also be spot on. Thanks guys! =)

Share this post


Link to post
Share on other sites
This is all implementation dependant to the extreme.

and quite honestly, not worth knowing.

If you're really curious, you could probably check the sources to something like glibc to find an example of one particular libc implementation.

Share this post


Link to post
Share on other sites
Thanks for the reply! Yeah, I do see your point =)

However, I'm making this for a specialized multithreaded webserver to run off my debian linux server. The current server does 4 million hits each day. I'm writing my own webserver as a research project, but mostly for kicks!

So here's the whole shpeal while still sparing most of the details in the interest of keeping the size of this post realtively short-

It tracks hits and statistical stuff like that. I want to be able to process the stats in batches so I want to buffer the data into the most efficient sized byte array, and then write it to a file in a single call. I have 3 hard drives in RAID0. Making a bunch of small reads to a file makes RAID0 less efficient. I'd make a huge char array and write that (which would be very efficient under RAID0) except that the data needs to be split into multiple files (a few thousand or so).


That's why I'm looking into this even though as I agree in most cases it would not be worth knowing =)

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by OuncleJulien
I know that fread/fwrite buffer file data. For example when you fread a byte of data, behind the scene a chunk of the file has actually been read in, a lot more than just that single byte you need. I know this is done to minimize context switching, but how much actually gets buffered in??

Also- do fseek and ftell require a context switch?

If anyone can point me to some resouces that would also be spot on.

Thanks guys! =)




Look for a constant you can define (or a compiler definition)
to set the size of the buffer.

If its appropriate to your APP, a larger buffer can speed things up by requiring fewer system calls (and process switches)

Share this post


Link to post
Share on other sites
Thanks for the replies everyone!

I just got home and am scouring over glibc and doing some more searching online. I'll post here if I'm able to turn anything up.

C-Junkie- you are right. Sorry about not being clear about that. I will be doing both reading and writing, though since I'll be reading in large batches, RAID0 will handle that nicely. I'm mostly interested in writing.

If anyone has anything else to add please do! =)

Share this post


Link to post
Share on other sites
Keep in mind that when you write to a disk, the OS doesn't actually WRITE it to disk instantly unless it's a removable device (like a Thumbdrive). Access to hard drives is put through a write cache in order to make disk access more efficient from multiple processes.

Share this post


Link to post
Share on other sites
Actually file buffering is pretty standard across OSes. And it has to be since it's controlled by the ANSI C library (a little known fact for non-unix programmers). Google for setvbuf and you'll get some interesting details.

Most OSes does their own buffering too and if you need better control you should probably look into the win32 api's file functions (eg. CreateFile).

Share this post


Link to post
Share on other sites
By the way you're talking about it, I don't believe that you know what a context switch actually is. Regardless, C-Junkie is right about it probably not being worth knowing.

Share this post


Link to post
Share on other sites

This topic is 4839 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this