fread/fwrite (a deeper question)
Author
I know that fread/fwrite buffer file data. For example when you fread a byte of data, behind the scene a chunk of the file has actually been read in, a lot more than just that single byte you need. I know this is done to minimize context switching, but how much actually gets buffered in??
Also- do fseek and ftell require a context switch?
If anyone can point me to some resouces that would also be spot on.
Thanks guys! =)
This is all implementation dependant to the extreme.
and quite honestly, not worth knowing.
If you're really curious, you could probably check the sources to something like glibc to find an example of one particular libc implementation.
and quite honestly, not worth knowing.
If you're really curious, you could probably check the sources to something like glibc to find an example of one particular libc implementation.
Author
Thanks for the reply! Yeah, I do see your point =)
However, I'm making this for a specialized multithreaded webserver to run off my debian linux server. The current server does 4 million hits each day. I'm writing my own webserver as a research project, but mostly for kicks!
So here's the whole shpeal while still sparing most of the details in the interest of keeping the size of this post realtively short-
It tracks hits and statistical stuff like that. I want to be able to process the stats in batches so I want to buffer the data into the most efficient sized byte array, and then write it to a file in a single call. I have 3 hard drives in RAID0. Making a bunch of small reads to a file makes RAID0 less efficient. I'd make a huge char array and write that (which would be very efficient under RAID0) except that the data needs to be split into multiple files (a few thousand or so).
That's why I'm looking into this even though as I agree in most cases it would not be worth knowing =)
However, I'm making this for a specialized multithreaded webserver to run off my debian linux server. The current server does 4 million hits each day. I'm writing my own webserver as a research project, but mostly for kicks!
So here's the whole shpeal while still sparing most of the details in the interest of keeping the size of this post realtively short-
It tracks hits and statistical stuff like that. I want to be able to process the stats in batches so I want to buffer the data into the most efficient sized byte array, and then write it to a file in a single call. I have 3 hard drives in RAID0. Making a bunch of small reads to a file makes RAID0 less efficient. I'd make a huge char array and write that (which would be very efficient under RAID0) except that the data needs to be split into multiple files (a few thousand or so).
That's why I'm looking into this even though as I agree in most cases it would not be worth knowing =)
Be a bit more specific, at one point you mention reads, at anotehr oyu metion writes. are you doing both?
Quote:Original post by OuncleJulien
I know that fread/fwrite buffer file data. For example when you fread a byte of data, behind the scene a chunk of the file has actually been read in, a lot more than just that single byte you need. I know this is done to minimize context switching, but how much actually gets buffered in??
Also- do fseek and ftell require a context switch?
If anyone can point me to some resouces that would also be spot on.
Thanks guys! =)
Look for a constant you can define (or a compiler definition)
to set the size of the buffer.
If its appropriate to your APP, a larger buffer can speed things up by requiring fewer system calls (and process switches)
Author
Thanks for the replies everyone!
I just got home and am scouring over glibc and doing some more searching online. I'll post here if I'm able to turn anything up.
C-Junkie- you are right. Sorry about not being clear about that. I will be doing both reading and writing, though since I'll be reading in large batches, RAID0 will handle that nicely. I'm mostly interested in writing.
If anyone has anything else to add please do! =)
I just got home and am scouring over glibc and doing some more searching online. I'll post here if I'm able to turn anything up.
C-Junkie- you are right. Sorry about not being clear about that. I will be doing both reading and writing, though since I'll be reading in large batches, RAID0 will handle that nicely. I'm mostly interested in writing.
If anyone has anything else to add please do! =)
Keep in mind that when you write to a disk, the OS doesn't actually WRITE it to disk instantly unless it's a removable device (like a Thumbdrive). Access to hard drives is put through a write cache in order to make disk access more efficient from multiple processes.
Actually file buffering is pretty standard across OSes. And it has to be since it's controlled by the ANSI C library (a little known fact for non-unix programmers). Google for setvbuf and you'll get some interesting details.
Most OSes does their own buffering too and if you need better control you should probably look into the win32 api's file functions (eg. CreateFile).
Most OSes does their own buffering too and if you need better control you should probably look into the win32 api's file functions (eg. CreateFile).
By the way you're talking about it, I don't believe that you know what a context switch actually is. Regardless, C-Junkie is right about it probably not being worth knowing.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
