Sign in to follow this  

[java] File i/o speed

This topic is 2843 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Trying to work out if calculating 3D data from a 176 byte file is faster than loading the data from a 1320 byte file. I've been doing some testing on the time it takes for my computer to read bytes from a file. I'm ignoring the file seek time and any calls to java.io because a file has to be read anyway. On average it takes 4.2ns to read each byte using datainputstream.read(abyte0, 0, i); 4.2ns per byte!!! Does that sound right? That is almost the same as my clock cycle so it must be fetching it from cache. Is there a better way to test this other than changing the file each time? Anyone know their byte read speed? I've read it's 40-50 clock cycles but need to know for sure.

Share this post


Link to post
Share on other sites
The loading times for 176 and 1320 byte files probably won't be much different, as the OS will most likely load complete sectors or pages (at least 4KiB) anyways and cache it.

File I/O is a very complicated process, so measuring it accurately, even for 1 machine will be difficult. I recommend duplicating your test files many (100+) times, loading those files and timing the whole process. The result will be more accurate.

Also, try running the test right after a reboot.

I assume the smaller file contains the same data as the bigger one, but is compressed. You can gain file read speed from compressing files, as long as decompressing is inverse-compression-rate faster than reading from disk.

If you have a *lot* of small files, it's beneficial to combine them to a larger file. Make sure they're in the order they're going to be read and you should see a significant speed up.

There's tons more to say about this topic, just ask!

Share this post


Link to post
Share on other sites
Quote:
Original post by SNARG
I've been doing some testing on the time it takes for my computer to read bytes from a file. I'm ignoring the file seek time and any calls to java.io because a file has to be read anyway.
On average it takes 4.2ns to read each byte using datainputstream.read(abyte0, 0, i);
4.2ns per byte!!! Does that sound right?

Definitely not. Like Rattenhirn said, your OS is caching the file. To compare the amount of time it takes to load your data, you have to compare the entirety of both operations from start to finish.

Also, there's more than one way to be fast. If your program is to be downloaded, it would save on server costs - and in the case of an applet, initialization time - to use the smaller versions of your files (and package them in a JAR).

Share this post


Link to post
Share on other sites
Thanks,
thought the time per bytes looked fishy.
Different tests need doing, if the 40 clock cycles is correct it doesn't make sense to load the extra data, but then if you combine the files into a single package the caching of that package, by the OS, could negate the extra load time.

The small file is a descriptor used to rebuild the 3D model, the larger file contains the 3D data as is, stored as signed SHORT. The 176 bytes compressed to 203 bytes, the larger one to 928.

With the descriptor you get a built in skeleton at 1/5th the size of a standard 3D file so thinking in bandwidth terms you could either save a packet or have a much larger 3D world to play in.
Could always use the descriptor to rebuild the model at the other end as part of the update.
Wanted to know if it's worth rebuilding on the fly or use the processor time for other things.

Cheers for that, the things you forget to think about when worrying about problems that don't exist.

Share this post


Link to post
Share on other sites
Quote:
I'm ignoring the file seek time and any calls to java.io because a file has to be read anyway.

This means you ignore the cost of 1,000,000 units and worry about the cost of 2 units.

The cost of opening a file will be several orders of magnitude higher than cost of reading 64kb block.

If you want an improvement, pack everything into a single file. Reading 1000 pieces from same file will be hundreds of times faster than reading single piece from 1000 files. Uncompressed zip is good enough for this purpose, and Java supports it out of box.

Quote:
The small file is a descriptor used to rebuild the 3D model, the larger file contains the 3D data as is, stored as signed SHORT. The 176 bytes compressed to 203 bytes, the larger one to 928.

In Java, this is pointless to optimize. In language with direct memory access, you could lay out the structure to fit memory model, then just memcpy that.

Shorts are irrelevant, there is no reason to use anything but ints, or bytes if you need byte-wise access.

Single bytes are irrelevant - RAM has throughput measured in gigabytes per second.

Quote:
thinking in bandwidth terms you could either save a packet or have a much larger 3D world to play in.


To save bandwidth, send less data. Optimizing existing structures is microoptimization. HTTP is used for most traffic these days anyway.

Quote:
Wanted to know if it's worth rebuilding on the fly or use the processor time for other things.

No, it's not worth rebuilding. There exist many strategies to manage caches of loaded data.

Share this post


Link to post
Share on other sites

This topic is 2843 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this