Hi,
for my game I have to store game sectors, each sector is an array of 16x16x256 byte.
What I did is make a method to write sectors into a single string inside a file named after it's coordinate,
and one to read them and rebuilt my object in RAM.
But my write time is about 65ms and read time 135ms wich is quite long since a scene is composed of 900 sectors..
Then someone told me to use serializable objects instead,
after reading some documentation about it I don't see how to use this.
If I serialize object A,B,C and D then I can only get them back in the same order,
but I need to acces files according to their coordinate, not their storage order.
is it me or the serializable class is not what I'm looking for?
thanks
java serializable class, really usefull?
First find out what exactly is slow then decide what to do about it.
Also make sure your benchmarking is correct and actually useful.
Also make sure your benchmarking is correct and actually useful.
You could try storing the sectors in a binary format instead of a text format, use one file for all sectors and have the first 16x16x256 bytes be the first sector etc.
Edit oops: java, no istream. , you can use FileInputStreams skip method ( filestream.skip(sectornumber*16*16*256); )
if your sectors are laid out in a 2D grid your sector number can be y*width+x;
This way you get one file (opening files are fairly slow) that you can keep open all the time. (each sector will only require you to do a skip and then a read of 64KiB data (Which really is nothing)).
If that is still too slow you can keep nearby sectors in RAM and load new ones in a background thread before they are needed.
Edit oops: java, no istream. , you can use FileInputStreams skip method ( filestream.skip(sectornumber*16*16*256); )
if your sectors are laid out in a 2D grid your sector number can be y*width+x;
This way you get one file (opening files are fairly slow) that you can keep open all the time. (each sector will only require you to do a skip and then a read of 64KiB data (Which really is nothing)).
If that is still too slow you can keep nearby sectors in RAM and load new ones in a background thread before they are needed.
But my write time is about 65ms and read time 135ms wich is quite long since a scene is composed of 900 sectors..[/quote]
1 sector is 64kB. 900 will be ~56MB. That takes time either way.
To write a sector:byte sector[] = new byte[65536];No magic here.
FileOutputStream fs = new FileOutputStream("sector413.dat");
fs.write(sector); // fs.read(sector); to load
fs.close();
How long this takes depends on disk and file system, especially if there are many files. 64/130 ms sounds a lot, but not impossible.
64k in a single file is very convenient since it doesn't cause overhead, storing the coordinates causes just about maximum overhead possible (99.7%
overhead for the coordinates due to typical 4k disk page size).
Serializable could be used for the above, but doesn't bring much to the table beyond.
One way is to keep index separate as a file which contains only:class Index {File is represented as a number, so '485.dat', '0.dat', ....
float x, y;
int filename;
}
List<Index> indices;
Author
the thing is, the world is generated as the player explore (minecraft clone).
if the player only goes toward the east this one file containing everything will be filled with 75% of 0.
Someone else told me that actually using a database such as MySQL could be usefull.
1 column for the coordinates (primary key), 1 column for the data.
Since I only open a session at the beginning of the game I won't lose time like when I need to acces 900 separated text files.
In addition to that I can even uses indexes for equality seach (on the coordinates) to speed those database access.
....
calculating the coordinate is not time consuming since the player position on the sector grid is always being tracked.
a sector contains more information but I'm only storing the data, since the coordinate are stored inside, storing the file with it's coordinate name is also quick.
I also noticed something weird today, using System.currentTimeMillis()
opening a file = 0ms
writing = 65ms
closing = 0 ms
same thing for reading, shouldn't opening the file be time consuming?
if the player only goes toward the east this one file containing everything will be filled with 75% of 0.
Someone else told me that actually using a database such as MySQL could be usefull.
1 column for the coordinates (primary key), 1 column for the data.
Since I only open a session at the beginning of the game I won't lose time like when I need to acces 900 separated text files.
In addition to that I can even uses indexes for equality seach (on the coordinates) to speed those database access.
....
calculating the coordinate is not time consuming since the player position on the sector grid is always being tracked.
a sector contains more information but I'm only storing the data, since the coordinate are stored inside, storing the file with it's coordinate name is also quick.
public class Sector{
byte[][][] data;
int x,y;
int DL;
//some methods
}
I also noticed something weird today, using System.currentTimeMillis()
opening a file = 0ms
writing = 65ms
closing = 0 ms
same thing for reading, shouldn't opening the file be time consuming?
64k in a single file is very convenient since it doesn't cause overhead, storing the coordinates causes just about maximum overhead possible (99.7%
overhead for the coordinates due to typical 4k disk page size).
You don't need to store the coordinates if you use one file for all sectors as the coordinates can be inferred from the sectors position in the file. using multiple small files will give significantly worse performance than a single larger file would. (more system calls to open files and higher data fragmentation)
I also noticed something weird today, using System.currentTimeMillis()
opening a file = 0ms
writing = 65ms
closing = 0 ms
same thing for reading, shouldn't opening the file be time consuming?
It is possible that the OS does the file open asynchronosly, so the actual open call returns immediatly and the cost of it gets tacked onto your writing/reading if it is done immediatly afterwards.
Try doubling the amount of data you write and see how much the time increases.
In general i would strongly recommend against using multiple files if you got a fixed maximum size of 900 cells (900 cells is only around 60MiB, and storing a single ~60MB file of all zeroes is a fairly low price to pay (you can generate this file on installation or store it compressed in the installer and it won't have any impact on the download size)
I also noticed something weird today, using System.currentTimeMillis()
opening a file = 0ms
writing = 65ms
closing = 0 ms
same thing for reading, shouldn't opening the file be time consuming?
What does this give?
import java.io.FileOutputStream;
import java.util.Random;
public class Test
{
public static void main(String[] args) {
final Random r = new Random();
final String path = "f:\ mp\\";
final int N = 100;
final byte[] chunk = new byte[65536];
long total = 0;
try {
for (int i = 0; i < N; i++) {
final long start = System.nanoTime();
final FileOutputStream fs = new FileOutputStream(path + String.valueOf(r.nextInt()));
fs.write(chunk);
// fs.getFD().sync();
fs.close();
final long end = System.nanoTime();
total += end-start;
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println(1.0 * total / N / 1e6 + "ms");
}
}What if you uncomment the sync() line?
Author
aaah, I'm not using byte, but short, wich double the size.
futhermore, i'm inserting a "," betwen each value, getting even a bigger file.
futhermore, i'm inserting a "," betwen each value, getting even a bigger file.
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]It is possible that the OS does the file open asynchronosly, so the actual open call returns immediatly and the cost of it gets tacked onto your writing/reading if it is done immediatly afterwards.[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]Try doubling the amount of data you write and see how much the time increases.[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)][/quote][/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]Amazing, I would never have though about that, your theory is indeed correct,[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]when doubling the amount of data to write the write time is only increased by 1-2ms.[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]Therefore the file access time must be about 62ms.[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][background=rgb(250, 251, 252)]The reading time is far longer because in addition to opening the file I'm also decoding it, getting each value betwen ",".[/background]
[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=1]...[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=1]trying your code Antheus =D[/font]
aaah, I'm not using byte, but short, wich double the size.
futhermore, i'm inserting a "," betwen each value, getting even a bigger file.
[/background][/font]
[background=rgb(250, 251, 252)]It is possible that the OS does the file open asynchronosly, so the actual open call returns immediatly and the cost of it gets tacked onto your writing/reading if it is done immediatly afterwards.[/background]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]Try doubling the amount of data you write and see how much the time increases.[/background][/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]Amazing, I would never have though about that, your theory is indeed correct,[/background][/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]when doubling the amount of data to write the write time is only increased by 1-2ms.[/background][/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]Therefore the file access time must be about 62ms.[/background][/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]
[background=rgb(250, 251, 252)]The reading time is far longer because in addition to opening the file I'm also decoding it, getting each value betwen ",".[/background][/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=1]...[/font]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=1]trying your code Antheus =D[/font]
[/quote]
if you are storing it as text rather than a binary format a short will take far more than 2 bytes, (the short value 14675 is 5 bytes if stored as text and 2 bytes if stored as a binary short), By storing it as text you're also forced to parse each value (which further slows things down), by using a binary format each value is exactly the same length (a short is always 2 bytes and a byte is always 1 byte) so there is no need to insert ',' to separate individual values.
You could have a file with a grid that tells where to index the chunk file to get the chunk for that grid position.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
