For efficient writes, here's an alternate way:
final short[] chunk = new short[65536];
final RandomAccessFile fs = new RandomAccessFile(path + String.valueOf(Math.abs(r.nextInt())), "rw");
final ByteBuffer bb = fs.getChannel().map(MapMode.READ_WRITE, 0, chunk.length * 2);
for (int i = 0; i < chunk.length; i++) bb.putShort(chunk);
fs.getChannel().force(true);
fs.getFD().sync();
fs.close();
For, this completes in 15ms, 1.4ms without sync vs 93ms if writing as text.

Above is also about as fast as it gets.

mmh, I'm not getting result as good as you Antheus with you code (slightly modified for 3D arrays) :



public void storeToDisk(String path) throws IOException{
long time = System.currentTimeMillis();
final RandomAccessFile fs = new RandomAccessFile(path +x+","+z+".txt", "rw");
final ByteBuffer bb = fs.getChannel().map(MapMode.READ_WRITE, 0, (Global.CHUNK_HEIGHT*Global.CHUNK_X*Global.CHUNK_Z)* 2);


for(short yy=0; yy<Global.CHUNK_HEIGHT; yy++){
for(short zz=0; zz<Global.CHUNK_Z; zz++){
for(short xx=0; xx<Global.CHUNK_X; xx++){
bb.putShort(sect[xx][yy][zz]);
}
}
}
fs.getChannel().force(true);
fs.getFD().sync();
fs.close();
System.out.println("write time : " + (System.currentTimeMillis() - time)+"ms");
Global.totalTime += (System.currentTimeMillis() - time);
}


//with synch

19ms, 16ms, 14ms, 57ms, 77ms, 21ms, 56ms, 15ms, 9ms, 74ms, 10ms, 9ms, 74ms, 18ms, 22ms, 66ms, 31ms, 50ms, 18ms, 49ms, 72ms, 18ms, 16ms, 68ms, 17ms, 23ms, 66ms, 17ms, 23ms, 72ms, 17ms, 17ms, 75ms, 17ms, 17ms, 66ms, total write time : 1308ms

//without synch
24ms, 32ms, 15ms, 57ms, 16ms, 22ms, 56ms, 28ms, 14ms, 75ms, 17ms, 17ms, 48ms, 26ms, 15ms, 66ms, 14ms, 60ms, 15ms, 17ms, 62ms, 28ms, 17ms, 58ms, 16ms, 14ms, 52ms, 29ms, 17ms, 78ms, 14ms, 17ms, 61ms, 15ms, 25ms, 68ms, total write time : 1208ms
[/quote]



Also, yesterday I changed my chunk2text methods, adding some bufferedWriter, it basically takes the same time


public void storeToDisk(String path) throws IOException{
long time = System.currentTimeMillis();

File file = new File(path+x+","+z+".txt");
FileWriter fw = new FileWriter(file);
BufferedWriter out = new BufferedWriter(fw);
for(short yy=0; yy<Global.CHUNK_HEIGHT; yy++){
for(short zz=0; zz<Global.CHUNK_Z; zz++){
String s = "";
for(short xx=0; xx<Global.CHUNK_X; xx++){
s += String.valueOf(sect[xx][yy][zz]);
s += ",";
}
out.write(s);
}
}
out.close();
System.out.println("write time : " + (System.currentTimeMillis() - time));
Global.totalTime += (System.currentTimeMillis() - time);
}


38ms, 33ms, 29ms, 30ms, 27ms, 28ms, 33ms, 31ms, 28ms, 28ms, 29ms, 33ms, 28ms, 28ms, 31ms, 28ms, 29ms, 32ms, 28ms, 30ms, 31ms, 30ms, 33ms, 30ms, 30ms, 33ms, 29ms, 29ms, 30ms, 28ms, 34ms, 30ms, 32ms, 58ms, 33ms, 31ms, total write time : 1124ms
[/quote]

mmh, I'm not getting result as good as you Antheus :

You are.

Explicit sync makes sure that each chunk is really absolutely positively written to disk. So it's the slowest possible case.


You'll notice two cases, one set of times is around 17ms, which is very close to what I get. The other is in 50ms range.

9-17ms is fairly easy to explain. Seek time of disk (around 8-10ms) plus the write.
50ms happens when OS need to force flush and wait, maybe it has something else going on, maybe another process is doing disk IO, so it takes longer.


Improvements from deferred writes vary. On laptop it might be disabled altogether for increased reliability (in case battery goes out) or the OS/disk cache might be full or too small or too slow. Deferred writes may improve things, but they aren't magic, they merely let your thread run ahead while OS does the work in the back. If that isn't possible, it won't be any faster.

Deferred writes also do not magically increase throughput. If enough data is written, times will settle at limits of disk IO, since OS cannot afford to buffer too much data it claimed to have written to disk. I could save 500MB then turn off the machine, thinking it's safe, while the OS would still need 2 minutes to flush everything from memory.


For the second case, you're using text serialization. You write roughly 2.5 times as much data. First example uses exactly 2 bytes per value. Second example uses 4-5. Timing is consistent with that.

The numbers above give hard limits on how long the disk IO takes.

damn, looks like I'll need to do as mojang did and implement a "region" system.

anyway serializable is not bringing anything usefull to the table, good to know.