[java] File-Format Design
Author
For my game, I require the storage of all types of media: text, images, sound, video, and most of the other types all games want to store.
I will store everything in one (or maybe more) file(s) of bytes.
Currently, I have it organised in a hierarchical (sp?) system. Each Node either contains another node(s), or it contains data.
Each block contains a header of 8 bytes: 4 for an identifier, 1 for length, and 3 blank (may be used in future).
My main question is: can I store the length effectively in bytes. What I am doing, is simply casting the number from int to byte, i.e.:
int i = 1000;
byte b = (byte) u;
What upper number can byte data go to? What will happen if it goes beyond this?
Do you mean what is the size of a byte and what happens when you cast a value larger than it?
If that is the case a byte is 8bit thus can store 2^8 combinations (0 to 255, or -128 to 127 depending on use of the sign bit).
Most implementation will simply modulo a larger value.
Say you have the value 1000d.
That is the same as 0x3E8 in hex.
Now to modulo that by 255 (or 0xFF) you just take the last 2 numbers
So it becomes E8 which is the same # as 1000d % 255.
Now a byte can only store 2 hex values.
So that if you try:
byte data = value;
if becomes the modulo of value by size of byte.
Now the problem with that is it loses data. To effectivley convert an int to bytes you must use 4 bytes since:
1 byte = 8 bits
1 int = 32 bits
thus 1 int = 4 bytes.
now to cut up an int is actually really easy.
you can think of modulo and divide as mask and shift
0x1234 / FF = 0x12
0x1234 % FF = 0x34
(0x123456 / FF) % FF = 0x34 since if shits right and then masks off all but the last two numbers.
So to cut up an int you just:
int value;
byte1 = value / 0xFFFFFF; (take it and shift right 24 bits) so use only the far left 8 bits BYTExxxxxxxxxxxx.
byte2 = (value / 0xFFFF) % 0xFF; (take it and shift right 16 bits and mask ) so use use xxxxBYTExxxxxxxx.
byte3 = (value / 0xFF) % 0xFF; (take it and shift right 8 bits and mask) so use use xxxxxxxxBYTExxxx.
byte4 = (value % 0xFF) (maske out 8 bits) so use use xxxxxxxxxxxxBYTE.
To recombine it''s not that hard:
value = (byte1 * 0x1000000) + (byte2 * 0x10000) + (byte3 * 0x100) + byte4;
I hope that helps.
But generally you don''t need to do this since if you used fixed sized data labels, you can then load the data in correct sized variables.
If that is the case a byte is 8bit thus can store 2^8 combinations (0 to 255, or -128 to 127 depending on use of the sign bit).
Most implementation will simply modulo a larger value.
Say you have the value 1000d.
That is the same as 0x3E8 in hex.
Now to modulo that by 255 (or 0xFF) you just take the last 2 numbers
So it becomes E8 which is the same # as 1000d % 255.
Now a byte can only store 2 hex values.
So that if you try:
byte data = value;
if becomes the modulo of value by size of byte.
Now the problem with that is it loses data. To effectivley convert an int to bytes you must use 4 bytes since:
1 byte = 8 bits
1 int = 32 bits
thus 1 int = 4 bytes.
now to cut up an int is actually really easy.
you can think of modulo and divide as mask and shift
0x1234 / FF = 0x12
0x1234 % FF = 0x34
(0x123456 / FF) % FF = 0x34 since if shits right and then masks off all but the last two numbers.
So to cut up an int you just:
int value;
byte1 = value / 0xFFFFFF; (take it and shift right 24 bits) so use only the far left 8 bits BYTExxxxxxxxxxxx.
byte2 = (value / 0xFFFF) % 0xFF; (take it and shift right 16 bits and mask ) so use use xxxxBYTExxxxxxxx.
byte3 = (value / 0xFF) % 0xFF; (take it and shift right 8 bits and mask) so use use xxxxxxxxBYTExxxx.
byte4 = (value % 0xFF) (maske out 8 bits) so use use xxxxxxxxxxxxBYTE.
To recombine it''s not that hard:
value = (byte1 * 0x1000000) + (byte2 * 0x10000) + (byte3 * 0x100) + byte4;
I hope that helps.
But generally you don''t need to do this since if you used fixed sized data labels, you can then load the data in correct sized variables.
Why not build your file format as a custom class that can be serialized.
That way reading and writing is piss easy and it can also be sent over a network!
:-)
That way reading and writing is piss easy and it can also be sent over a network!
:-)
Serialization is the way to go! You can serialize it to a compressed stream, and then scramble the bytes if you don''t want anyone to break the data. It''s so much easier than creating a custom format.
I think the only time a custom format is more useful is for efficiency when you really want the minimum to be sent over a network, or want the smallest packet or file size.
But unless such issues are really important, use Serialization - it was written for your exact situation.
I think the only time a custom format is more useful is for efficiency when you really want the minimum to be sent over a network, or want the smallest packet or file size.
But unless such issues are really important, use Serialization - it was written for your exact situation.
Author
Thanks for the responses...
Jindocai...
I''m not entirely sure I understand what you mean here. Could you post to say what you mean?
In your explanation of cutting up ints, I haven''t got a clue how it all works, but I understand what it does; thanks.
About serialization:
True, Serialization is probably much easier, but in the long run, it could be a bit of a problem. In my files, I will be storing images (currently GIF images of dimension of anything from 320x240 to 640x480), sounds (not sure, but probably MIDI, MP3, and WAV), and just normal text. Eventually, the file could get really large. If I''m using a byte stream, I can skip so many bytes, and just load the bytes I need, at the start of the level. ObjectInputStream has no skipBytes() or similar method, so I would have to load the entire file, and then access just what I want. I know I could use separate files to overcome this, but I would prefer just one file.
Also, and correct me if this is wrong, I have heard that Serialization is different across different SDK''s and JRE''s.
My game is not network oriented (or do I mean orientated?), and will never be; it is not the kind of game that can be networked.
What I was thinking of doing, was to store the block length in Hexadecimal (using the method in the class Integer), then decode it with Integer.parseInt().
I may decide to use Serialization. But I''m not sure - true, it is much easier, but I''m concerned about loading huge files. And also, it kind of feels a bit like cheating
.
Oh, and by the way, long time no speak Michael; how''s FBA going? Email me if you want.
Also, just something that'' confusing me: Is there no JCanvas in Swing? It''s just that I''ve been using the JDK 1.1 for ages, and have only just moved over to SDK 1.4. Currently, all my painting is being done on a JPanel, it just seems odd that JCanvas has disappeared.
Jindocai...
quote:But generally you don''t need to do this since if you used fixed sized data labels, you can then load the data in correct sized variables.
I''m not entirely sure I understand what you mean here. Could you post to say what you mean?
In your explanation of cutting up ints, I haven''t got a clue how it all works, but I understand what it does; thanks.
About serialization:
True, Serialization is probably much easier, but in the long run, it could be a bit of a problem. In my files, I will be storing images (currently GIF images of dimension of anything from 320x240 to 640x480), sounds (not sure, but probably MIDI, MP3, and WAV), and just normal text. Eventually, the file could get really large. If I''m using a byte stream, I can skip so many bytes, and just load the bytes I need, at the start of the level. ObjectInputStream has no skipBytes() or similar method, so I would have to load the entire file, and then access just what I want. I know I could use separate files to overcome this, but I would prefer just one file.
Also, and correct me if this is wrong, I have heard that Serialization is different across different SDK''s and JRE''s.
My game is not network oriented (or do I mean orientated?), and will never be; it is not the kind of game that can be networked.
What I was thinking of doing, was to store the block length in Hexadecimal (using the method in the class Integer), then decode it with Integer.parseInt().
I may decide to use Serialization. But I''m not sure - true, it is much easier, but I''m concerned about loading huge files. And also, it kind of feels a bit like cheating
.Oh, and by the way, long time no speak Michael; how''s FBA going? Email me if you want.
Also, just something that'' confusing me: Is there no JCanvas in Swing? It''s just that I''ve been using the JDK 1.1 for ages, and have only just moved over to SDK 1.4. Currently, all my painting is being done on a JPanel, it just seems odd that JCanvas has disappeared.
java.io.DataInputStream
java.io.DataOutputStream
More work for you than using Serialization, but it will do what you want.
"There is no reason good should not triumph at least as often as evil. The triumph of anything is a matter of organization. If there are such things as angels, I hope that they''re organized along the lines of the mafia." -Kurt Vonnegut
java.io.DataOutputStream
More work for you than using Serialization, but it will do what you want.
"There is no reason good should not triumph at least as often as evil. The triumph of anything is a matter of organization. If there are such things as angels, I hope that they''re organized along the lines of the mafia." -Kurt Vonnegut
Hi,
I think you can decide whether you should use Serialization, or create a custom file format, rather very quickly. Just answer yes or no to the following questions.
Note:
if (yes-count >= no-count) {
use Serialization
} else create the custom file format
1] Will your code be run in a homogenous (single JRE/JDK version) enviroment ?
2] Would you like to share data between running instances of your program ?
3] Do you intend to use networking to share the data/objects ?
4] Do you intend to run the same versions of your program, that should share the data ?
soooo simple
I think you can decide whether you should use Serialization, or create a custom file format, rather very quickly. Just answer yes or no to the following questions.
Note:
if (yes-count >= no-count) {
use Serialization
} else create the custom file format
1] Will your code be run in a homogenous (single JRE/JDK version) enviroment ?
2] Would you like to share data between running instances of your program ?
3] Do you intend to use networking to share the data/objects ?
4] Do you intend to run the same versions of your program, that should share the data ?
soooo simple
Author
quote:1] Will your code be run in a homogenous (single JRE/JDK version) enviroment ?
Probably not; I would like it to work on a number of JREs, but it isn''t absolutely vital.
quote:2] Would you like to share data between running instances of your program ?
3] Do you intend to use networking to share the data/objects ?
4] Do you intend to run the same versions of your program, that should share the data ?
No to all of the above.
But I''m still not sure; I really don''t want to have to load a whole data file just for a few bytes.
I suppose I could create separate serialized files inside a ZIP file, but that would be a pain as I would need to handle the retrieval of them.
It''''s over for now... until yesterday begins again... tomorrow... tomorrow... tomorrow...
[Powerman 5000 - Watch the Sky for Me]
Hi,
but what I tried to say is ..
it is just enough to look at the dominant answer ... and that is yes or no ...
you CAN use serialization everytime ... but not your custom file format. Use of it that must be cautiously planned and designed.
but what I tried to say is ..
it is just enough to look at the dominant answer ... and that is yes or no ...
you CAN use serialization everytime ... but not your custom file format. Use of it that must be cautiously planned and designed.
Author
Thanks Petr.
I just want to run my ideas by you - or anyone else that wants to respond (for Serialization):
I will use a Node hierarchy (like the one used for Java3D scenegraphs...) to store all my stuff in, using LeafNodes to store the actual image/movie/sound/etc. data in. I will then just Serialize the entire lot via a call to writeObject() passing the root Node.
One thing: are the classes for Nodes (like in Java3D), or will I have to write my own?
It''''s over for now... until yesterday begins again... tomorrow... tomorrow... tomorrow...
[Powerman 5000 - Watch the Sky for Me]
I just want to run my ideas by you - or anyone else that wants to respond (for Serialization):
I will use a Node hierarchy (like the one used for Java3D scenegraphs...) to store all my stuff in, using LeafNodes to store the actual image/movie/sound/etc. data in. I will then just Serialize the entire lot via a call to writeObject() passing the root Node.
One thing: are the classes for Nodes (like in Java3D), or will I have to write my own?
It''''s over for now... until yesterday begins again... tomorrow... tomorrow... tomorrow...
[Powerman 5000 - Watch the Sky for Me]
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
