Sign in to follow this  
KulSeran

Endian. When do you care?

Recommended Posts

Really this comes down to: when do you actually care about the endian of something? I understand the difference of big and little endian, and that it affects what you save out to a file. If a system is big endian, and I need to read in from a file saved in little endian, what changes? If I save an int(32bit) then I should be concernerned. What about a float(32bit) or a char array(4*8bit)? ---- More background, if I have a Read function can I make it deal with endianness without making: Read_uint16_LE2BE Read_uint16_BE2LE ..ect and without having Read ( data*, size, bool_endian_sensitive );

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
int, yes. float, probably not. char array - not likely.

Share this post


Link to post
Share on other sites
I only care about endian when working with networks.
Since I am only working on little endian platforms I don t care about it in conjunction with binary files.
If you have to port your code to a big endian platform once, you add some compilation flags and add some conversion functions in your io implementation for binary data (htonl,....)

Share this post


Link to post
Share on other sites
It mostly matters when saving to a file that you want to be able to load on a different platform or when sending data over a network. It matters for any data type larger than 1 byte - int, float, long, double, bool (on some platforms), etc.

Share this post


Link to post
Share on other sites
Its the "atomic" types, that are greater than 1-byte in length, that matter..

So if I write on my PC


struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);




Then when I read from my Mac, Xbox360, PS3, etc. I data.name will be ok, but I must swap a, b, and c.




fread(&data, sizeof(MyStruct), 1, myFile);
MYSWAPMACRO_32(&data.a);
MYSWAPMACRO_16(&data.b);
MYSWAPMACRO_32(&data.c);



Share this post


Link to post
Share on other sites
Thanks.
So, for just about anything that is not 8bit I would need to have support to swap the bytes around.
But since the char arrays are unaffected by endian, I can't just make a system that says:

read ( void *data, size_t size )
{
size_t bytes_read = fread ( file, size,1, data );
if ( bytes_read == 8 ) FixEndian64 ( data );
if ( bytes_read == 4 ) FixEndian32 ( data );
if ( bytes_read == 2 ) FixEndian16 ( data );
}



I would actually have to leave the swapping to the next higher level, where the datatype is known?

Share this post


Link to post
Share on other sites
Yes you have to know what data it is your reading to swap it correctly... So you could have your own ReadShort, ReadInt, etc. functions that do the appropriate swapping. But you could not just assume from the size of the read command (a read of 4 bytes could be of structure of two shorts or a single int etc..).

Edit... You could also rely on polymorphism, which could make you life a little easier...

e.g.


class MyReader
{
bool Read(int *pVal) { ReadSwapped32(pVal,sizeof(int)); )
bool Read(short *pVal) { ReadSwapped16(pVal,sizeof(short)); )
bool Read(char *pData, size_t n);// No swapping
.
.
}

reader.Read(myInt);
reader.Read(myShort);
reader.Read(myStr,32);



Share this post


Link to post
Share on other sites
Well, thanks griffin,
I'm mimiking the fstream.read and fread functionality, and so have no connection to the data,
but at a higher level I will need endian correctness, so I will probably come up with a polymorphic endian utility class that can be used by the subsystems that need it.

Share this post


Link to post
Share on other sites
you should also be aware that some casting that works on little-endian doesnt work at all on big endian-


typedef unsigned char byte;
int i = 255;
byte c = *((byte*)&i);
cout << c;


This will output 255 on a little endian system, and 0 on a big endian system.

If you need to do stuff like this, use a reinterpret_cast instaed of the crappy C-style cast I've used here.

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Quote:
Original post by mattnewport
It mostly matters when saving to a file that you want to be able to load on a different platform or when sending data over a network. It matters for any data type larger than 1 byte - int, float, long, double, bool (on some platforms), etc.


Surely not float? Isn't it the same on all platforms with IEEE floating point?

Share this post


Link to post
Share on other sites
i only care about endian when reading LightWave files on Windows and on winsock programming. a float yes, a char array no (unless that 4-byte character array represents a 32-bit value loaded from a file or something that you need to reverse and reinterpret).

Share this post


Link to post
Share on other sites
Quote:

struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);


Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.

Share this post


Link to post
Share on other sites
Quote:
Original post by asp_
Quote:

struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);


Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.
Never say never. If you're doing it for a file that will only ever be loaded by a particular platform (and you should be using platform-specific data files for best performance), then the padding will always be the same, and this kind of direct memory read/write is precisely what you should be doing if you want good load/save times.

Share this post


Link to post
Share on other sites
If I would venture a guess I doubt your bottleneck would ever be that code, no matter if you split it to one call per field. I doubt there's any sane reason to do it except for lazyness.

It's also not just platform specific but compiler specific and affected by a lot of factors such as build settings and so on.

I'd say this is as close to a never as you'll get.

Share this post


Link to post
Share on other sites
Quote:
Original post by asp_
Quote:

struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);


Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.


Using compiler packing hints can go along way to ensureing that this works correctly if you need to do this in a cross platform way.

Cheers
Chris

Share this post


Link to post
Share on other sites
Quote:
Original post by asp_
If I would venture a guess I doubt your bottleneck would ever be that code, no matter if you split it to one call per field. I doubt there's any sane reason to do it except for lazyness.
On certain platforms - mainly consoles like the PS2 - it's much more efficient to schedule a single large read than lots of small ones. These are systems with very little in the way of filesystem cache; they are designed to stream data in from discs as quickly as possible, without any stopping/starting or faffing about.

If you really can't cope with it, you could read the data into a buffer and then 'read' that buffer out member by member, but there's really no need.

It's not lazyness, it's Sony/MS/Ninty telling you that if your game takes longer than X seconds to load a level, they won't let you release it.

Quote:
It's also not just platform specific but compiler specific and affected by a lot of factors such as build settings and so on.
Yes, it's compiler specific (and in fact, as a result it's not platform specific, if you run the same binary on two different platforms the padding will not change), but it's still deterministic. If you know what the padding is like, you can account for it. Repacking and converting assets into the most efficient layout/format for a given platform is a common practice for cross-platform titles (e.g. DDS on Xbox vs. TM2 on PS2), and writing out datastructures with the correct amount of padding is frequently rolled into that.

Share this post


Link to post
Share on other sites
Quote:
Original post by chollida1
Quote:
Original post by asp_
Quote:

struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);


Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.


Using compiler packing hints can go along way to ensureing that this works correctly if you need to do this in a cross platform way.

Cheers
Chris


Uh, compiler packing hints are highly compiler-specific, and not all compilers are available on all platforms. That's quite opposed to "cross-platform". The "cross-platform", and proper, way is to read and write members individually. Zomg!

Share this post


Link to post
Share on other sites
Quote:
Original post by Zahlman
Quote:
Original post by chollida1
Quote:
Original post by asp_
Quote:

struct MyStruct
{
char name[32];
int a;
short b;
float c;
};
.
.
MyStruct data;
.
.
fwrite(&data, sizeof(a), 1, myFile);


Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.


Using compiler packing hints can go along way to ensureing that this works correctly if you need to do this in a cross platform way.

Cheers
Chris


Uh, compiler packing hints are highly compiler-specific, and not all compilers are available on all platforms. That's quite opposed to "cross-platform". The "cross-platform", and proper, way is to read and write members individually. Zomg!


Very much agreed that compiler hints are OS specific:) Perhaps I wasn't clear in my point. By using compiler pragma's you can ensure that the structure's packing is in a format you know.

Now that you know the packing you can read back in the file with one read on any OS:)

Actually SuperPig summed up my point very well in his post above.
Quote:

Yes, it's compiler specific (and in fact, as a result it's not platform specific, if you run the same binary on two different platforms the padding will not change), but it's still deterministic. If you know what the padding is like, you can account for it. Repacking and converting assets into the most efficient layout/format for a given platform is a common practice for cross-platform titles (e.g. DDS on Xbox vs. TM2 on PS2), and writing out datastructures with the correct amount of padding is frequently rolled into that.


Cheers
Chris

Share this post


Link to post
Share on other sites
Quote:
Original post by asp_
If I would venture a guess I doubt your bottleneck would ever be that code, no matter if you split it to one call per field. I doubt there's any sane reason to do it except for lazyness.

It's also not just platform specific but compiler specific and affected by a lot of factors such as build settings and so on.

I'd say this is as close to a never as you'll get.


Your ventured guess is wrong. Most console games (at least the ones that have decent load times) do something exactly like this. Data structures are read straight off disk into memory, pointers are fixed up and off you go, loading is nothing more than a sequential read into memory and a pointer fix up pass. Hell, some games even skip the pointer fix-up and read the pointers in unchanged, but that really is playing with fire.

It is platform specific but that's what you have asset pipelines for - to convert your non-platform-specific source assets into very-platform-specific runtime assets that load quickly and are formatted optimally for the target platform.

It may not be 'clean' or very cross-platform but sometimes this kind of thing is necessary for performance, particularly on consoles. If more PC games did things this way maybe we wouldn't see PC games that take way longer to load from hard disk than console games load from optical disks.

Share this post


Link to post
Share on other sites
Yeah I overlooked DVD and CD drives where the penalty for a seek and cache misses is extremely high. I'm having some issues seeing how doing large block reads with a high percentage of waste data is efficient when there is a hardware cache which would clearly prevent repeated seeks with each read.

I can see this coming with a fairly large penalty if there is no software caching resulting in data being requested over the bus for each request. I guess older consoles which are RAM starved would generally have a small stream cache.

Seems to me that the best method in this scenario would be to implement a generic container object which serializes to and from memory, reading and writing optimal sizes to disk. It would also mean you do not waste throughput to crap data.

Quote:

If more PC games did things this way maybe we wouldn't see PC games that take way longer to load from hard disk than console games load from optical disks.

I don't think there's any performance difference on systems with larger software caches. Maybe I'm misstaken here?

Share this post


Link to post
Share on other sites

Quote:

Never simply dump a structure to file like this. Write each field separately because the structure will contain padding between fields in order to align data effectively. This means you'll end up writing crap to the file and the padding may be different between platforms.


This is safe (and the most efficent way of working, as don't have to bother with little reads from disk) as long as you manually pad the structure so all members are aligned to the size of their types. So actually to be safe the structure should be re-written as:


struct MyStruct
{
char name[32];
int a;
float c;<<On some platforms this will align to 32-bit boundary
short b;
char pad[2];<<Explicitally pad to 32-bit boundary just to be ultra safe (don't think this is nessacary)
};



Share this post


Link to post
Share on other sites
Quote:

So actually to be safe the structure should be re-written as...

That sounds like an even worse idea. Now you're exposing padding to the user of the data structure. What about IA-64? Are you going to include #if defined()'s for all possible platforms?

Share this post


Link to post
Share on other sites
Quote:
Original post by asp_
Yeah I overlooked DVD and CD drives where the penalty for a seek and cache misses is extremely high. I'm having some issues seeing how doing large block reads with a high percentage of waste data is efficient when there is a hardware cache which would clearly prevent repeated seeks with each read.
It's less the seek penalty - if your small reads are sequential, you won't really be seeking around - and more the housekeeping overhead. Checking that you don't need to seek. Packing and sending the read request over the bus. Etcetera.

Quote:
I can see this coming with a fairly large penalty if there is no software caching resulting in data being requested over the bus for each request. I guess older consoles which are RAM starved would generally have a small stream cache.
Aye.

Quote:
Seems to me that the best method in this scenario would be to implement a generic container object which serializes to and from memory, reading and writing optimal sizes to disk. It would also mean you do not waste throughput to crap data.
So... a software cache [wink] Moving stuff around in memory is going to be faster, sure. But why move it around at all when you can prepare all the data offline? Bandwidth on the consoles we're talking about isn't usually the problem - it's more the latency. Making 1/10th the read requests with 10x the data can still end up faster.

Quote:

I don't think there's any performance difference on systems with larger software caches. Maybe I'm misstaken here?
There's a performance difference - moving stuff around in memory doesn't come for free - but it's not nearly as big as reading field-by-field from a CD/DVD.

Another benefit of single-large-read is asynchronous transfer: after telling it "read N bytes from position X to location Y," you can get on with running a loading screen or FMV or something without needing to micromanage the data. Sure, on some platforms you can achieve that using a seperate thread, but if the hardware itself has support for asynchronous reads from CD/DVD, it's best to use it. Especially if the CPU isn't multi-core, where running a CPU thread for it would cut into whatever else you want to be doing. (I believe the original Xbox is like this).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this