Endians and portability
I have a nice game project I''m working on that I know works on all Windows and X systems which have OpenGL and SDL support. The problem is, I''m almost definite that it''s little endian only, and possibly even x86 only (not so sure).
Most importantly, I want to write it so that it runs on the G* series of CPUs (e.g. G3, G4, G5)., under OS X. I load textures and map files and stuff manually, with the assumption that both the files and my memory are in the same endian (i.e. little).
What I''d like to do is convert this to work on a big-endian system when built for it. Unfortunately, I only have sketchy knowledge about how exactly the byte order works and how I switch it between endians...
First, how are 32-bit and 64-bit vars switched? Are they simply backwards from either platform, or is there more complex switching going on?
Second, in a packed structure, how do I deal with that? Is it switched per member, or as a whole, or some other way? How do I deal with packing?
Third, is it possible to switch endians without actually knowing the type of an object? Basically, can I take a void* and simply loop through the bytes and switch to toggle the endian?
I really very little clue how to do this, and I never really understood how the Q2 code dealt with it (I saw endian functions, but they always just returned their parameter, and I didn''t get wtf they were doing). I would really appreciate some explanation and/or resources.
the byte order changes for each variable, so if you have an int, you have to flip 4 bytes, if you have a double you have to flip the 8 bytes, and for a byte you don''t have to flip anything at all. structs and strings are not flipped, only the variables that are in it, so you can''t just load a struct from a file by passing a void pointer to fread or so.
you can of course write a special version for each endian type, (only flip if it is high endian, by example), but that''s ugly. the best way to make it endian safe is to read one byte a time from a file, and then construct the variables this way:
long v = byte1 + (byte2 << 8) + (byte3 << 16) + (byte4 << 24);
this will always set your bytes in the right order, no mather what endian system you are using
My Site
you can of course write a special version for each endian type, (only flip if it is high endian, by example), but that''s ugly. the best way to make it endian safe is to read one byte a time from a file, and then construct the variables this way:
long v = byte1 + (byte2 << 8) + (byte3 << 16) + (byte4 << 24);
this will always set your bytes in the right order, no mather what endian system you are using
My Site
And I have to do the same with structures?
Damn, this could get ugly fast. Maybe OS X just slid out of my TO DO list...I can't even think of some easy way to abstract this all in a pretty way...I can't really write a modified fread that will deal with stuff automatically, since it has to know what type it's reading to...damnit!
[edited by - Promit on October 8, 2003 6:56:28 PM]
Damn, this could get ugly fast. Maybe OS X just slid out of my TO DO list...I can't even think of some easy way to abstract this all in a pretty way...I can't really write a modified fread that will deal with stuff automatically, since it has to know what type it's reading to...damnit!
[edited by - Promit on October 8, 2003 6:56:28 PM]
what I did is, I just wrote my own endian safe stream classes, and use that everywhere. Reading a struct isn't too ugly, like:
and for most basic structs, I added a seperated stream function, so I can stream the whole struct at once
now I can just do
stream >> matrix;
it's a bit more work than a single fread, but it takes some effort to make your code cross platform. And I like it this way
My Site
[edited by - Quasar3D on October 8, 2003 7:09:11 PM]
[edited by - Quasar3D on October 8, 2003 7:09:50 PM]
stream >> header.bmpSize >> header.reserved1 >> header.reserved2 >> header.bitsOffset >> header.bmiSize >> header.width >> header.height >> header.nPlanes >> header.bpp >> header.compression >> header.imageSize >> header.xPixelsPerMeter >> header.yPixelsPerMeter >> header.clrUsed >> header.clrImportant;and for most basic structs, I added a seperated stream function, so I can stream the whole struct at once
ISTREAMLE& operator >> (ISTREAMLE &a,MATRIX4 &b){ for(int i = 0;i < 4;i++) for(int j = 0;j < 4;j++) { a >> b.m[i][j]; } return a;}now I can just do
stream >> matrix;
it's a bit more work than a single fread, but it takes some effort to make your code cross platform. And I like it this way
My Site
[edited by - Quasar3D on October 8, 2003 7:09:11 PM]
[edited by - Quasar3D on October 8, 2003 7:09:50 PM]
quote:And I have to do the same with structures?
- No, only DWORDs and WORDs. The position of that DWORD/WORD within the structure won''t change.
- If you do things "byte by byte" instead of as DWORDs then the endianness doesn''t matter...
- ...as long as you don''t need to support Vax mainframe CPUs
[which you don''t]- Bear in mind that some CPUs require different alignments, particularly for 64bit types.
- Bear in mind that certain Motorola CPUs and some from other vendors can''t read WORD or DWORD values from odd memory addresses so:
struct{ byte blah; word blah2;} is bad on those (you''ll get a "privilege violation").
- Beware of C++ "bool" and "enum" types in structures - the number of bytes they''ll take in the structure can differ depending on the compiler!!. On some they''ll take 1 byte, others 4 bytes etc...
- Putting an unused entry at the end of your enum equal to 0x7fffffff or similar will force the size of the type to 32bits (you''ll notice that in the DirectX headers for this exact reason). People often use their own "bool" types that are guaranteed to be 32bits.
--
Simon O''Connor
3D Game Programmer &
Microsoft DirectX MVP
Ok, let''s say I have a little endian file. I read from that in the conventional way, and then I want to convert it to big.
The way I want to do this is by having each structure fix it''s own endian. So I need to know how to:
1) Swap a long
2) swap a float
And that''s all, I think...
The way I want to do this is by having each structure fix it''s own endian. So I need to know how to:
1) Swap a long
2) swap a float
And that''s all, I think...
quote:Original post by S1CAquote:And I have to do the same with structures?
- No, only DWORDs and WORDs. The position of that DWORD/WORD within the structure won''t change.
and QWORDS, floats and doubles, and of course signed long and short
quote:
- Bear in mind that some CPUs require different alignments, particularly for 64bit types.
- Bear in mind that certain Motorola CPUs and some from other vendors can''t read WORD or DWORD values from odd memory addresses so:struct{ byte blah; word blah2;}
is bad on those (you''ll get a "privilege violation").
the compiler does align this automatically, right? then if you are reading var by var, you also don''t have problems with this. that struct might only use much more memory than you want
My Site
quote:and QWORDS, floats and doubles, and of course signed long and short
Yep. And indeed any integral type consisting of multiple bytes. Beware that QWORDs (and anything of that size or larger) aren''t integral ANSI types so whether you swap the order of all the the bytes or only per-DWORD depends on the compiler in use (and often the target CPU).
quote:the compiler does align this automatically, right?
Yes it does, however the default alignment is *different* for different compilers and different target platforms.
The main reason for someone to care about endianness is data transfer between platforms.
The intention of me mentioning internal struct alignment was for the OP to be aware that it differs per platform and compiler.
Which is very important if you''re saving and loading complete structs - a very common thing...
> ...as long as you don''t need to support Vax mainframe CPUs
why would vax be different?
i know that they have a non ieee fp format, but i''ve written some (hopefully) endian safe netcode and it ran fine both on a vaxstation and a (vax)mainframe.
why would vax be different?
i know that they have a non ieee fp format, but i''ve written some (hopefully) endian safe netcode and it ran fine both on a vaxstation and a (vax)mainframe.
Hello Promit,
I am all to familiar to the edian swapping issue, since at work we run VAX, DEC UNIX, SUN, and PCs (window/linux).
What we have is a small class the has methods to swap shorts(2bytes),
ints(4bytes), long/int64(8 bytes) this class has method to check endianess of the system also.
We store files base on the system used and only need to worry about swapping when reading a file on a different endain system.
Floats and double are swap just like ints and longs.
There is on VAX different float formats G float, D float and IEEE.
But don''t worry about these, VAX is a dieing OS.
Any the swap is in byte order: that if it is 1234 for an int, it becomes 4321
short 12 21, long/Int64 12345678 87654321.
I would have each of your classes that are reading in files to swap after reading only if the endiannes of file is different.
Also do something like quasar3d has for steaming data of a struct or class.
This will remove any padding the compiler added for alignment of structures/classes.
One note in the endian class have all swapping methods (one each for shorts, ints, and ints64) take a void pointer and a default size of array of 1.
That way when swapping just one this in simply call with address works, or if you have a whole array of same data type call it with address of first data element and number of elements to swap.
Lord Bart
I am all to familiar to the edian swapping issue, since at work we run VAX, DEC UNIX, SUN, and PCs (window/linux).
What we have is a small class the has methods to swap shorts(2bytes),
ints(4bytes), long/int64(8 bytes) this class has method to check endianess of the system also.
We store files base on the system used and only need to worry about swapping when reading a file on a different endain system.
Floats and double are swap just like ints and longs.
There is on VAX different float formats G float, D float and IEEE.
But don''t worry about these, VAX is a dieing OS.
Any the swap is in byte order: that if it is 1234 for an int, it becomes 4321
short 12 21, long/Int64 12345678 87654321.
I would have each of your classes that are reading in files to swap after reading only if the endiannes of file is different.
Also do something like quasar3d has for steaming data of a struct or class.
This will remove any padding the compiler added for alignment of structures/classes.
One note in the endian class have all swapping methods (one each for shorts, ints, and ints64) take a void pointer and a default size of array of 1.
That way when swapping just one this in simply call with address works, or if you have a whole array of same data type call it with address of first data element and number of elements to swap.
Lord Bart
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
