Sign in to follow this  

reading float from binary file (endian problem?)

This topic is 1183 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

So I've been working on writing and reading binary file, and I'm surprised that I have gone for as long as I did without any big problems, but I seem to have run into this roadblock that I can't get around. I'm having trouble reading float values from the files I've written, and I'm not exactly sure what causes the problem.

// say this is the file
//  0.0
// 40.0
//  1.0

int main(ig var, nore nvar)
{
	float a, b, c;
	std::ifstream f("myfile.fff", std::ios::binary);
	
	f.read((char*)&a, sizeof(float));
	f.read((char*)&b, sizeof(float));
	f.read((char*)&c, sizeof(float));

	printf("a: %f\nb: %f\nc: %f\n", a, b, c);
}
	

The results I get are:

 

a: 0 (good, that's what I wanted)

 

b: 0.0000000000000000000000000000000000000000237211804 (... that's not 40)

 

c: 0.000000000000000000000000000000000000373583929 (uuugghh)

 

 

I searched on google and found that it might be an endian problem and this is where it get confusing for me. I looked in the debugger, changed the value view to 'bytes hex' and I get:

 

the value of 40:  00 00 20 42

 

the value of the result:  20 42 00 00

 

Now because I'm self-taught in c++ there are numerous gaps in my skill set, but I have been understanding that endian-ness is like reading bytes from left to right, or right to left, but this result seems more "shuffled". I also understand that endian-ness differs across processor architectures, so why would this be happening when the file is being written and read on the same machine? 

 

Do I need to fix this with byte swapping?(shuffling) or is there something else that could be the culprit?

Share this post


Link to post
Share on other sites

Show the code that writes it out. Assuming you're writing this out on the same machine as you are reading it from, and assuming you did not apply any transforms to the bytes... No, this would not be an endian problem.

 

Hard to say without a bit more detail, such as the code for writing it.

Edited by Washu

Share this post


Link to post
Share on other sites

float x = 0.0f, y = 40.0f, z = 1.0f;

void writeTheFile()
{
	std::ofstream f;
	f.open(wFileExt, std::ios::trunc | std::ios::binary | std::ios::out);

	f.write((const char*)&x, sizeof(float));
	f.write((const char*)&y, sizeof(float));
	f.write((const char*)&z, sizeof(float));

	f.close() // did I forget this in the last post? I know it's in the program code...
}

This is basically it.

Share this post


Link to post
Share on other sites

What happens if you remove the casts to const char* and char*?

The casts are necessary. Read the signatures of the functions (1 2).

Although I would have used reinterpret_cast just to express it better. Edited by Washu

Share this post


Link to post
Share on other sites

Ah yes, they are indeed necessary. I was on a tablet before, so couldn't immediately check it out, sorry about that.

 

I now tried the code here (in Visual Studio 2012) and it does indeed work correctly.

 

NSDuo: maybe you could try the code as Washu posted it ("two snippets together") and see what that gives?

 

Also, what compiler do you use? Has the file been written on the same system that it was read on?

Share this post


Link to post
Share on other sites

Since the 20 and 42 are in the same order, it's not a byte endian problem. [...] The most likely problem is that your reader and writer code don't match the amount of data they're handling. [...]

 

That right there.

 

Needless to say posting your actual code is necessary because often the bug you experience is not a direct result of the "basically it" code - if it were that easy you wouldn't have asked for help. Usually it is some small detail (sometimes even a typo) that is present in the original code which you are missing. Then it takes all this extra conversation to finally see the real code with the actual problem. If it seems "there is too much code", consider narrowing the problem area down to a small bit of code which reproduces the problem you are experiencing. This could even lead you to identify and debug the issue prior to posting the narrowed down code.

Edited by achild

Share this post


Link to post
Share on other sites

I suppose I was being a little stupid by not posting the full code ( I don't know if it was my ego or whatever...) but I asked for help, so here it is...

int scene::dumpToFile(std::string file)
	{
		
		std::string wFileExt = file;
		wFileExt.append(".sinncascene");
		//wFileExt.append(".txt");
		// open for writing only
		//f.open(wFileExt);
		std::ofstream f;
		f.open(wFileExt, std::ios::trunc | std::ios::binary | std::ios::out);
		if (!f.good())
		{
			printf("The file '%s' was not good", wFileExt.c_str());
			return -1;
		
		} else if (!f.is_open())
		{
			printf("The file '%s' could not be opened", wFileExt.c_str());
			return -1;
		}
		
		// set header
		// 8 bytes
		ui8* header = (ui8*)"sinnca_";
		f.write((char*)header, 8);
		// 2 bytes
		ui16 secondaryHeader = 18420;
		f.write((const char*)&secondaryHeader, 2);
		
		/*
		// set size of file
		ui64 fSize = sizeof(uint) +
					 sizeof(long) +
					 sizeof(entity) * entityRef.size() +
					 sizeof(grid) * gridRef.size() +
					 sizeof(scene) * sceneRef.size() +
					 sizeof(image) * imageRef.size() +
					 sizeof(texture) * textureRef.size();
		// 8 bytes
		f << fSize;
		 */
		
		// write the number of assets
		// 4 bytes
		ui32 noOfAssets = (ui32)(imageRef.size() +
						  textureRef.size());
		f.write((const char*)&noOfAssets, 4);
		
		// write number of nodes
		// 4 bytes
		ui32 noOfNodes = (ui32)(entityRef.size() +
						 gridRef.size() +
						 sceneRef.size());
		f.write((const char*)&noOfNodes, 4);
		
		
		
		/* ********************************************************************************
		 WRITE IMAGES
		 ******************************************************************************* */
		if (imageRef.size() > 0)
		{
			for (int i = 0; i < imageRef.size(); i++)
			{
				// what kind of asset is this? (2 bytes)
				//f.write((const char*) 1, 2);
				f.put((const char) 1); // 1 = image
				
				// what number asset are we at? (2 bytes)
				f.write((const char*) &i, 2);
				
				// what is the size of this asset? (2 bytes)
				ui16 sizeofthing = sizeof(ui16) + imageRef[i]->name.length() + imageRef[i]->path.length() + ((ui8)imageRef[i]->tb);
				f.write((const char*)&sizeofthing, 2);
				// this is for just in case the engine doesn't support this type of object yet
				
				
				// the size of the name and the name itself
				sizeofthing = imageRef[i]->name.length();
				f.write((const char*)&sizeofthing, 2);
				f << imageRef[i]->name.c_str();
				
				// the size of the filepath and the filepath itself
				sizeofthing = imageRef[i]->path.length();
				f.write((const char*)&sizeofthing, 2);
				f << imageRef[i]->path.c_str();
				
				// the texture blend mode
				f.write((const char*)&imageRef[i]->tb, 1);
				
			}
		}
		
		/* ********************************************************************************
		 WRITE TEXTURES AND COLORS
		 ******************************************************************************* */
		if (textureRef.size() > 0)
		{
			for (int i = 0; i < textureRef.size(); i++)
			{
				// what kind of asset is this? (2 bytes)
				//f.write((const char*) 2, 2);
				f.put((const char) 2); // 2 = texture
				
				// what number asset are we at? (2 bytes)
				f.write((const char*) &i, 2);
				
				// what is the size of this asset? (2 bytes)
				ui16 sizeofthing = (sizeof(ui16) + textureRef[i]->name.length() + (sizeof(ui16) * 5));
				f.write((const char*)&sizeofthing, 2);
				// this is for just in case the engine doesn't support this type of object yet
				
				
				// the size of the name and the name itself
				sizeofthing = textureRef[i]->name.length();
				f.write((const char*)&sizeofthing, 2);
				f << textureRef[i]->name.c_str();
				
				// find the image this texture is linking to
				if (imageRef.size() > 0)
				{
					for (int j = 0; j < imageRef.size(); j++)
					{
						// theres got to be a faster way to do this
						if (imageRef[j] == textureRef[i]->getSource())
						{
							// what image does this texture use? (2 bytes)
							f.write((const char*) &j, 2);
							break;
						}
					}
				}
				
				// write the x and y offset into the file (2 + 2 bytes)
				static int tempx, tempy;
				
				textureRef[i]->getOffset(tempx, tempy);
				f.write((const char*) &tempx, 2);
				f.write((const char*) &tempy, 2);
				
				// write the x and y size into the file (2 + 2 bytes)
				textureRef[i]->getSize(tempx, tempy);
				f.write((const char*) &tempx, 2);
				f.write((const char*) &tempy, 2);
				
				// write the base color
				f.write((const char*)&textureRef[i]->r, 1);
				f.write((const char*)&textureRef[i]->g, 1);
				f.write((const char*)&textureRef[i]->b, 1);
				f.write((const char*)&textureRef[i]->a, 1);
			}
		}
		
		// terminate the asset list, move on to nodes
		//f.write((const char*) -1, 2);
		f.put((const char) -1);
		// -1 is used to tell the engine to move on to other stuff
		
		/* ********************************************************************************
		 WRITE ENTITIES
		 ******************************************************************************* */
		if (entityRef.size() > 0)
		{
			for (int i = 0; i < entityRef.size(); i++)
			{
			
				// put it in!
				//f.write((const char*) 1, 2);
				f.put((const char) 1);
				// 1 = entity
				
				// what number node is this?
				f.write((const char*) &i, 2);
				
				// size of the entity entry
				ui16 sizeofthing = entityRef[i]->name.length() + 26;
				f.write((const char*)&sizeofthing, 2);
				
				
				// name and size of name
				sizeofthing = entityRef[i]->name.length();
				f.write((const char*)&sizeofthing, 2);
				f << entityRef[i]->name.c_str();
				
				// render object
				// currently only supports the sprite class
				//f.write((const char*) 0, 2);
				f.put((const char) 0);
				
				// but is it drawable?
				f.write((const char*) &entityRef[i]->draw, 1);
				
				// color
				for (int j = 0; j < colorRef.size(); j++)
				{
					if (colorRef[j] == entityRef[i]->col)
					{
						f.write((const char*) &j, 2);
						break;
					}
				}
				
				float position[9] = {
					(float)entityRef[i]->pos.x,
					(float)entityRef[i]->pos.y,
					(float)entityRef[i]->pos.z,
					(float)entityRef[i]->rot.x,
					(float)entityRef[i]->rot.y,
					(float)entityRef[i]->rot.z,
					(float)entityRef[i]->scl.x,
					(float)entityRef[i]->scl.y,
					(float)entityRef[i]->scl.z
				};
				
				// entity position
				f.write((const char*) &position[0], sizeof(float));
				f.write((const char*) &position[1], sizeof(float));
				f.write((const char*) &position[2], sizeof(float));
				
				// entity rotation
				f.write((const char*) &position[3], sizeof(float));
				f.write((const char*) &position[4], sizeof(float));
				f.write((const char*) &position[5], sizeof(float));
				
				// entity scale
				f.write((const char*) &position[6], sizeof(float));
				f.write((const char*) &position[7], sizeof(float));
				f.write((const char*) &position[8], sizeof(float));
				
				// find the parent!
				if (entityRef[i]->parent != Tree->currentScene)
				{
					for (int j = 0; j < nodeRef.size(); j++)
					{
						if (entityRef[i]->parent == nodeRef[j])
						{
							f.write((const char*) &j, 2);
						}
					}
					
				} else {
					
					//f.write((const char*) -2, 2);
					f.put((const char) -2);
					// -2 is essentially a null pointer
					// because -1 is "move on" for the loader
				}
				
			}
		}
		
		/* ********************************************************************************
		 WRITE GRIDS
		 ******************************************************************************* */
		if (gridRef.size() > 0)
		{
			for (int i = 0; i < gridRef.size(); i++)
			{
				// put it in!
				//f.write((const char*) 2, 2);
				f.put((const char) 2);
				// 2 = grid
				
				// what number node is this?
				f.write((const char*) &i, 2);
				
				// size of the grid entry
				ui16 sizeofthing = gridRef[i]->name.length() + 30 + ((gridRef[i]->getX() * gridRef[i]->getY()) * 2);
				f.write((const char*)&sizeofthing, 2);
				
				// name and size of name
				sizeofthing = gridRef[i]->name.length();
				f.write((const char*)&sizeofthing, 2);
				f << gridRef[i]->name.c_str();
				
				// render object
				// currently only supports the sprite class
				//f.write((const char*) 0, 2);
				f.put((const char) 0);
				
				// but is it drawable?
				f.write((const char*) &gridRef[i]->draw, 1);
				
				// color
				// we don't need it for the whole grid
				
				// write grid size
				ui16 tempint = gridRef[i]->getX();
				f.write(reinterpret_cast<const char*>(&tempint), 2);
				tempint = gridRef[i]->getY();
				f.write(reinterpret_cast<const char*>(&tempint), 2);
				
				// go through the attributes of the tiles
				for (int j = 0; j < gridRef[i]->getX(); j++)
				{
					for (int k = 0; k < gridRef[i]->getY(); k++)
					{
						static bool tempbool = gridRef[i]->getTile(j, k)->getSolid();
						f.write((const char*) &tempbool, 1);
						
						for (int l = 0; l < textureRef.size(); l++)
						{
							if (gridRef[i]->getTile(j, k)->getTex() == textureRef[l])
							{
								f.write((const char*) &l, 2);
								break;
								// loop-ception!
							}
						}
					}
				}
				
				// grid position
				f.write((const char*) &gridRef[i]->pos.x, 4);
				f.write((const char*) &gridRef[i]->pos.y, 4);
				f.write((const char*) &gridRef[i]->pos.z, 4);
				
				// grid rotation
				f.write((const char*) &gridRef[i]->rot.x, 4);
				f.write((const char*) &gridRef[i]->rot.y, 4);
				f.write((const char*) &gridRef[i]->rot.z, 4);
				
				// grid scale
				f.write((const char*) &gridRef[i]->scl.x, 4);
				f.write((const char*) &gridRef[i]->scl.y, 4);
				f.write((const char*) &gridRef[i]->scl.z, 4);
				
				// find the parent!
				if (gridRef[i]->parent != nullptr)
				{
					for (int j = 0; j < nodeRef.size(); j++)
					{
						if (gridRef[i]->parent == nodeRef[j])
						{
							f.write((const char*) &j, 2);
						}
					}
					
				} else {
					
					//f.write((const char*) -2, 2);
					f.put((const char) -2);
					// -2 is essentially a null pointer
					// because -1 is "move on" for the loader
				}
			}
		}
		
		f.close();
		return 0;
	}
	
	int scene::readFromFile(std::string file)
	{
		std::string wFileExt = file;
		if (!wFileExt.find_last_of("."))
		{
			// no file extention. put one on
			wFileExt.append(".sinncascene");
		}
		
		std::ifstream f(wFileExt, std::ios::binary);
		if (!f.good())
		{
			printf("File stream is not good.");
			return -1;
			
		} else if (!f.is_open())
		{
			printf("Could not open %s", wFileExt.c_str());
			return -1;
		}
		/*
		std::streampos pos = f.tellg();
		char* buffer = (char*)Heap->allocate(pos, __alignof(char*));
		
		// read the file
		f.seekg(0, std::ios::beg);
		f.read(buffer, pos);
		f.close();
		// now we're done with the file and we can read from the buffer
		*/
		char buffer[8];
		ui16 secondaryHeader = 0;
		f.read(buffer, 8);
		f.read((char*)&secondaryHeader, 2);
		
		if (!strcmp(buffer, "sinnca_") && secondaryHeader != 18420)
		{
			f.close();
			printf("This is not a valid sinnca scene file");
			return -1;
		}
		
		ui32 noOfAssets = 0;
		f.read((char*)&noOfAssets, 4);
		ui32 noOfNodes = 0;
		f.read((char*)&noOfNodes, 4);
		
		for (int i = 0; i < noOfAssets; i++)
		{
			int obType = 0, numAsset = 0, assetSize = 0;
			
			f.read((char*)&obType, 1);
			f.read((char*)&numAsset, 2);
			f.read((char*)&assetSize, 2);
			
			switch (obType)
			{
				case 1:
				{
					// It's an image!
					int nameSize = 0;
					f.read((char*)&nameSize, 2);
					
					std::string thingName;
					for (int j = 0; j < nameSize; j++)
					{
						static char c;
						f.read(&c, 1);
						thingName.append(&c);
					}
					
					
					// how long is the path?
					int pathSize = 0;
					f.read((char*)&pathSize, 2);
					// and the path itself
					std::string pathName;
					for (int j = 0; j < pathSize; j++)
					{
						static char c;
						f.read(&c, 1);
						pathName.append(&c);
					}
					
					
					ui8 blendMode = 0;
					f.read((char*)&blendMode, 1);
					
					
					image* im = createImage(thingName);
					im->load(pathName, (texBlend)blendMode);
					Tree->currentScene->imageRef.push_back(im);
					
					break;
				}
				case 2:
				{
					// It's a texture!
					// how long is the name?
					int nameSize = 0;
					f.read((char*)&nameSize, 2);
					// and the name itself
					std::string thingName;
					for (int j = 0; j < nameSize; j++)
					{
						static char c;
						f.read(&c, 1);
						thingName.append(&c);
					}
					
					int imageLink = 0;
					f.read((char*)&imageLink, 2);
					
					// in-image offset
					int xOff = 0, yOff = 0;
					f.read((char*)&xOff, 2);
					f.read((char*)&yOff, 2);
					
					// texture dimentions
					int xSize = 0, ySize = 0;
					f.read((char*)&xSize, 2);
					f.read((char*)&ySize, 2);
					
					// base color
					ui8 col[4] = {0, 0, 0, 0};
					f.read((char*)&col[0], 1);
					f.read((char*)&col[1], 1);
					f.read((char*)&col[2], 1);
					f.read((char*)&col[3], 1);
					
					texture* tx = createTexture(thingName);
					tx->setSource(Tree->currentScene->imageRef[imageLink]);
					tx->setOffset(xOff, yOff);
					tx->setSize(xSize, ySize);
					
					tx->r = col[0];
					tx->g = col[1];
					tx->b = col[2];
					tx->a = col[3];
					
					break;
				}
				default:
				{
					int nameSize = 0;
					f.read((char*)&nameSize, 2);
					
					std::string thingName;
					f.read((char*)&thingName, nameSize);
					printf("Object '%s' was not recognized by this version of the engine. A newer version may be required or there was a mis-write when the file was created", thingName.c_str());
					
					break;
				}
			}
		}
		
		int tempint = 0;
		f.read((char*)&tempint, 1);
		
		for (int i = 0; i < noOfNodes; i++)
		{
			int obType = 0, numNode = 0, nodeSize = 0;
			
			f.read((char*)&obType, 1);
			f.read((char*)&numNode, 2);
			f.read((char*)&nodeSize, 2);
			
			switch (obType)
			{
				case 1:
				{
					// It's an entity!
					int nameSize = 0;
					f.read((char*)&nameSize, 2);
					
					std::string thingName;
					for (int j = 0; j < nameSize; j++)
					{
						static char c;
						f.read(&c, 1);
						thingName.append(&c);
					}
					
					// this space reserved for renderObj reading
					int renderLink = 0;
					f.read((char*)&renderLink, 1);
					
					// will this entity draw by default?
					bool willDraw = true;
					f.read((char*)&willDraw, 1);
					
					// Which texture will we use?
					int textureLink = 0;
					f.read((char*)&textureLink, 2);
					
					float position[9] = {0, 0, 0, 0, 0, 0, 0, 0, 0};
					for (int j = 0; j < 9 ; j++)
					{
						f.read((char*)&position[j], sizeof(float));
						//position[j] = endian32bit(position[j]);
					}
					
					
					int parentLink = -2;
					f.read((char*)&parentLink, 1);
					
					entity* en = createEntity(thingName);
					
					en->draw = willDraw;
					
					en->col = textureRef[textureLink];
					/*
					en->pos.x = position[0];
					en->pos.y = position[1];
					en->pos.z = position[2];
					en->rot.x = position[3];
					en->rot.y = position[4];
					en->rot.z = position[5];
					en->scl.x = position[6];
					en->scl.y = position[7];
					en->scl.z = position[8];
					*/
					if (parentLink == -2 || parentLink < 0)
					{
						en->parent = this;
						
					} else {
						
						en->parent = nodeRef[parentLink];
					}
					
					break;
				}
				
				case 2:
				{
					
					break;
				}
				default:
				{
					break;
				}
			}
		}
		
		f.close();
		return 0;
	}
	

I know there are some weird characters being written, but they were for safeguards I thought I needed at the time, and I really thought I was reading the file correctly.

Share this post


Link to post
Share on other sites
The spot that achild pointed out looks like the culprit to me as well. Any time you conditionally write to the file, you need to make sure the reader also follows the exact same condition check.

Sometimes this means writing a "is the next section present ? 1 : 0" byte into the file. More advanced formats include things like "the next section is type X, and it is length Y" so that a reader that doesn't know how to read type X (or perhaps one which doesn't care and just wants to skip to a section that it's currently interested in reading) can just skip over Y bytes and try the next section.

A pattern that comes up a lot is:
 
// Writing
WriteByte(obj != null ? 1 : 0);
if (obj != null)
{
   WriteObjectFields(obj);
}

// Reading
byte objectPresent = ReadByte();
if (objectPresent)
{
    obj = new Object();
    ReadObjectFields(obj);
}
else
{
    obj = null;
}
Edited by Nypyren

Share this post


Link to post
Share on other sites

Got it!

 

achild, you were right. The texture reference wasn't being written because it was looking at the wrong reference container, and now it's fixed!

 

 

Thank you everyone!

Share this post


Link to post
Share on other sites

Even if you're now looking at the correct container, all these "if's" look like a horrible idea. At this point your code should abort and return an error, not happily continue on to create a corrupt and broken file, so you can then spend days debugging your file reading function or wonder about endian issues.

 

Alternatively, at least write some kind of marker that will tell your reading function if a block exists or not.

Share this post


Link to post
Share on other sites

Even if you're now looking at the correct container, all these "if's" look like a horrible idea. At this point your code should abort and return an error, not happily continue on to create a corrupt and broken file, so you can then spend days debugging your file reading function or wonder about endian issues.

 

Alternatively, at least write some kind of marker that will tell your reading function if a block exists or not.

 

Indeed. Containerized formats and markup languages are wonderful that way. Yes, they add some overhead, but when your file formats are in flux it is good to include all kinds of details in the file rather than only writing the raw data.

 

Even something as simple as key/size/value combinations can help with surviving data migration as fields are added and removed over time. If you use them serialization can take a little bit longer, but when you put out an expansion pack that introduces a new variable it means you can simply fill in new defaults during load, ignore any now-unused values, and silently update to a new version when they next save.

 

If you only have a block of values you still need some metadata, generally at least a version number and probably also the length of the data block.

Share this post


Link to post
Share on other sites

This topic is 1183 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this