Writing char arrays (binary)

Started by
21 comments, last by Alberth 6 years, 5 months ago

Hi all,

For practice I'm trying to write 3D mesh data to a binary format. To eventually also be able to read meshes from binary files (provided by the asset pipeline).

I understand how to write floats, vectors of floats, ints etc by using a write((char*)myVar, sizeof(type) to the file, for which I'm using std::ofstream.

My goal is to write a 3D vector or other type (struct) to the file and be able to read it back the same way. From what I understand, I need to create function that returns a char array combining all members of the struct.

So in short; how can I create this char array, in a "safe" way?

So far I've come up with this approach:

- calculate number of needed bytes/chars, i.e 12 bytes for a 3D vector with 3 floats

- char result[12]

- cast the 3 individual floats to char[4]'s and use strcpy to result for the 1st and strcat for the others

Is this the way to go or would you suggest other approaches?

question 2; How would I read/convert the char array back to the 3 floats when reading it from the binary file?

Any input is appreciated.

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

Advertisement

I believe, for a POD struct, you can just call write( (myType*)myArray, sizeof (myType) * SizeOfArray(myArray)).

You can read the data back in a similar way.

I'm not even sure what's your problem there. Let's start from the basics: the only thing that truly exist is memory and bits. As long as you're going with 'basic' types such as floats the following will do: 


ubyte blobby[12];
memcpy(blobby, src, sizeof(blobby));
float *bruh = (float*)blobby;

Hoping `blobby` to be properly aligned.

First you have to ensure your `int` is the same as the serialized `int` and that's why you have `__int32` or `std::int64_t`. And of course you have endianess problems but let's assume you'll be loading on 'coherent' machines.

When `struct` enters the picture `memcpy` goes out due to possibility of different compiler packing.

There are various options for (de)serialization, I'd suggest pupping all the things.

That or google protobuffers but in my experience they're not quite convenient for game purposes due to their everything can be optional nonsense.

 

 

Previously "Krohm"

Thanks guys.

I've been playing around using this input. Unfortunately writing the struct or array (std::vector) at once, doesn't work by trying to cast to (myType*), it does work when I cast it to (char*). See the code samples below, tested all 3 positive.

Would I have any risks on this approach, depending on platform? (size of float etc.).
I'll make sure the struct I use as a POD struct, which is the case in the example below.


void ReadWriteStruct()
{
	std::ofstream writeFile;
	writeFile.open("data.bin", std::ios::out, std::ios::binary);
	writeFile.write((char*)&srcVector, sizeof(VECTOR3));
	writeFile.close();

	MessageBox(NULL, L"Vector written to binary file - full struct!", L"Done", MB_OK);

	std::ifstream readFile;
	readFile.open("data.bin", std::ios::in, std::ios::binary);

	VECTOR3 myReadVec3;
	readFile.read((char*)&myReadVec3, sizeof(VECTOR3));
	readFile.close();

	char tempText[100];
	sprintf_s(tempText, "Read: %f, %f, %f\n", myReadVec3.x, myReadVec3.y, myReadVec3.z);
	OutputDebugStringA(tempText);
}

void ReadWriteArray()
{
	std::vector<VECTOR3> myVectors(2);
	myVectors[0].x = 0.2f;
	myVectors[0].y = 0.7f;
	myVectors[0].z = 0.95f;
	myVectors[1].x = 5.2f;
	myVectors[1].y = 4.7f;
	myVectors[1].z = 7.75f;

	std::ofstream writeFile;
	writeFile.open("data.bin", std::ios::out, std::ios::binary);
	writeFile.write((char*)&myVectors[0], myVectors.size() * sizeof(VECTOR3));
	writeFile.close();

	MessageBox(NULL, L"Vector written to binary file - full struct!", L"Done", MB_OK);

	std::ifstream readFile;
	readFile.open("data.bin", std::ios::in, std::ios::binary);

	std::vector<VECTOR3> readVectors(2);
	readFile.read((char*)&readVectors[0], sizeof(VECTOR3));
	readFile.read((char*)&readVectors[1], sizeof(VECTOR3));
	readFile.close();

	char tempText[100];
	sprintf_s(tempText, "Read 1: %f, %f, %f\n", readVectors[0].x, readVectors[0].y, readVectors[0].z);
	OutputDebugStringA(tempText);
	sprintf_s(tempText, "Read 2: %f, %f, %f\n", readVectors[1].x, readVectors[1].y, readVectors[1].z);
	OutputDebugStringA(tempText);
}

 

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

What's with the downvote? Can someone enlighten me so I know how to do it right/ better.

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

17 hours ago, cozzie said:

Thanks guys.

I've been playing around using this input. Unfortunately writing the struct or array (std::vector) at once, doesn't work by trying to cast to (myType*), it does work when I cast it to (char*). See the code samples below, tested all 3 positive.

Would I have any risks on this approach, depending on platform? (size of float etc.).
I'll make sure the struct I use as a POD struct, which is the case in the example below.



void ReadWriteStruct()
{
	std::ofstream writeFile;
	writeFile.open("data.bin", std::ios::out, std::ios::binary);
	writeFile.write((char*)&srcVector, sizeof(VECTOR3));
	writeFile.close();

	MessageBox(NULL, L"Vector written to binary file - full struct!", L"Done", MB_OK);

	std::ifstream readFile;
	readFile.open("data.bin", std::ios::in, std::ios::binary);

	VECTOR3 myReadVec3;
	readFile.read((char*)&myReadVec3, sizeof(VECTOR3));
	readFile.close();

	char tempText[100];
	sprintf_s(tempText, "Read: %f, %f, %f\n", myReadVec3.x, myReadVec3.y, myReadVec3.z);
	OutputDebugStringA(tempText);
}

void ReadWriteArray()
{
	std::vector<VECTOR3> myVectors(2);
	myVectors[0].x = 0.2f;
	myVectors[0].y = 0.7f;
	myVectors[0].z = 0.95f;
	myVectors[1].x = 5.2f;
	myVectors[1].y = 4.7f;
	myVectors[1].z = 7.75f;

	std::ofstream writeFile;
	writeFile.open("data.bin", std::ios::out, std::ios::binary);
	writeFile.write((char*)&myVectors[0], myVectors.size() * sizeof(VECTOR3));
	writeFile.close();

	MessageBox(NULL, L"Vector written to binary file - full struct!", L"Done", MB_OK);

	std::ifstream readFile;
	readFile.open("data.bin", std::ios::in, std::ios::binary);

	std::vector<VECTOR3> readVectors(2);
	readFile.read((char*)&readVectors[0], sizeof(VECTOR3));
	readFile.read((char*)&readVectors[1], sizeof(VECTOR3));
	readFile.close();

	char tempText[100];
	sprintf_s(tempText, "Read 1: %f, %f, %f\n", readVectors[0].x, readVectors[0].y, readVectors[0].z);
	OutputDebugStringA(tempText);
	sprintf_s(tempText, "Read 2: %f, %f, %f\n", readVectors[1].x, readVectors[1].y, readVectors[1].z);
	OutputDebugStringA(tempText);
}

 

I made a mistake, it should have been char* not myType*. I don't see a structure in your post but usually a vector struct is a POD.

You can read the vector array in a single call as well.

Krohm already posted the problems you may face.

Thanks. The VECTOR3 is my struct, so it's working for both an individual struct as a std::vector of them. Thanks

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

On 10/11/2017 at 12:57 AM, Krohm said:

There are various options for (de)serialization, I'd suggest pupping all the things.

I second this approach.

Break it down so that you're only serializing fundamental types. This allows you to avoid all issues with alignment and padding, properly handle endian-ness, and trivially support composites of serializeable types.

There's nothing you gain from trying to read/write entire buffers of typed data at once, other than the potential for hard-to-find bugs.

A little bit off topic but i always first writr then length of char array and then i writr char array anyway

Memcpy is your pal, you just memcpy(&floatvar, &chartable[index],3);

 

Something kike this you copy some amount memory to float pointer

Do you mean first writing everything to a char array (aka buffer) and then write that buffer to file?

That could work, but then I have to calculate the index manually, which could be a risk if some type has a different bytesize on another problem (writing per variable and using sizeof, I think prevents this). In your example, shouldn't 3 be 4 (bytes for a float)?

But perhaps there's a way to use a buffer while keeping this in mind (or don't manually calculate the index of each variable).

Crealysm game & engine development: http://www.crealysm.com

Looking for a passionate, disciplined and structured producer? PM me

This topic is closed to new replies.

Advertisement