Jump to content
  • Advertisement
Sign in to follow this  
rXpSwiss

[Fixed]Image parsing : Read all file or x bytes at a time ?

This topic is 1430 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello,

 

I am parsing TGA files for my textures and everything works fine.

The issue is when I get to 4096*4096 or 2048*2048 resolution on those images it is very very very slow !

I went into the code and optimized the memory allocation (now there is no waste) but the loading as fast at it can be.

 

This is my question : Should I read all the file at once in a char array and then parse the array (how can I do it when the TGA is compressed ?) ? Because as of yet I read the header at once and then when it comes to the image's data I read pixel by pixel, I guess this is what makes it slow...

 

I don't read all the file at once because if there is a lot (lot) of elements to load it would take a lot of memory (at worst twice).

 

Just a simple question smile.png

 

rXp

Edited by rXpSwiss

Share this post


Link to post
Share on other sites
Advertisement

I am parsing TGA files ...

 

As the word "parse" applies to analysis of text, it's not clear what operation you're trying to perform. Are you storing text in an image file? Creating image fonts?

 

Without knowing what you're doing with the data, helping you determine how to handle it is difficult at best.

 

With regard to reading data, disk I/O is much slower than memory access, so load as much as you can afford with respect to memory. I.e., get the filesize, allocate memory and read it in.

 

 

 


I read pixel by pixel

 

As mentioned, load the file (header and data) all at once, if possible. Definitely NOT pixel by pixel.

 

You can then cast a pointer to the header data as "struct TGAheader*" or similar to access the specification.

 

Help others to help you by describing what operation you're making on the data and further help will be forthcoming.

 

EDIT: With regard to compression, IIRC the header info is never compressed so you can read it as loaded. With regard to image data, allocate a second block of memory the size of the final data required. Loop through the compressed data (RLE IIRC), expanding it into the second block.

Edited by Buckeye

Share this post


Link to post
Share on other sites

Sorry about the "parsing", I am reading a TGA file (exported from gimp or photoshop).

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

What I do now : I get the header and prepare a unsigned char vector for X pixels. I load the data image into a temporary buffer and go trough that buffer to get the data. It is faster but on one PC takes 30sec to load a 4096*4096 file and on the other is takes 1 sec. The difference is that on the second PC there is 2 HDD and the files are on the second unused HDD.

Could it slow the process that much ?

Share this post


Link to post
Share on other sites

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

Share this post


Link to post
Share on other sites

Even if it were an uncompressed 24-bit bitmap image, it shouldn't take anywhere near 30 seconds to load a 4096x4096 image on any remotely modern computer.  Can you post the code that loads the image data from file?

Share this post


Link to post
Share on other sites

 

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

 

Really ? Because I based myself on the complete specifications and there is the color map, the ID, the extra information that could be added by the person making the image.

 

And yes this is the code, since I come from java I am still learning the good practice of C++. DO not hesitate to correct me on every little details. Always happy to learn smile.png

void OBJParser::parseTGA(const char* file, TGAData* const tga)
{
	boost::filesystem::path path(file);
	if (boost::filesystem::exists(path))
	{
		char header[18];
		unsigned short colorMapLength = 0;
		std::ifstream infile;
		infile.open(file, std::ifstream::binary | std::ifstream::in);
		//read header
		infile.read(header, 18);

		//read id length
		if (header[0] > 0)tga->data.imageID = (char*)malloc(header[0]);

		//read colormap type
		tga->header.isColorMap = (header[1] == 1);

		//read image type
		if (header[2] == (char)0)tga->header.imageType = TGAData::NONE;
		else if (header[2] == (char)1)tga->header.imageType = TGAData::UNCOMPRESSED_COLOR_MAP;
		else if (header[2] == (char)2)tga->header.imageType = TGAData::UNCOMPRESSED_TRUE_COLOR;
		else if (header[2] == (char)3)tga->header.imageType = TGAData::UCOMPRESSED_BLACK_AND_WHITE;
		else if (header[2] == (char)9)tga->header.imageType = TGAData::RUN_LENGTH_COLOR_MAP;
		else if (header[2] == (char)10)tga->header.imageType = TGAData::RUN_LENGTH_TRUE_COLOR;
		else tga->header.imageType = TGAData::NONE;

		//if there is a color map
		if (tga->header.isColorMap)
		{
			//colormap length to know how to skip it
		}

		//image specification
		char size[2];
		tga->header.xOrigin = *(static_cast<unsigned short*>(static_cast<void*>(&header[8]))); //read x origin
		tga->header.yOrigin = *(static_cast<unsigned short*>(static_cast<void*>(&header[10])));//read y origin
		size[0] = header[12];
		size[1] = header[13];
		tga->header.width = *(static_cast<unsigned short*>(static_cast<void*>(&header[12])));//read width
		tga->header.height = *(static_cast<unsigned short*>(static_cast<void*>(&header[14])));//read height
		tga->header.pixelDepth = header[16];
		tga->header.imageDescriptor = header[17];


		//read image
		//format : NUMBER[R G B  & A ssi pixeldepth > 0]
		const size_t pixelDepth = tga->header.pixelDepth / 8;
		const size_t totalPixel = tga->header.height*tga->header.width;
		tga->data.parsedImageData.reserve(totalPixel * 4);
		unsigned int counter = 0;
		char* chunkSize = (char*)malloc(1);
		char* rgba = (char*)malloc(pixelDepth);
		unsigned int bufferSize = boost::filesystem::file_size(file) - 18;
		char* bufferRestOfFile = (char*)malloc(bufferSize);//minus header
		infile.read(bufferRestOfFile, bufferSize);
		unsigned int index = 0;
		switch (tga->header.imageType)
		{
		case TGAData::RUN_LENGTH_TRUE_COLOR:
		{
			const char maskMSBToZero = 0x7F; //01111111
			while (counter < totalPixel)
			{
				unsigned char unsignedChunkSize = *static_cast<unsigned char*>(static_cast<void *>(&bufferRestOfFile[index]));
				index++;
				unsigned char nb = 0;

				if (unsignedChunkSize < 128) //raw chunk -> X next pixel
				{
					nb = unsignedChunkSize + 1;//number of same pixels in a row
					for (short i = 0; i<nb; i++)
					{
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
						if (pixelDepth > 3)
						{
							tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
							index += 4;
						}
						else
						{
							tga->data.parsedImageData.push_back(255);
							index += 3;
						}

						counter++;
					}
				}
				else //X * COLOR
				{
					nb = (unsignedChunkSize & maskMSBToZero) + 1;
					for (short i = 0; i<nb; i++)
					{
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
						tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
						if (pixelDepth > 3)
							tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
						else
							tga->data.parsedImageData.push_back(255);

						counter++;
					}
					if (pixelDepth > 3)	index += 4;
					else index += 3;
				}
			}

		}
			break;
		case TGAData::UNCOMPRESSED_TRUE_COLOR:
		{
			while (counter<totalPixel)
			{
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
				if (pixelDepth > 3)
				{
					tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
					index += 4;
				}
				else
				{
					tga->data.parsedImageData.push_back(255);
					index += 3;
				}

				counter++;
			}
		}
			break;
		case TGAData::UCOMPRESSED_BLACK_AND_WHITE: //not tested at all
		{
			while (counter < totalPixel)
			{
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
				tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);

				tga->data.parsedImageData.push_back(255);
				counter++;
				index++;
			}
		}
			break;
		}
		free(rgba);
		free(chunkSize);
		infile.close();
	}
}

Maybe you wonder why I use a vector and not an array, well it is because a lot of people told me to so I wouldn't need to worry too much about the memory allocation.

I don't think free is a big deal but at first I wanted something that worked, not I want something that work well. (do then optimize)

Edited by rXpSwiss

Share this post


Link to post
Share on other sites

 

 

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

 

Really ? Because I based myself on the complete specifications and there is the color map, the ID, the extra information that could be added by the person making the image.

If you don't consider those as part of the header, then just parse their size and subtract from the total. It doesn't change the idea that you derive the size of the remaining image data from the file size.

Share this post


Link to post
Share on other sites
infile.read(bufferRestOfFile, sizeof(bufferRestOfFile));
I don't think this line does what you think it does. sizeof gives the static size of the pointer, not the dynamic size of what it points to.

Share this post


Link to post
Share on other sites
infile.read(bufferRestOfFile, sizeof(bufferRestOfFile));
I don't think this line does what you think it does. sizeof gives the static size of the pointer, not the dynamic size of what it points to.

 

Sorry that was an old version. I actually use the size of the fil minus the 18byte of header now. It works fine. (by that I mean that my texture works)

 

The post has been updated with the correct code.

Edited by rXpSwiss

Share this post


Link to post
Share on other sites

I prefer to memory map the file and skip the memory allocation part entirely. TGA of course requires some decompression code, so you will have to allocate a destination array. If your slowdown is in debug builds only, then you may be running up against the debug time sanity checks in std::vector.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!