# [Fixed]Image parsing : Read all file or x bytes at a time ?

This topic is 1508 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Hello,

I am parsing TGA files for my textures and everything works fine.

The issue is when I get to 4096*4096 or 2048*2048 resolution on those images it is very very very slow !

I went into the code and optimized the memory allocation (now there is no waste) but the loading as fast at it can be.

This is my question : Should I read all the file at once in a char array and then parse the array (how can I do it when the TGA is compressed ?) ? Because as of yet I read the header at once and then when it comes to the image's data I read pixel by pixel, I guess this is what makes it slow...

I don't read all the file at once because if there is a lot (lot) of elements to load it would take a lot of memory (at worst twice).

Just a simple question

rXp

Edited by rXpSwiss

##### Share on other sites

I am parsing TGA files ...

As the word "parse" applies to analysis of text, it's not clear what operation you're trying to perform. Are you storing text in an image file? Creating image fonts?

Without knowing what you're doing with the data, helping you determine how to handle it is difficult at best.

With regard to reading data, disk I/O is much slower than memory access, so load as much as you can afford with respect to memory. I.e., get the filesize, allocate memory and read it in.

As mentioned, load the file (header and data) all at once, if possible. Definitely NOT pixel by pixel.

You can then cast a pointer to the header data as "struct TGAheader*" or similar to access the specification.

Help others to help you by describing what operation you're making on the data and further help will be forthcoming.

EDIT: With regard to compression, IIRC the header info is never compressed so you can read it as loaded. With regard to image data, allocate a second block of memory the size of the final data required. Loop through the compressed data (RLE IIRC), expanding it into the second block.

Edited by Buckeye

##### Share on other sites

Sorry about the "parsing", I am reading a TGA file (exported from gimp or photoshop).

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

What I do now : I get the header and prepare a unsigned char vector for X pixels. I load the data image into a temporary buffer and go trough that buffer to get the data. It is faster but on one PC takes 30sec to load a 4096*4096 file and on the other is takes 1 sec. The difference is that on the second PC there is 2 HDD and the files are on the second unused HDD.

Could it slow the process that much ?

##### Share on other sites

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

##### Share on other sites

Even if it were an uncompressed 24-bit bitmap image, it shouldn't take anywhere near 30 seconds to load a 4096x4096 image on any remotely modern computer.  Can you post the code that loads the image data from file?

##### Share on other sites

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

Really ? Because I based myself on the complete specifications and there is the color map, the ID, the extra information that could be added by the person making the image.

And yes this is the code, since I come from java I am still learning the good practice of C++. DO not hesitate to correct me on every little details. Always happy to learn

void OBJParser::parseTGA(const char* file, TGAData* const tga)
{
boost::filesystem::path path(file);
if (boost::filesystem::exists(path))
{
unsigned short colorMapLength = 0;
std::ifstream infile;
infile.open(file, std::ifstream::binary | std::ifstream::in);

//if there is a color map
{
//colormap length to know how to skip it
}

//image specification
char size[2];

//format : NUMBER[R G B  & A ssi pixeldepth > 0]
const size_t pixelDepth = tga->header.pixelDepth / 8;
tga->data.parsedImageData.reserve(totalPixel * 4);
unsigned int counter = 0;
char* chunkSize = (char*)malloc(1);
char* rgba = (char*)malloc(pixelDepth);
unsigned int bufferSize = boost::filesystem::file_size(file) - 18;
unsigned int index = 0;
{
{
const char maskMSBToZero = 0x7F; //01111111
while (counter < totalPixel)
{
unsigned char unsignedChunkSize = *static_cast<unsigned char*>(static_cast<void *>(&bufferRestOfFile[index]));
index++;
unsigned char nb = 0;

if (unsignedChunkSize < 128) //raw chunk -> X next pixel
{
nb = unsignedChunkSize + 1;//number of same pixels in a row
for (short i = 0; i<nb; i++)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
if (pixelDepth > 3)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
index += 4;
}
else
{
tga->data.parsedImageData.push_back(255);
index += 3;
}

counter++;
}
}
else //X * COLOR
{
nb = (unsignedChunkSize & maskMSBToZero) + 1;
for (short i = 0; i<nb; i++)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
if (pixelDepth > 3)
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
else
tga->data.parsedImageData.push_back(255);

counter++;
}
if (pixelDepth > 3)	index += 4;
else index += 3;
}
}

}
break;
{
while (counter<totalPixel)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 2]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 1]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
if (pixelDepth > 3)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index + 3]);
index += 4;
}
else
{
tga->data.parsedImageData.push_back(255);
index += 3;
}

counter++;
}
}
break;
case TGAData::UCOMPRESSED_BLACK_AND_WHITE: //not tested at all
{
while (counter < totalPixel)
{
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);
tga->data.parsedImageData.push_back((unsigned char)bufferRestOfFile[index]);

tga->data.parsedImageData.push_back(255);
counter++;
index++;
}
}
break;
}
free(rgba);
free(chunkSize);
infile.close();
}
}


Maybe you wonder why I use a vector and not an array, well it is because a lot of people told me to so I wouldn't need to worry too much about the memory allocation.

I don't think free is a big deal but at first I wanted something that worked, not I want something that work well. (do then optimize)

Edited by rXpSwiss

##### Share on other sites

About the compression I know the header is never compressed but the header doesn't say how big is the "data image" part of the file.

The size of the image data is the size of the file minus the size of the header.

Really ? Because I based myself on the complete specifications and there is the color map, the ID, the extra information that could be added by the person making the image.

If you don't consider those as part of the header, then just parse their size and subtract from the total. It doesn't change the idea that you derive the size of the remaining image data from the file size.

##### Share on other sites
infile.read(bufferRestOfFile, sizeof(bufferRestOfFile));
I don't think this line does what you think it does. sizeof gives the static size of the pointer, not the dynamic size of what it points to.

##### Share on other sites
infile.read(bufferRestOfFile, sizeof(bufferRestOfFile));
I don't think this line does what you think it does. sizeof gives the static size of the pointer, not the dynamic size of what it points to.

Sorry that was an old version. I actually use the size of the fil minus the 18byte of header now. It works fine. (by that I mean that my texture works)

The post has been updated with the correct code.

Edited by rXpSwiss

##### Share on other sites

I prefer to memory map the file and skip the memory allocation part entirely. TGA of course requires some decompression code, so you will have to allocate a destination array. If your slowdown is in debug builds only, then you may be running up against the debug time sanity checks in std::vector.

1. 1
2. 2
3. 3
Rutin
13
4. 4
5. 5

• 26
• 11
• 9
• 9
• 11
• ### Forum Statistics

• Total Topics
633696
• Total Posts
3013401
×