Sign in to follow this  
Vortez

512kB array won't compile, "internal heap limit reached"...

Recommended Posts

Hi, i've just ran into a little snag while trying to compile a global variable array, compiler give me this error message: "fatal error C1076: compiler limit : internal heap limit reached; use /Zm to specify a higher limit". So, i know i could use this switch to potentially solve the problem, but i though there was only a size limit for large array allocated on the stack. The array is big, but not that big, 512k only. Maybe it's the way i stored it, in a text file, just like this:

[code]
BYTE Cipher[CIPHER_LENGTH] = {
0x8D, 0x97, 0xD2, 0x83, 0xCD, 0x8F, 0xC0, 0xF9, 0x6C, 0xD5, 0xFE, 0x1F, 0x34, 0xD6, 0x11, 0x3A,
0xCE, 0x9C, 0xEC, 0xF2, 0xB5, 0x52, 0x20, 0x77, 0x63, 0xA1, 0x62, 0x90, 0x1A, 0x9F, 0x7F, 0xA9,
0x6A, 0xC5, 0x30, 0x26, 0x81, 0x43, 0xAD, 0xA5, 0xBA, 0x7B, 0x92, 0x1B, 0x3A, 0xEB, 0x48, 0x99,
// ... and so on
[/code]

[code]
#include "Encryptor.h"
#include "Cipher.h"

//-----------------------------------------------------------------------------
// Constructor
//-----------------------------------------------------------------------------
CEncryptor::CEncryptor()
{
LoadCipher();
}

//-----------------------------------------------------------------------------
// Load the cipher data from memory
//-----------------------------------------------------------------------------
void CEncryptor::LoadCipher()
{
memcpy(&Cipher[0], CIPHER_LENGTH);
}
...
[/code]

I was loading it from file before and it worked because i allocated the memory dynamically, but since this array will be used to encrypt some data, i don't want it stored in a binary file for obvious reason. That's why i tryed this method.

Is there a way around this?

Share this post


Link to post
Share on other sites
512k is a pretty big heap. From what I remember, it's smaller on a lot of hardware. Well, it's not really going to matter. I mean, you don't want to store it in a binary file, but the thing is that even if you increase the stack size and define it in the executable, it won't be that hard to extract it from the executable. In the exe, that data is going to be somewhere in there, defined exactly as you put it in, otherwise it wouldn't work.

So there's little difference between having it in a file, or having it defined in the exe. I don't know exactly what the array is. Is it the data that you want to encrypt, or is it data that you use to encrypt other data? Is it possible to calculate it on the fly? Even just up front so you can allocate onto the heap and calculate it all at once?

Share this post


Link to post
Share on other sites
Humm, forget the code above, seem like the problem is that the text file is too big (3-4 Mo). Still, im not sure how i could compile this 512k bytes directly into the executable. Should i use a resources file?

Endar: Well i dont need my application to be incredibly secure, but i just find that saving the decryption key in a file is not very good. Thing is, i created this key using CryptGenRandom, wich is said to be very good compared to c++ RNG... Of course i could just use a seed and generate it at run time, i just feeled like that was a better way.

Share this post


Link to post
Share on other sites
Since in LoadFile, you are copying the data to some other memory, I guess you could just spread the data across multiple arrays, include them all, then just concatenate them into &cipher[0]. (i.e. load the second array into &cipher[first_array_size]), and soforth for the other arrays.

edit: now that I read it, your memcpy is invalid.

Share this post


Link to post
Share on other sites
There are a few things I don't quite understand with your code:

Though I'm not sure if that affects this particular compiler error (don't know why a large array should consume any more resources at compile time than a small one, and under gcc it surely doesn't according to [font="Courier New"]-fmem-report[/font]), I would try making that constant data [font="Courier New"]const[/font]. if nothing else, it's correct to have [font="Courier New"]const [/font]data [font="Courier New"]const[/font], and it will save some executable size because the compiler can just dump the whole contents into [font="Courier New"].data[/font] (this might incidentially fix the error too, who knows).

What does [font="Courier New"]memcpy(&Cipher[0], CIPHER_LENGTH);[/font] do? Does that even compile?

Assuming the above [font="Courier New"]memcpy [/font]does compile and work (which I doubt), what avail? You could read the constant array directly or via a reference.

Since the array is called [font="Courier New"]Cipher [/font]and the class is called [font="Courier New"]CEncryptor[/font], I assume this is an attempt to obfuscate an encryption algorithm (maybe for copy protection?). You are aware that a cipher needs not be secret (unless it is an abysmally bad one) to function, and embedding it in such a way adds zero to security anyway?

Share this post


Link to post
Share on other sites
How is this encrypted data going to be used? Embedding a symmetric encryption key in the source buys you very little security, as any technically competent individual trying to hack your program/protocol can probably figure this out.

Encryption keys are usually expressed in terms of bits, not bytes. Why does your cipher require half a million bits? That seems excessive to me.

Share this post


Link to post
Share on other sites
He's probably using this data as a [url="http://en.wikipedia.org/wiki/One-time_pad"]one-time pad[/url].

I still don't understand the original problem. If I make a program with a global array of size 512Kbytes, I don't think I run into any trouble. What compiler/platform is this on? And can you post an actual compilable program that shows the problem? You can leave out the exact content of your array.

Share this post


Link to post
Share on other sites
[quote name='rip-off' timestamp='1310395910' post='4833750']
I would be very suspicious of this being used as a one time pad. I would need to know how it was being used before commenting.
[/quote]

Oh, I didn't mean to say that what he is doing makes sense. I was just offering a guess.

Share this post


Link to post
Share on other sites
Hi, thx for your answer. As you noticed, i was dead tired yesterday when i posted my code. Also since the compiler was getting stuck because of the large size of the text file i included, i could't see that memcpy was missing an argument :P. Thing is, i dont want the encryptor to be too slow since i really need speed in this case, it's used to encryp data of a program working similar to VNC, to remote control computer at a distance. Since most of the data is compressed either using an mp1 encoder or using zlib, it's already kinda hard for anyone intercepting those packet to decode them to image, or take control of the machine being controled, since i have the source code and i myself find my own code to be confusing ^^. Also it's only allow connection to one machine at the time (one server, one client). So i though about using encryption as an additionnal feature, although i really doubt someone will ever try to crack this (im not intending to distribute it execpt for some people i know). Well i can always post my code so that you can see what im trying to do, since i need speed, im only using a simple xor encryption atm, no key just simple xor with the cipher. As i said all i want to do is to embeed the cipher into the executable, i dont care much if it's not extremely secure for now, i can always encode it then decode it at run time in the constructor or something like that later.

[code]
#ifndef _CENCRYPTOR_H_
#define _CENCRYPTOR_H_
#ifdef __cplusplus
//----------------------------------------------------------------------//
#include "Windows.h"
#include "stdio.h"
//----------------------------------------------------------------------//

#define CIPHER_LENGTH 524288

class CEncryptor {
public:
CEncryptor();
private:
BYTE Cipher[CIPHER_LENGTH];
//void LoadCipher();
bool LoadCipherFromFile(char *fname);
void XCrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset);
public:
void Encrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset = 0);
void Decrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset = 0);
};
//----------------------------------------------------------------------//
//----------------------------------------------------------------------//
#endif
#endif //_CENCRYPTOR_H_
[/code]

[code]
#include "Encryptor.h"

//-----------------------------------------------------------------------------
// Constructor
//-----------------------------------------------------------------------------
CEncryptor::CEncryptor()
{
//LoadCipher();
LoadCipherFromFile("Entropy.rnd");
}

//-----------------------------------------------------------------------------
// Load the cipher data from memory
//-----------------------------------------------------------------------------
/*void CEncryptor::LoadCipher()
{
memcpy(&Cipher[0], &MyCipher[0], CIPHER_LENGTH);
}*/

//-----------------------------------------------------------------------------
// Load the cipher data from a file
//-----------------------------------------------------------------------------
bool CEncryptor::LoadCipherFromFile(char *fname)
{
ZeroMemory(&Cipher[0], CIPHER_LENGTH);

FILE *f = fopen(fname, "rb");
if(f){
fread(&Cipher[0], 1, CIPHER_LENGTH, f);
fclose(f);
return true;
}

return false;
}

//-----------------------------------------------------------------------------
// XOR the given packet buffer data with the cipher
//-----------------------------------------------------------------------------
inline void CEncryptor::XCrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset)
{
DWORD Indx = Offset;

for(DWORD Cpt = 0; Cpt < BufSize; Cpt++){
if(Indx >= CIPHER_LENGTH)
Indx = Offset % CIPHER_LENGTH;

pBuf[Cpt] ^= Cipher[Indx++];
}
}

//-----------------------------------------------------------------------------
// Encrypt the given packet buffer
//-----------------------------------------------------------------------------
void CEncryptor::Encrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset)
{
XCrypt(pBuf, BufSize, Offset);
}

//-----------------------------------------------------------------------------
// Decrypt the given packet buffer
//-----------------------------------------------------------------------------
void CEncryptor::Decrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset)
{
XCrypt(pBuf, BufSize, Offset);
}
[/code]

Share this post


Link to post
Share on other sites
the problem was when im trying to include a file with the cipher in a text file instead of using a binary one, the file is too big and compiler complain. For example, the text file looked like this:

BYTE MyCipher[CIPHER_LENGTH] = {
0x8D, 0x97, 0xD2, 0x83, 0xCD, 0x8F, 0xC0, 0xF9, 0x6C, 0xD5, 0xFE, 0x1F, 0x34, 0xD6, 0x11, 0x3A,
0xCE, 0x9C, 0xEC, 0xF2, 0xB5, 0x52, 0x20, 0x77, 0x63, 0xA1, 0x62, 0x90, 0x1A, 0x9F, 0x7F, 0xA9,
0x6A, 0xC5, 0x30, 0x26, 0x81, 0x43, 0xAD, 0xA5, 0xBA, 0x7B, 0x92, 0x1B, 0x3A, 0xEB, 0x48, 0x99,
// ... and so on

i extracted this using the "Copy as Text" from an hexEditor, but the file make almost 4mo in text mode, instead of 512k in binary, i think that's why the compiler is complaining. What im trying to do is to get rid of the binary file and embeed the data into the executable itself. I think i should try using a ressource file instead.

Share this post


Link to post
Share on other sites
4MB source file and 512KB should not be too large for MSVC compiler.
At work we are using 22MB generated source code that includes several large (up to 2 MB) generated data tables. It compiles just fine without any /Zm switch.

Does your MyCipher array changes? try adding static const modifiers. Then include header with large array into your cpp file and use it directly without any memcpy.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this