512kB array won't compile, "internal heap limit reached"...

This topic is 2596 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

Recommended Posts

Hi, i've just ran into a little snag while trying to compile a global variable array, compiler give me this error message: "fatal error C1076: compiler limit : internal heap limit reached; use /Zm to specify a higher limit". So, i know i could use this switch to potentially solve the problem, but i though there was only a size limit for large array allocated on the stack. The array is big, but not that big, 512k only. Maybe it's the way i stored it, in a text file, just like this:

 BYTE Cipher[CIPHER_LENGTH] = { 0x8D, 0x97, 0xD2, 0x83, 0xCD, 0x8F, 0xC0, 0xF9, 0x6C, 0xD5, 0xFE, 0x1F, 0x34, 0xD6, 0x11, 0x3A, 0xCE, 0x9C, 0xEC, 0xF2, 0xB5, 0x52, 0x20, 0x77, 0x63, 0xA1, 0x62, 0x90, 0x1A, 0x9F, 0x7F, 0xA9, 0x6A, 0xC5, 0x30, 0x26, 0x81, 0x43, 0xAD, 0xA5, 0xBA, 0x7B, 0x92, 0x1B, 0x3A, 0xEB, 0x48, 0x99, // ... and so on 

 #include "Encryptor.h" #include "Cipher.h" //----------------------------------------------------------------------------- // Constructor //----------------------------------------------------------------------------- CEncryptor::CEncryptor() { LoadCipher(); } //----------------------------------------------------------------------------- // Load the cipher data from memory //----------------------------------------------------------------------------- void CEncryptor::LoadCipher() { memcpy(&Cipher[0], CIPHER_LENGTH); } ... 

I was loading it from file before and it worked because i allocated the memory dynamically, but since this array will be used to encrypt some data, i don't want it stored in a binary file for obvious reason. That's why i tryed this method.

Is there a way around this?

Share on other sites
512k is a pretty big heap. From what I remember, it's smaller on a lot of hardware. Well, it's not really going to matter. I mean, you don't want to store it in a binary file, but the thing is that even if you increase the stack size and define it in the executable, it won't be that hard to extract it from the executable. In the exe, that data is going to be somewhere in there, defined exactly as you put it in, otherwise it wouldn't work.

So there's little difference between having it in a file, or having it defined in the exe. I don't know exactly what the array is. Is it the data that you want to encrypt, or is it data that you use to encrypt other data? Is it possible to calculate it on the fly? Even just up front so you can allocate onto the heap and calculate it all at once?

Share on other sites
Humm, forget the code above, seem like the problem is that the text file is too big (3-4 Mo). Still, im not sure how i could compile this 512k bytes directly into the executable. Should i use a resources file?

Endar: Well i dont need my application to be incredibly secure, but i just find that saving the decryption key in a file is not very good. Thing is, i created this key using CryptGenRandom, wich is said to be very good compared to c++ RNG... Of course i could just use a seed and generate it at run time, i just feeled like that was a better way.

Share on other sites
Since in LoadFile, you are copying the data to some other memory, I guess you could just spread the data across multiple arrays, include them all, then just concatenate them into &cipher[0]. (i.e. load the second array into &cipher[first_array_size]), and soforth for the other arrays.

Share on other sites
There are a few things I don't quite understand with your code:

Though I'm not sure if that affects this particular compiler error (don't know why a large array should consume any more resources at compile time than a small one, and under gcc it surely doesn't according to [font="Courier New"]-fmem-report[/font]), I would try making that constant data [font="Courier New"]const[/font]. if nothing else, it's correct to have [font="Courier New"]const [/font]data [font="Courier New"]const[/font], and it will save some executable size because the compiler can just dump the whole contents into [font="Courier New"].data[/font] (this might incidentially fix the error too, who knows).

What does [font="Courier New"]memcpy(&Cipher[0], CIPHER_LENGTH);[/font] do? Does that even compile?

Assuming the above [font="Courier New"]memcpy [/font]does compile and work (which I doubt), what avail? You could read the constant array directly or via a reference.

Since the array is called [font="Courier New"]Cipher [/font]and the class is called [font="Courier New"]CEncryptor[/font], I assume this is an attempt to obfuscate an encryption algorithm (maybe for copy protection?). You are aware that a cipher needs not be secret (unless it is an abysmally bad one) to function, and embedding it in such a way adds zero to security anyway?

Share on other sites
How is this encrypted data going to be used? Embedding a symmetric encryption key in the source buys you very little security, as any technically competent individual trying to hack your program/protocol can probably figure this out.

Encryption keys are usually expressed in terms of bits, not bytes. Why does your cipher require half a million bits? That seems excessive to me.

Share on other sites
He's probably using this data as a one-time pad.

I still don't understand the original problem. If I make a program with a global array of size 512Kbytes, I don't think I run into any trouble. What compiler/platform is this on? And can you post an actual compilable program that shows the problem? You can leave out the exact content of your array.

Share on other sites
I would be very suspicious of this being used as a one time pad. I would need to know how it was being used before commenting.

Share on other sites

I would be very suspicious of this being used as a one time pad. I would need to know how it was being used before commenting.

Oh, I didn't mean to say that what he is doing makes sense. I was just offering a guess.

Share on other sites
Hi, thx for your answer. As you noticed, i was dead tired yesterday when i posted my code. Also since the compiler was getting stuck because of the large size of the text file i included, i could't see that memcpy was missing an argument . Thing is, i dont want the encryptor to be too slow since i really need speed in this case, it's used to encryp data of a program working similar to VNC, to remote control computer at a distance. Since most of the data is compressed either using an mp1 encoder or using zlib, it's already kinda hard for anyone intercepting those packet to decode them to image, or take control of the machine being controled, since i have the source code and i myself find my own code to be confusing ^^. Also it's only allow connection to one machine at the time (one server, one client). So i though about using encryption as an additionnal feature, although i really doubt someone will ever try to crack this (im not intending to distribute it execpt for some people i know). Well i can always post my code so that you can see what im trying to do, since i need speed, im only using a simple xor encryption atm, no key just simple xor with the cipher. As i said all i want to do is to embeed the cipher into the executable, i dont care much if it's not extremely secure for now, i can always encode it then decode it at run time in the constructor or something like that later.

 #ifndef _CENCRYPTOR_H_ #define _CENCRYPTOR_H_ #ifdef __cplusplus //----------------------------------------------------------------------// #include "Windows.h" #include "stdio.h" //----------------------------------------------------------------------// #define CIPHER_LENGTH 524288 class CEncryptor { public: CEncryptor(); private: BYTE Cipher[CIPHER_LENGTH]; //void LoadCipher(); bool LoadCipherFromFile(char *fname); void XCrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset); public: void Encrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset = 0); void Decrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset = 0); }; //----------------------------------------------------------------------// //----------------------------------------------------------------------// #endif #endif //_CENCRYPTOR_H_ 

 #include "Encryptor.h" //----------------------------------------------------------------------------- // Constructor //----------------------------------------------------------------------------- CEncryptor::CEncryptor() { //LoadCipher(); LoadCipherFromFile("Entropy.rnd"); } //----------------------------------------------------------------------------- // Load the cipher data from memory //----------------------------------------------------------------------------- /*void CEncryptor::LoadCipher() { memcpy(&Cipher[0], &MyCipher[0], CIPHER_LENGTH); }*/ //----------------------------------------------------------------------------- // Load the cipher data from a file //----------------------------------------------------------------------------- bool CEncryptor::LoadCipherFromFile(char *fname) { ZeroMemory(&Cipher[0], CIPHER_LENGTH); FILE *f = fopen(fname, "rb"); if(f){ fread(&Cipher[0], 1, CIPHER_LENGTH, f); fclose(f); return true; } return false; } //----------------------------------------------------------------------------- // XOR the given packet buffer data with the cipher //----------------------------------------------------------------------------- inline void CEncryptor::XCrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset) { DWORD Indx = Offset; for(DWORD Cpt = 0; Cpt < BufSize; Cpt++){ if(Indx >= CIPHER_LENGTH) Indx = Offset % CIPHER_LENGTH; pBuf[Cpt] ^= Cipher[Indx++]; } } //----------------------------------------------------------------------------- // Encrypt the given packet buffer //----------------------------------------------------------------------------- void CEncryptor::Encrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset) { XCrypt(pBuf, BufSize, Offset); } //----------------------------------------------------------------------------- // Decrypt the given packet buffer //----------------------------------------------------------------------------- void CEncryptor::Decrypt(BYTE *pBuf, DWORD BufSize, DWORD Offset) { XCrypt(pBuf, BufSize, Offset); } 

1. 1
2. 2
3. 3
4. 4
5. 5
frob
11

• 16
• 13
• 20
• 12
• 19
• Forum Statistics

• Total Topics
632172
• Total Posts
3004569

×