Jump to content
  • Advertisement
Sign in to follow this  
corrington_j

problem with very large float array

This topic is 4968 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I am having some trouble trying to allocate a very large float array. It was originally a three dim array, that could be as large as 1024 * 1024 * 392. Now I have made it a one dim array, and the program crashes trying to allocate it. It says something like bad alloc. I am trying this on a system with 2 gigs of ram. This array should only take 1.6 gigs. What is the problem.

Share this post


Link to post
Share on other sites
Advertisement
That is 1.6GB of contigous memory you are requesting if I'm not mistaking. The OS might not be able to give you that much in one go. Also why are you requesting so much memory up front?

Share this post


Link to post
Share on other sites
Quote:
Original post by corrington_j
I am having some trouble trying to allocate a very large float array. It was originally a three dim array, that could be as large as 1024 * 1024 * 392. Now I have made it a one dim array, and the program crashes trying to allocate it. It says something like bad alloc. I am trying this on a system with 2 gigs of ram. This array should only take 1.6 gigs. What is the problem.


Are you putting the array into a function (just like float array[size]) then it would overflow your heap. If you are using new/delete or malloc/free then i'm not sure. Maybe some sort of limit within windows (can't allocate big block, if its making it all continous) or your program. Can you split it in 4 sections maybe.

The memory size shouldn't matter as all things don't fit into main memory anyway and are paged in and out as needed.

Share this post


Link to post
Share on other sites
so, why does it work with a three dim array, is that not contigeous memory. If i made it two array would it maybe work. Also, it used to be a global, non dynamic array, and it worked. Why

Share this post


Link to post
Share on other sites
it's defined as ProcessedImage = new float[ImageDimX * ImageDimX * NumPixelsZ].
it can be as large as 1024 * 1024 * 392. It crashes when it is 1024 * 1024 * 300 or greater. It seems like it should just be able to flow over into virtual memory, even if there wasn't enough memory. Also i am using VC++ .NET 2003, are there any settings i need to change for large mem blocks. Also i tried splitting it into two blocks, and it fails trying to create the second block. Please help This array interfaces with lots of existing code, and i don't want to have to change all the other code.

Share this post


Link to post
Share on other sites
You are trying to reserve and commit about 1.5 GB of physical space. You could easily be running out of virtual addresses or physical space requesting so much memory at once.

What you want to try first is see if there's even that much virtual space available to reserve. You can do this with VirtualAlloc with MEM_RESERVE. Next, commit pieces of this address space bit by bit until you find your limit.

If that limit is far below the memory you need, then you might just have to manually page your data to temporary files and only process small portions of it at once.

Share this post


Link to post
Share on other sites
What do you want to do with this array ? Is it for a simulation ?

If most of the array contains zeros, then another implementation can reduce memory usage. Maybe you should take a look at Blitz.

http://www.oonumerics.org/blitz/

Share this post


Link to post
Share on other sites
Quote:
Original post by corrington_j
so, why does it work with a three dim array, is that not contigeous memory. If i made it two array would it maybe work. Also, it used to be a global, non dynamic array, and it worked. Why


A 3 dim array example: float t[3][3][3]. This allocates 3x3 memory chunks of 12 bytes (3 floats) than can be placed anywhere (but it most likely to be placed contigeous), if I'm not mistaken, and therefore it may or may not be contigeous memory.

Share this post


Link to post
Share on other sites
If you need that much RAM, you probably need to rethink your algorithm. What are you actually doing? Usually you can break it up so you don't need 1.6GB of memory in one go.

Anyway, requesting that much RAM and wondering why it fails is silly: it must fail since you probably don't have that much physical memroy free with Windows running, let alone in a contiginous block.

Share this post


Link to post
Share on other sites
Consider using a memory mapped file, where the OS pages individual memory pages in and out automatically for you. Look up mmap() on Linux and CreateFileMapping() on Win32.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!