Huges arrays

Started by
14 comments, last by Floru 22 years, 5 months ago
16 gigs of mem. thought you were coding the Matrix.
------------------General Equation, this is Private Function reporting for duty, sir!a2k
Advertisement
Devil''s Advocate here

Doesn''t the hash table need the array to plug the numbers into? I think you are suggesting the solutions that this person already had, but with a possible faster search/indexing capability.

I think you need to sample the file and go with the linked list option. I don''t think you can reasonable read in the entire file and then parse it. The linked list could kak too if too many of the numbers are unique. Too many nodes would be allocated and the whole memory problem pops up again.

Maybe I''m not thinking about these problems the right way?

sounds like a homework problem to me. Surely with something like this the teacher has told you that there is going to be a low value that the number wont go below and a high number the number wont go above.
Or perhaps there will be no more then x hundred or x thousand numbers in the file.
You really need to provide details like that if they exist.

"I pity the fool, thug, or soul who tries to take over the world, then goes home crying to his momma."
- Mr. T
Like I said, just a starting point - not necessarily the most efficient
"Absorb what is useful, reject what is useless, and add what is specifically your own." - Lee Jun Fan

I tried vector (list) and it was already too slow for 1MB file. With std::map I got quite good results.

Thanks.
quote:Original post by Anonymous Poster
Devil''s Advocate here

Doesn''t the hash table need the array to plug the numbers into? I think you are suggesting the solutions that this person already had, but with a possible faster search/indexing capability.


The idea behind using a hash table or a map is that you aren''t likely to enconter all of the numbers (as pointed out, that would be a 16GB file...). Here, you only use the storage you really need. A hash table is typically smaller than the potential number of entries : that''s what they''re for, storing sparse data.
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." — Brian W. Kernighan

This topic is closed to new replies.

Advertisement