• Advertisement
Sign in to follow this  

how to delete content from deque ?

This topic is 3001 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

how to delete content from deque ? // deque::max_size #include <iostream> #include <deque> using namespace std; int main () { unsigned int i; deque<char *> d; cout << "Enter number of elements: "; i=255; char *h; int a; a=0;; while(a++<100){ h=new char[1024*1024*4]; sprintf(h,"item id=%i time=%i",a,clock()); d.push_back(h); _sleep(10); }; cout << "That size exceeds the limit.\n"; cout<<"deque maz size="<<d.max_size(); d.clear(); //wtf ? this is not working :| a=0;; cout<<"\n--size="<<d.size(); //ewen you see zero there content is still accessible and wastes system resources :teach: while(a++<100){ cout<<d[a]<<"\n"; }; system("pause"); return 0; }

Share this post


Link to post
Share on other sites
Advertisement
When you allocate pointers with new [], you must release them with delete [].

It is undefined behaviour to access the elements of a container after calling clear() on it. You should always use the containers iterators, or in the case of a random access container, a for loop bounded by the containers size() to access the elements.

Your program would be simplified by using more elements from the C++ standard library, such as std::string and std::stringstream:

#include <iostream>
#include <string>
#include <deque>
#include <sstream>
using namespace std;

int main ()
{
deque<string> d;

cout << "Enter number of elements: ";
int elements = 255;

for(int i = 0 ; i < elements ; ++i)
{
// string stream is like "cout"
// Except it writes to a string
stringstream stream;

stream << "item id=" << i << " time=" << clock();

// Here, we get the string from the stream and
// add it to our deque
d.push_back(stream.str());
_sleep(10);
};

cout << "That size exceeds the limit.\n";
cout << "deque maz size=" << d.max_size() << '\n';


d.clear();

cout << "--size=" << d.size();

system("pause");
return 0;
}


Note, even with this program (which cleans up 100% of allocations), if you attempted to access the elements you see the data, like before. However, this is the nature of undefined behaviour. What might be happening is that the allocator is holding onto the memory you have released, anticipating that you will request more (that is how applications typically proceed).

Finally, is there any particular reason you are using std::deque over, say, std::vector?

Share this post


Link to post
Share on other sites
Quote:
Original post by dragonnnnnnnnnnnnnn
is this means char ** arrays are not supported ?


They are supported, but using std::string over them can save you from a world of hurt (see [1], [2]).

Share this post


Link to post
Share on other sites
Quote:
Original post by rip-off
When you allocate pointers with new [], you must release them with delete [].

It is undefined behaviour to access the elements of a container after calling clear() on it. You should always use the containers iterators, or in the case of a random access container, a for loop bounded by the containers size() to access the elements.

Your program would be simplified by using more elements from the C++ standard library, such as std::string and std::stringstream:
*** Source Snippet Removed ***
Note, even with this program (which cleans up 100% of allocations), if you attempted to access the elements you see the data, like before. However, this is the nature of undefined behaviour. What might be happening is that the allocator is holding onto the memory you have released, anticipating that you will request more (that is how applications typically proceed).

Finally, is there any particular reason you are using std::deque over, say, std::vector?


well ok.
but all the data is still there.
i mean locked and in use by program.
check this code with process explorer and you will see it stiil hold memory locked for this program while it virtually uses zero memory !
#include <iostream>
#include <string>
#include <deque>
#include <sstream>
using namespace std;

int main ()
{
deque<string> d;

cout << "Enter number of elements: ";
int elements = 3000*1000;
for(int i = 0 ; i < elements ; ++i)
{
// string stream is like "cout"
// Except it writes to a string
stringstream stream;

stream << "item id=" << i << " time=" << clock();

// Here, we get the string from the stream and
// add it to our deque
d.push_back(stream.str());
// _sleep(10);
};

cout << "That size exceeds the limit.\n";
cout << "deque maz size=" << d.max_size() << '\n';





d.clear();

cout << "--size=" << d.size();





system("pause");
return 0;
}

Share this post


Link to post
Share on other sites
Virtual memory requested and committed by process has next to nothing to do with C++'s new and delete operators.

Short answer: process explorer shows very inaccurate numbers.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Virtual memory requested and committed by process has next to nothing to do with C++'s new and delete operators.

Short answer: process explorer shows very inaccurate numbers.


i dont think so.
why would process explorer fail about that ?
its not even using c++ interface.
its using windows kernel messges afaik

Share this post


Link to post
Share on other sites
Quote:
Original post by dragonnnnnnnnnnnnnn
i dont think so.


You are free to think that fire isn't hot.

Quote:
why would process explorer fail about that ?
its using windows kernel messges afaik


Yes, and Windows (or any OS) kernel is a complex beast. Suffice to say that the way memory assigned to process (which is reported by OS tools) is something completely different from what C++ operators new or malloc do.

Aside from OS-related bookkeeping, memory mapped exe image, kernel resources, thread stacks and plethora of other bits and bytes, part of the memory used is also that claimed by your very application.


Quote:
its not even using c++ interface.
Which is why such tools will never be able to report memory usage accurately.

The only way to accurately track memory usage in C++ application is to manually track each allocation and deallocation. Tools such as valgrind are used for this purpose.

Share this post


Link to post
Share on other sites
Quote:
Original post by dragonnnnnnnnnnnnnn
is this means char ** arrays are not supported?

What you are looking for is called std::vector<std::string> in C++.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by dragonnnnnnnnnnnnnn
i dont think so.


You are free to think that fire isn't hot.

Quote:
why would process explorer fail about that ?
its using windows kernel messges afaik


Yes, and Windows (or any OS) kernel is a complex beast. Suffice to say that the way memory assigned to process (which is reported by OS tools) is something completely different from what C++ operators new or malloc do.

Aside from OS-related bookkeeping, memory mapped exe image, kernel resources, thread stacks and plethora of other bits and bytes, part of the memory used is also that claimed by your very application.


Quote:
its not even using c++ interface.
Which is why such tools will never be able to report memory usage accurately.

The only way to accurately track memory usage in C++ application is to manually track each allocation and deallocation. Tools such as valgrind are used for this purpose.


then so.
process explorer fails ? right ?
so can you explain why this breaks when exeeds the ram size while it should not if its coded right.
and if there is no memory allocated on system after clear/delete :P


#include <iostream>
#include <string>
#include <deque>
#include <sstream>
using namespace std;

int main ()
{
deque<string> d,d2,d3,d4,d5;

cout << "testing truts states";
int elements ;


a:
elements = 1048576*1024;
for(int i = 0 ; i < elements ; ++i)
{
// string stream is like "cout"
// Except it writes to a string
stringstream stream;

stream << "wooooooooooooooooooooohhhhhhhhhhhhhhhhhhhooooooooooooooooooooo.........................item id=" << i << " time=" << clock();

// Here, we get the string from the stream and
// add it to our deque
d.push_back(stream.str());
// _sleep(10);
};

cout << "That size exceeds the limit.\n";
cout << "deque maz size=" << d.max_size() << '\n';





d.clear();

cout << "insane sizes on virtual zeros--size=" << d.size();


goto a;




system("pause");
return 0;
}

Share this post


Link to post
Share on other sites
You are focusing on the wrong thing. Process Explorer gives the correct information, however this information is different from your program. As far as Windows is concerned the memory is allocated. However, that doesn't mean that the memory should be used by C++.

I gave a short explanation at the end of my first post. I will expand on that here.

When you call new, or new [], the default C++ allocator in the C++ runtime will attempt to locate a memory block for you (unless you write a custom allocator). A naive implementation might request a block of memory from the OS every time. The OS has a coarse grained view of memory, it might think in terms of 4 Kiliobyte chunks.

Such an implementation has two implications for user programs:

  • Allocation is slow, because it involves switching from user space to kernel space

  • Allocation is too coarse, small allocations will waste large amounts of memory


In practise, no one would write or use such an allocator. Instead, the default allocator acts something like this: it allocates medium chunks from the OS and then serves small requests from these blocks. Releasing is a little more complicated. First of all, the allocator cannot release part of a chunk back to the OS, it must wait until all portions from that chunk are released. Secondly, it might want to keep a few empty chunks around anyway, because most programs will request more memory in future, and as mentioned asking the OS for memory is (relatively) expensive.

This is a high level overview, but that is the basics. What you must understand is that the figures in process explorer don't matter. You cannot control memory on that level without writing an allocator, and I can guarantee you that for the moment you won't be able to beat the default allocator.

At the end of the day, there is nothing to worry about. The allocator won't be so stupid to hang onto megabytes of empty pages if it can release them back to the OS. Your original program is too small to draw conclusions from.

Share this post


Link to post
Share on other sites
Quote:
Original post by rip-off
Also, use proper loops rather than using "goto". In general, you shouldn't be using "goto".


may be but . my program uses a giant database which works fine by default but when its time to get data and do some work and place it back. the old memory used stays allocated. and no stl do not deallocates it automatically for me on the way i use them. or may be im missing somethings .
but that way memoryused x 2 wasted on system means the program definitely will crash in future [after some seconds or more :P ].

eg if i push_back(something) read it and place something newly allocated. there is no way to delete old data. actually the thing i want to do is deleting completely part of array . deleting old allocated data .

also i tryed deque.erase(..) and it too does not works ...


ps. do you have any ideas about dynamic char/void random access type memory allocation code ?
if so thats what i need

Share this post


Link to post
Share on other sites
Post your program. If you are running out of memory, it is because of a bug in your code, or you are trying to process more data than your computer can handle in RAM. What kind of database size are we talking about here?

Quote:

the old memory used stays allocated. and no stl do not deallocates it automatically for me on the way i use them.

The types in the Standard C++ Library are used by thousands of programmers across the world, in commercial programs as well as hobbyist/free ones. Bugs tend to be caught rather quickly. The chances are vanishingly against you finding a bug in the standard library.

Share this post


Link to post
Share on other sites
Quote:
Original post by rip-off
Post your program. If you are running out of memory, it is because of a bug in your code, or you are trying to process more data than your computer can handle in RAM. What kind of database size are we talking about here?

Quote:

the old memory used stays allocated. and no stl do not deallocates it automatically for me on the way i use them.

The types in the Standard C++ Library are used by thousands of programmers across the world, in commercial programs as well as hobbyist/free ones. Bugs tend to be caught rather quickly. The chances are vanishingly against you finding a bug in the standard library.


well did you take a look at application code posted 12/5/2009 12:40:43 PM ,it newer deallocates or stabilizes the size of memory used.
instead it allocated after every goto operation .
if stl holds memory for reuse why there its not using allready allocated memory ... instead allocating new one ...?
that simply part of application im using which is buffering the data thats going to be worked on. first operation im doing with this is sort by using specific tag inside line. so standard stl sort algorithm is not applicable
anyway thats not actually the case if we use std::string afaik.
but simply the problem is .clear and deque.erase(..) operatios seems unefficient for real memory . so it works for stl operations effectively. but in the name of allocated or overallocated memory it fails.

Share this post


Link to post
Share on other sites
Quote:
Original post by dragonnnnnnnnnnnnnn


well did you take a look at application code posted 12/5/2009 12:40:43 PM ,it newer deallocates or stabilizes the size of memory used.


You are storing 1 billion strings (1e9), each string is about 80 characters. That means 80 gigabytes of memory before the for loop even finishes. This would break hard on 32 bit build, and likely take a very long time on 64 bit. I'm really not quite sure what address space limits are in practice (not to mention it would need very large page file).


Change elements to some sensible number, such as 1024, and run the whole thing again.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
Quote:
Original post by dragonnnnnnnnnnnnnn


well did you take a look at application code posted 12/5/2009 12:40:43 PM ,it newer deallocates or stabilizes the size of memory used.


You are storing 1 billion strings (1e9), each string is about 80 characters. That means 80 gigabytes of memory before the for loop even finishes. This would break hard on 32 bit build, and likely take a very long time on 64 bit. I'm really not quite sure what address space limits are in practice (not to mention it would need very large page file).


Change elements to some sensible number, such as 1024, and run the whole thing again.


allright but it is just for make sure it crashes .
its a infinite allocate deallocate function.
the problem is stl::deques deallocation function is not doing it work physically .
and this is for proving that its deallocation function is not working .

Share this post


Link to post
Share on other sites
The deallocation never gets a chance to happen, because you are trying to allocate more memory than you computer can handle.

Here is a modified version. It calls the deallocation function every 10,000th element. It should eventually settle on some upper limit of memory use and run forever.

int main ()
{
deque<string> d;

for(int i = 0 ; /* never terminate */ ; ++i)
{
stringstream stream;



stream << "wooooooooooooooooooooohhhhhhhhhhhhhhhhhhhooooooooooooooooooooo.........................item id=" << i << " time=" << clock();




d.push_back(stream.str());

if((i % 10000) == 0)
{
cout << "emptying deque!";
d.clear();
}
}
}

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement