How much memory are you actually allocating? Memory is allocated in rather large pages, so freeing juste a few kB of memory may not actually cause any page to get deallocated.Instead, from the Task Manager (I've also another application called Process Explorer that shows memory usage more in detail) the memory is not freed.
It is my understanding that vectors will only free memory allocated to storing the structs themselves. If the structs contain any reference to other data (such as pointers to objects), the data will still be there - only the memory allocated to storing the reference will have been freed. So if you have anything other than primitive types (like ints, etc...) in your struct, you need to explicitly free the objects they point to since they are actually references.