new[] is flawed?

Started by
37 comments, last by tivolo 10 years, 6 months ago

Maybe it's ASLR? But I don't know if it's effects would be detectable in a debugger like in your case.

See this compiler option: http://msdn.microsoft.com/en-us/library/bb384887.aspx

And I think in Windows 7 ASLR is always used, even for programs compiled without that option?



ASLR has nothing to do with this. It's purely a function of memory obtained from the OS which is going to be a level lower than memory obtained from the language runtime (or a custom allocator).

Also, not completely sure, but some sources say that ASLR does randomization at the virtual address space level of a process, more specifically - it randomizes image, stack and heap addresses. I'm not sure what you mean by "from the OS", but last I checked, the CRT allocation operators in C++ use heap functions for memory allocation (AFAIK, the operator new() allocates memory with HeapAlloc). I also don't understand why you think that "the language runtime" has it's own memory-manager implementation.

But I was wrong about not being able to disable ASLR on Windows 7 - Process Explorer has a column that shows the state of ASLR for each process.

Advertisement
The use of HeapAlloc() is entirely implementation-dependent and not mandatory. In fact, it is perfectly legitimate - even common - for C or C++ runtimes to have their own allocators running between the OS and the application.

You also missed the other important factor, which is that we're talking about a custom allocator here, which by definition is a layer of indirection between whatever the runtime offers (whether that goes straight to the OS or not) and the application itself.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

The use of HeapAlloc() is entirely implementation-dependent and not mandatory. In fact, it is perfectly legitimate - even common - for C or C++ runtimes to have their own allocators running between the OS and the application.

You also missed the other important factor, which is that we're talking about a custom allocator here, which by definition is a layer of indirection between whatever the runtime offers (whether that goes straight to the OS or not) and the application itself.

Unless his custom allocator (or the runtime allocator) uses virtual memory directly, or if it uses non-virtual memory, the memory allocated by his own allocator will come from a process heap.

Anyway, I have no idea how ASLR works, but Goolge'ing it, I did find mentions that the randomized addresses it produces can be seen in WinDbg...

So I think it should also be visible in Visual Studio.

Yes, you can recognize the presence of ASLR in a debugger. That should be pretty obvious.

What ASLR does not do is modify addresses that the runtime/application layer already have access to. That would just be insanity. You will get random addresses from the OS but any memory management implemented on top of that is immune.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

I thought that's exactly what it does. :)

Edit:
Nevermind, and sorry if I caused any confusion... I was confused a bit myself. I read more about it, and it seems that ASLR does not modify virtual addresses at run-time, only at image load-time (or for the heap, at heap-creation time).

So the wikipedia page seems to be a bit wrong in this case. :)

I mean this: "Entropy is increased by either raising the amount of virtual memory area space over which the randomization occurs or reducing the period over which the randomization occurs. The period is typically implemented as small as possible, so most systems must increase VMA space randomization."

From here: http://en.wikipedia.org/wiki/Address_space_layout_randomization#Effectiveness

It seems to imply that randomization is done at "periods" of time, while the application is running, and that those periods are really small... Hmm. "Period" does not refer to an amount of time in this case, does it? :D

Why don't they store the housekeeping data preceding the allocation then, making sure the returned pointer is aligned? Operator delete[] could find the housekeeping information based on the pointer passed to it, via the magic of subtraction.

Seems a no-brainer to me... (maybe the implementation was implemented before alignment became a major issue though, and it is retained for backwards compatibility).

This is how I always thought it worked. AFAIK, the housekeeping data is just the length of the array, so delete[] knows how many objects to call the destructor on. I just ran a test in GCC and this is indeed how it works (count stored at ReturnedAddress - 4)

I tried placement new[] as well, and the pointer returned is the same as the pointer passed in, and I can't find the count stored in memory anywhere shortly before or after the data. Also, calling delete[] on it crashes. I'm pretty sure destructors have to be manually called on placement new'd things, since there's nothing marking that the delete shouldn't also release the memory either.

Yes, if you call placement new you need to call the destructor manually, via obj->~ObjectType() (since placement new didn't allocate the memory, and doesn't know where it came from, it can come from anywhere, a global pool, on the stack, etc.).

You never call delete on a placement new constructed object, since it wasn't created via operator new.

"Most people think, great God will come from the sky, take away everything, and make everybody feel high" - Bob Marley

Hmm I think I remember what you're talking about, somethinga bout knowing how many objects to call the destructor for. The strange thing was that it wouldn't do it on some pointers, but would on others. I just wanted to make sure there wasn't a way to solve this issue, because I really like having arrays of objects. It's so nice for cache coherency!

The way to solve this would presumably be to overload operator new[], such that it returns properly aligned data. Possibly writing it in terms of regular operator new. Don't forget top overload operator delete[] too if you do that.

did you try

__declspec (align(16))

?

To answer the original question: yes, new[] is a bit flawed in that regard.

As soon as you allocate an array of non-PODs with new[], most compilers will add an extra 4 bytes to store the number of elements in the array. This is needed in order to call the destructors as soon as delete[] is invoked. I've written about the mechanism in more detail on my blog. I've yet to see a compiler which does it differently, do note though that this is not required per the standard - the compiler can store this book-keeping data where and how he sees fit.

Note that if you properly declare the corresponding non-POD class type as being aligned (e.g. using __declspec(align(n)) on MSVC), you can use your own overload of operator new, and will get a properly aligned pointer. However, if you need to allocate an array of aligned non-PODs whose declaration you cannot change (e.g. a class type in a 3rd-party library), you're basically out of luck. Because of that, several engines I know of do not use new, delete, new[] or delete[], but rather their own implementations.

Hope that helps!

This topic is closed to new replies.

Advertisement