Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


new[] is flawed?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
38 replies to this topic

#21 tonemgub   Members   -  Reputation: 1143

Like
1Likes
Like

Posted 03 October 2013 - 02:56 PM

 

Maybe it's ASLR? But I don't know if it's effects would be detectable in a debugger like in your case.
 
See this compiler option: http://msdn.microsoft.com/en-us/library/bb384887.aspx
 
And I think in Windows 7 ASLR is always used, even for programs compiled without that option?



ASLR has nothing to do with this. It's purely a function of memory obtained from the OS which is going to be a level lower than memory obtained from the language runtime (or a custom allocator).

 

 

Also, not completely sure, but some sources say that ASLR does randomization at the virtual address space level of a process, more specifically - it randomizes image, stack and heap addresses. I'm not sure what you mean by "from the OS", but last I checked, the CRT allocation operators in C++ use heap functions for memory allocation (AFAIK, the operator new() allocates memory with HeapAlloc). I also don't understand why you think that "the language runtime" has it's own memory-manager implementation.

 

But I was wrong about not being able to disable ASLR on Windows 7 - Process Explorer has a column that shows the state of ASLR for each process.



Sponsor:

#22 ApochPiQ   Moderators   -  Reputation: 16080

Like
1Likes
Like

Posted 03 October 2013 - 03:10 PM

The use of HeapAlloc() is entirely implementation-dependent and not mandatory. In fact, it is perfectly legitimate - even common - for C or C++ runtimes to have their own allocators running between the OS and the application.

You also missed the other important factor, which is that we're talking about a custom allocator here, which by definition is a layer of indirection between whatever the runtime offers (whether that goes straight to the OS or not) and the application itself.

#23 tonemgub   Members   -  Reputation: 1143

Like
0Likes
Like

Posted 03 October 2013 - 03:19 PM

The use of HeapAlloc() is entirely implementation-dependent and not mandatory. In fact, it is perfectly legitimate - even common - for C or C++ runtimes to have their own allocators running between the OS and the application.

You also missed the other important factor, which is that we're talking about a custom allocator here, which by definition is a layer of indirection between whatever the runtime offers (whether that goes straight to the OS or not) and the application itself.

 

Unless his custom allocator (or the runtime allocator) uses virtual memory directly, or if it uses non-virtual memory, the memory allocated by his own allocator will come from a process heap.

 

Anyway, I have no idea how ASLR works, but Goolge'ing it, I did find mentions that the randomized addresses it produces can be seen in WinDbg...

 

So I think it should also be visible in Visual Studio.


Edited by tonemgub, 03 October 2013 - 03:20 PM.


#24 ApochPiQ   Moderators   -  Reputation: 16080

Like
0Likes
Like

Posted 03 October 2013 - 03:29 PM

Yes, you can recognize the presence of ASLR in a debugger. That should be pretty obvious.

 

What ASLR does not do is modify addresses that the runtime/application layer already have access to. That would just be insanity. You will get random addresses from the OS but any memory management implemented on top of that is immune.



#25 tonemgub   Members   -  Reputation: 1143

Like
0Likes
Like

Posted 03 October 2013 - 03:41 PM

I thought that's exactly what it does. :)

Edit:
Nevermind, and sorry if I caused any confusion... I was confused a bit myself. I read more about it, and it seems that ASLR does not modify virtual addresses at run-time, only at image load-time (or for the heap, at heap-creation time).

So the wikipedia page seems to be a bit wrong in this case. :)

I mean this: "Entropy is increased by either raising the amount of virtual memory area space over which the randomization occurs or reducing the period over which the randomization occurs. The period is typically implemented as small as possible, so most systems must increase VMA space randomization."

From here: http://en.wikipedia.org/wiki/Address_space_layout_randomization#Effectiveness

It seems to imply that randomization is done at "periods" of time, while the application is running, and that those periods are really small... Hmm. "Period" does not refer to an amount of time in this case, does it? :D

#26 DekuTree64   Members   -  Reputation: 986

Like
0Likes
Like

Posted 03 October 2013 - 04:09 PM

Why don't they store the housekeeping data preceding the allocation then, making sure the returned pointer is aligned? Operator delete[] could find the housekeeping information based on the pointer passed to it, via the magic of subtraction.

 

Seems a no-brainer to me... (maybe the implementation was implemented before alignment became a major issue though, and it is retained for backwards compatibility).

This is how I always thought it worked. AFAIK, the housekeeping data is just the length of the array, so delete[] knows how many objects to call the destructor on. I just ran a test in GCC and this is indeed how it works (count stored at ReturnedAddress - 4)

 

I tried placement new[] as well, and the pointer returned is the same as the pointer passed in, and I can't find the count stored in memory anywhere shortly before or after the data. Also, calling delete[] on it crashes. I'm pretty sure destructors have to be manually called on placement new'd things, since there's nothing marking that the delete shouldn't also release the memory either.


Edited by DekuTree64, 03 October 2013 - 04:13 PM.


#27 Paradigm Shifter   Crossbones+   -  Reputation: 5413

Like
4Likes
Like

Posted 03 October 2013 - 04:12 PM

Yes, if you call placement new you need to call the destructor manually, via obj->~ObjectType() (since placement new didn't allocate the memory, and doesn't know where it came from, it can come from anywhere, a global pool, on the stack, etc.).

 

You never call delete on a placement new constructed object, since it wasn't created via operator new.


Edited by Paradigm Shifter, 03 October 2013 - 04:17 PM.

"Most people think, great God will come from the sky, take away everything, and make everybody feel high" - Bob Marley

#28 King Mir   Members   -  Reputation: 2032

Like
0Likes
Like

Posted 03 October 2013 - 11:34 PM

Hmm I think I remember what you're talking about, somethinga bout knowing how many objects to call the destructor for. The strange thing was that it wouldn't do it on some pointers, but would on others. I just wanted to make sure there wasn't a way to solve this issue, because I really like having arrays of objects. It's so nice for cache coherency!

The way to solve this would presumably be to overload operator new[], such that it returns properly aligned data. Possibly writing it in terms of regular operator new. Don't forget top overload operator delete[] too if you do that.

#29 Vortez   Crossbones+   -  Reputation: 2704

Like
0Likes
Like

Posted 04 October 2013 - 01:37 AM

did you try

 

__declspec (align(16))

 

?



#30 tivolo   Members   -  Reputation: 971

Like
3Likes
Like

Posted 04 October 2013 - 03:04 PM

To answer the original question: yes, new[] is a bit flawed in that regard.

 

As soon as you allocate an array of non-PODs with new[], most compilers will add an extra 4 bytes to store the number of elements in the array. This is needed in order to call the destructors as soon as delete[] is invoked. I've written about the mechanism in more detail on my blog. I've yet to see a compiler which does it differently, do note though that this is not required per the standard - the compiler can store this book-keeping data where and how he sees fit. 

 

Note that if you properly declare the corresponding non-POD class type as being aligned (e.g. using __declspec(align(n)) on MSVC), you can use your own overload of operator new, and will get a properly aligned pointer. However, if you need to allocate an array of aligned non-PODs whose declaration you cannot change (e.g. a class type in a 3rd-party library), you're basically out of luck. Because of that, several engines I know of do not use new, delete, new[] or delete[], but rather their own implementations.

 

Hope that helps!



#31 Seabolt   Members   -  Reputation: 633

Like
0Likes
Like

Posted 08 October 2013 - 10:16 AM

Sorry I'm a little late to replying, I didn't think this thread would blow up like it did.
 

I do have a question for tivolo, we initially thought about not using new delete new[] or delete[] but we didn't want to forsake our constructors and the like. Is there a good way around this, or do you just use Initialize() methods on all of your objects?


Perception is when one imagination clashes with another

#32 ApochPiQ   Moderators   -  Reputation: 16080

Like
1Likes
Like

Posted 08 October 2013 - 11:31 AM

The canonical fix is to write a macro that invokes placement new on memory returned from your custom allocator.



#33 Seabolt   Members   -  Reputation: 633

Like
0Likes
Like

Posted 08 October 2013 - 02:16 PM

Will placement new still have house keeping information that it needs to store? Would the pointer returned to the client be at the same position as the placement?


Perception is when one imagination clashes with another

#34 Brother Bob   Moderators   -  Reputation: 8455

Like
0Likes
Like

Posted 08 October 2013 - 02:30 PM

The standard guarantees that the pointer returned by placement new (both scalar and array variants) is equal to, and that the objects are constructed at, the pointer you pass to them. If you need any information about the number of objects to destroy later, you need to keep that information yourself.



#35 SiCrane   Moderators   -  Reputation: 9630

Like
1Likes
Like

Posted 08 October 2013 - 02:39 PM


The standard guarantees that the pointer returned by placement new (both scalar and array variants) is equal to, and that the objects are constructed at, the pointer you pass to them.

No, it doesn't. Not for the array placement new. 5.3.4 paragraph 12 of the standard specifically states that the pointer returned by array placement new may be offset, and in practice most compilers will for complex types.



#36 Brother Bob   Moderators   -  Reputation: 8455

Like
1Likes
Like

Posted 08 October 2013 - 02:43 PM

Section 18.6.1.3 lists the special forms of placement new where you pass the pointer to the memory location where the objects are constructed.



#37 SiCrane   Moderators   -  Reputation: 9630

Like
1Likes
Like

Posted 08 October 2013 - 02:50 PM

Yes, they exist. I never said that they didn't. But implementations are still permitted to add array allocation overhead to array placement new as described in 5.3.4. And even if they weren't allowed to do it, it doesn't change the fact that most implementations do anyways.



#38 Brother Bob   Moderators   -  Reputation: 8455

Like
1Likes
Like

Posted 08 October 2013 - 03:01 PM

Actually, the way I see it now is that section 18.6.1.3 lists the behavior of the function named operator new that the operator new calls to acquire memory before calling the constructors and whatever other book keeping it will do. Different functions to acquire the memory, but it's still the same operator new for initializing the array; book keeping included.



#39 tivolo   Members   -  Reputation: 971

Like
0Likes
Like

Posted 10 October 2013 - 01:42 PM

Sorry I'm a little late to replying, I didn't think this thread would blow up like it did.
 

I do have a question for tivolo, we initially thought about not using new delete new[] or delete[] but we didn't want to forsake our constructors and the like. Is there a good way around this, or do you just use Initialize() methods on all of your objects?

 

You don't have to forsake constructors and the likes.

But you have to do the things the compiler normally does for you yourself. That means that you have to use placement new on memory returned by your custom allocator, and also have to make sure the constructor/destructors are correctly invoked for arrays. I use regular placement new both for single instances as well as arrays, and don't rely on array placement new.

 

You can find really detailed info on my blog, search for "memory system", it's a five-part series.






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS