22 replies to this topic

[yckx]

I switched to the DirectXMath library when I switched to D3D11, and I'm running into this issue of Microsoft's std::vector implementation being incompatible with aligned data types. Looking around the 'Net it looks like the standard fixes are to avoid using std::vector, use a custom allocator, patch vector::rebind vector::resize to take a const reference, or a combination of the latter two measures. But all the relevant pages I found were from 2008 or older.

So, what does the GD.net community suggest? And does anyone have a custom aligned allocator, or do I need to learn to write my own?

[lpcstr]

Well, one solution would be not to use MSVC at all. If you are doing cross platform projects, you might as well be using MinGW on Windows, since you are likely using GCC already on other operating systems.

If you are not targeting Non-Microsoft OSes at all, you can still use MinGW.

If you must use MSVC, then you can replace the horrible Microsoft STL implementation with STLport.

[Bregma]

I switched to the DirectXMath library when I switched to D3D11, and I'm running into this issue of Microsoft's std::vector implementation being incompatible with aligned data types. Looking around the 'Net it looks like the standard fixes are to avoid using std::vector, use a custom allocator, patch vector::rebind to take a const reference, or a combination of the latter two measures. But all the relevant pages I found were from 2008 or older.

So, what does the GD.net community suggest? And does anyone have a custom aligned allocator, or do I need to learn to write my own?

If you have a specific additional memory allocation requirement (like a specific alignment) and you want to use a standard library container, you have to replace the default allocator with one that satisfies the additional requirement. This is not specific to any one C++ runtime. It is why allocators are a customization point in the standard library.

It is not difficult to write something that satisfies the allocator requirements, but you might want to have a good reference handy (like Josuttis) just in case.

[lpcstr]

[If you have a specific additional memory allocation requirement (like a specific alignment) and you want to use a standard library container, you have to replace the default allocator with one that satisfies the additional requirement. This is not specific to any one C++ runtime. It is why allocators are a customization point in the standard library.

It is not difficult to write something that satisfies the allocator requirements, but you might want to have a good reference handy (like Josuttis) just in case.

No, the problem OP is having isn't related to memory allocators, it's related to the fact that Microsoft's STL uses the wrong parameter types in functions like vector::resize, effectively making their STL completely incompatible with aligned types, regardless of what allocator is used.

[Bregma]

If you must use MSVC, then you can replace the horrible Microsoft STL implementation with STLport.

MSVC has a fairly decent standard library implementation (the days of the lawsuits that prevented Microsoft from bundling a conformant library with MSVC6 after the previous C++ standard was released have been over for quite some time).

STLport does not come close to conforming to the current C++ standard.

[Hodgman]

One solution is to use a std::vector replacement. For example, the bullet physics source code contains a class called btAlignedObjectArray, which is designed to solve your problem.

No, the problem OP is having isn't related to memory allocators, it's related to the fact that Microsoft's STL uses the wrong parameter types in functions like vector::resize, effectively making their STL completely incompatible with aligned types, regardless of what allocator is used.

No, the problem OP is having is related to standard allocators not satisfying non-default allocation requirements -- this is the same problem no matter what compiler/STL implementation you're using.
For example, on GCC, if you use the non-standard __attribute__ syntax to specify a non-default alignment value, then std::vector won't work out of the box any more, due to the fact that this kind of structure alignment is non-standard.

Secondly, your information about Microsoft's 'STL' being non-standards compliant is incorrect, as of about 5 years ago.

[TheUnbeliever]

No, the problem OP is having is related to standard allocators not satisfying non-default allocation requirements -- this is the same problem no matter what compiler/STL implementation you're using.

MSVC (9 and 10, at least) both pass resize's _Val parameter by value, not const reference. Consequently, even if you write a custom allocator, you get a compiler error warning that the parameter cannot be aligned. So, whilst the allocator is needed, it seems disingenuous to say that the library implementation is entirely faultless.

[lpcstr]

MSVC has a fairly decent standard library implementation (the days of the lawsuits that prevented Microsoft from bundling a conformant library with MSVC6 after the previous C++ standard was released have been over for quite some time).

STLport does not come close to conforming to the current C++ standard.

I was recommended STLport by multiple users here at GD.net just a year ago when I hade the exact same issue when using MSVC's srd::vector with 16 byte aligned float structures. I had an appropriate allocator made up, but because MSVC's STL passes by value in vector::resize, it is impossible to get anything buy compiler errors out of it.

This problem doesn't exist with STLport or GNU STL, so I stand by my statements.

[Hodgman]

MSVC (9 and 10, at least) both pass resize's _Val parameter by value, not const reference.

Does the standard specify the signature as: void resize(size_type sz, T c = T()) ?
On my current project, the 3 different compilers that we use all comply with the above signature.

[AllEightUp]

MSVC has a fairly decent standard library implementation (the days of the lawsuits that prevented Microsoft from bundling a conformant library with MSVC6 after the previous C++ standard was released have been over for quite some time).

STLport does not come close to conforming to the current C++ standard.

I was recommended STLport by multiple users here at GD.net just a year ago when I hade the exact same issue when using MSVC's srd::vector with 16 byte aligned float structures. I had an appropriate allocator made up, but because MSVC's STL passes by value in vector::resize, it is impossible to get anything buy compiler errors out of it.

This problem doesn't exist with STLport or GNU STL, so I stand by my statements.

Standing by observation is a good thing but understanding the problem is probably more important in this case. Basically the problem has nothing to do with the STL of VC but in a fundamental issue of C++ and how to interpret a semi-vague standard in this area. The STL definition of an allocator has nothing to do with memory alignment, worse, by definition the amount of allocation is requested by using "sizeof". So, a fully compliant STL only allows you to specify the initial address of a vector by overloading the allocator, it will (by standard) use sizeof to figure out how much memory to allocate for each element and as such the delta between elements in that memory. The STL versions you mention support "hacks" which work with padded structures instead of the basic "sizeof", it makes sense because that is "usually" what you want, but it is actually "against" standards compliance.

C++ doesn't support "alignment" directly, that's all done via extensions using pragma's, attributes etc. All such concepts are compiler specific and while rather stupid, VC's STL implementation doesn't take into account for these items. Stupid versus non compliant are very different things in this case. VC's STL since 2k8 versions have been exceptionally compliant.

The difference here is critical. VC's STL is actually the more compliant in this case, unfortunately it is not the most usable for what you want to do with it. Which version is better? One is very compliant and the others are cheating to be more usable. Given all the complexities in C++, I prefer more compliance so there are fewer invisible things going on behind my back.

[TheUnbeliever]

MSVC (9 and 10, at least) both pass resize's _Val parameter by value, not const reference.

Doesn't the standard specify the signature as: void resize(size_type sz, T c = T()) ?
On my current project, the 3 different compilers that we use all comply with the above signature.

I don't have access to a copy of the standard, so I can't honestly say. It's entirely plausible. In that case, maybe the word 'fault' isn't right to use, certainly not of the library. My point was really just that the combination of library and compiler mean that writing a custom allocator is insufficient to solve the problem - you must also patch vector to take a const reference. This was definitely the case with MSVC9. The library in 10 has stuck with the same signature, but perhaps the compiler allows alignment of function parameters. I can't remember if I've tried.

So, yeah - perhaps my wording is a little strong, but there's at least one additional problem which we don't have enough information (define 'incompatible', and which development environment are we using) to rule out. :o)

[lpcstr]

Standing by observation is a good thing but understanding the problem is probably more important in this case. Basically the problem has nothing to do with the STL of VC but in a fundamental issue of C++ and how to interpret a semi-vague standard in this area. The STL definition of an allocator has nothing to do with memory alignment, worse, by definition the amount of allocation is requested by using "sizeof". So, a fully compliant STL only allows you to specify the initial address of a vector by overloading the allocator, it will (by standard) use sizeof to figure out how much memory to allocate for each element and as such the delta between elements in that memory. The STL versions you mention support "hacks" which work with padded structures instead of the basic "sizeof", it makes sense because that is "usually" what you want, but it is actually "against" standards compliance.

C++ doesn't support "alignment" directly, that's all done via extensions using pragma's, attributes etc. All such concepts are compiler specific and while rather stupid, VC's STL implementation doesn't take into account for these items. Stupid versus non compliant are very different things in this case. VC's STL since 2k8 versions have been exceptionally compliant.

The difference here is critical. VC's STL is actually the more compliant in this case, unfortunately it is not the most usable for what you want to do with it. Which version is better? One is very compliant and the others are cheating to be more usable. Given all the complexities in C++, I prefer more compliance so there are fewer invisible things going on behind my back.

Thanks for the clarification. Standards aside, OP as well as many of us find ourselves needing dynamic arrays of aligned data, so this is an issue which needs a solution.

[Hodgman]

Another (mostly) drop-in replacement for std::vector is rde::vector.

[yckx]

So, yeah - perhaps my wording is a little strong, but there's at least one additional problem which we don't have enough information (define 'incompatible', and which development environment are we using) to rule out. )

"Incompatible" means I'm getting this error:
error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
I apologize for not posting it earlier.

And I'm using MSVC 2010 Express on a Win7 64-bit machine (but compiling for 32-bit debug).

I guess I'm going to look into alternative containers and custom allocators and see which I prefer.

[TheUnbeliever]

"Incompatible" means I'm getting this error:
error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
I apologize for not posting it earlier.

Writing a custom allocator won't fix this: you either need to, standard be damned, patch the line in the header (the error presumably points to it) or use a replacement container. Either way, you will need a custom allocator as well.

[Bregma]

I don't have access to a copy of the standard, so I can't honestly say. It's entirely plausible.

For the record, ISO-14882 says this.
[23.3.6.2](11) void resize(size_type sz, const T& c);

If the Microsoft implementation does not meet this requirement, it is nonconforming (ie. broken). Very unfortunate. The OP's problem will still obtain with other implementations because he needs to provide a custom allocator.

Also, for the record, STLport does not implement about 50% of the current standard library.

[smasherprog]

So, yeah - perhaps my wording is a little strong, but there's at least one additional problem which we don't have enough information (define 'incompatible', and which development environment are we using) to rule out. )

"Incompatible" means I'm getting this error:
error C2719: '_Val': formal parameter with __declspec(align('16')) won't be aligned
I apologize for not posting it earlier.

And I'm using MSVC 2010 Express on a Win7 64-bit machine (but compiling for 32-bit debug).

I guess I'm going to look into alternative containers and custom allocators and see which I prefer.

The above error you received is a problem with the Visual studio compiler cannot guarantee 16 byte alignment for variables that are copied on the stack. A fix to your problem is to compile your application as a 64 bit compile and you will not receive any errors. I would suggest going for 64 bit compiles as that is going to be the dominant os version.

You can read this http://thetweaker.wordpress.com/2010/05/05/stdvector-of-aligned-elements/

[yckx]

If the Microsoft implementation does not meet this requirement, it is nonconforming (ie. broken). Very unfortunate.

After doing some more informed Googling, it appears that this is a relatively recent change to the standard, and that Microsoft actually complied here while many other vendors chose not to for the sake of usability. I can post a link later today that goes into more detail.

Edit: Here's the blog entry and followup I was talking about. I'm not sure when the revision occurred, but it looks like Microsoft's implementation of std::vector::resize() was compliant until the standard was revised.

[yckx]

A fix to your problem is to compile your application as a 64 bit compile and you will not receive any errors.

I was under the impression that one could not compile for 64-bit applications with the Express edition of MSVC.

[AllEightUp]

Thanks for the clarification. Standards aside, OP as well as many of us find ourselves needing dynamic arrays of aligned data, so this is an issue which needs a solution.

I completely agree that we need a solution, sorry if I sounded like I was defending some gallant adherence to standards, that's not the case at all. I actually use a custom vector container because of these problems. If STL had been written with a policy based front end (i.e. not just the allocator but also internal behavior), it would have greatly helped correct the deficiencies and allow stl containers to adapt better to ongoing changes and requirements in programming. Unfortunately they were not and now they are starting to show deficiencies. (Personally, I thought they had a LOT of deficiencies for optimization reasons and hence wrote my own in many places as drop in replacements when it makes sense.)

Look at the basic problems with std::vector as examples:

1. Pod items are detected automatically and in cases where something "looks" non-pod but could be treated as pod, you can't do anything to change the STL behavior of copy construct/delete, which in "many" cases is stupid overhead. Deal with it or use a non-stl container.
2. No way to define stable/non-stable erasure. 90% of the time I don't care about stable erasures in a vector and as such some STL variations supply "fast_erase" or related functions which just move the last element to the erased elements location instead of shifting everything down by one. Of course, unless you target a single platform, you can't always use this and that causes big performance differences between platforms/libraries.
3. POD detection is so flakey given the lack of decent metadata in C++ that a huge portion of the time STL guesses wrong and by definition "most" of your std::vector's are doing stupid work behind your back.

This is a huge problem with C++ that needs to be corrected to make stl something even performance critical applications can/should use. Adding the concept of "alignment" requirements to C++ is also extremely important.