# Running x86 build crashes

## Recommended Posts

I have 4 build configurations in Visual Studio 2017 15.5.2 (/std:c++latest) that build without a single error or warning (though, I suppressed some warnings about adding padding for alignment and about the usage of anonymous structs): Release|Debug x64 and Release|Debug x86. All the builds run fine except for Release x86 which crashes at some weird fixed location (in fact two weird fixed locations, since I have alternatives for my input file formats).

The output mentions a very informative message:

Quote

**__that** was 0x1.

Some of my C++ Locals (std::strings) had a value equal to some of my HLSL statements and comments, but that does not seem to repeat itself.

I suspect an alignment bug (my x64 builds never had a problem), but I have no idea how to track these down?

• I checked if all my struct/classes that are declared alignas are allocated with a custom allocator when stored in std::vector.
• All structs/classes storing XMVECTOR/XMMATRIX data are declared alignas(16).
• Furthermore, I checked all my call conventions for functions with XMVECTOR/XMMATRIX return or input argument types (this fixed the Debug x86 build, which didn't crash but produced weird shadows).

Any ideas?

##### Share on other sites

Are you generating dump files?  Adding crash handlers that call MiniDumpWriteDump() is incredibly useful if you are keeping the pdb files around. By saving all the memory with the right flags you can see exactly where the program was and exactly what was in memory when it died, including full stacks for each thread.

Without tools like that it is mostly guesswork.  You guessed several items, alignment issues, structure packing, wrong calling conventions, differences between structures in different builds.  Maybe any of those, maybe none of those. Get an actual dump file so you can verify for certain, or find steps to reproduce it inside a debugger.

##### Share on other sites

Try gradually enabling the same optimisation settings in your debug build that you use in your release build to see if you can catch the issue in the debugger - you might be able to catch it that way.

##### Share on other sites
9 hours ago, C0lumbo said:

Try gradually enabling the same optimisation settings in your debug build that you use in your release build to see if you can catch the issue in the debugger - you might be able to catch it that way.

Unfortunately, the MVC++ compiler flags triggering the problem are anything except /Od (no optimizations) + /GL (Whole Program Optimization). So the /GL flag is the culprit, but cannot be debugged properly with Visual Studio itself.

I am going to try the file dump method later on (reminder to self: http://www.debuginfo.com/examples/effmdmpexamples.html)

Edited by matt77hias

##### Share on other sites

I couldn't use __try due to the object unwinding. So I switched /EHsc to /EHa and put all my startup code in a noexcept function. The program crashes with an "exception" (SEH?), though my crash dump function is never called?

__try {
Run(hinstance, nCmdShow);
}
__except (CreateMiniDump(GetExceptionInformation()), EXCEPTION_EXECUTE_HANDLER) {}

##### Share on other sites

There are many other ways to do it because there are many different reasons for programs to get killed.

You can set an unhandled exception handler for your program, std::set_terminate() for C++ exceptions. If you use Windows Structured Exceptions you also need SetUnhandledExceptionFilter(). Depending on how your program dies you may want to register functions with std::atexit() and/or std::at_quick_exit().

In current versions of Windows you can also set some registry values to turn on Windows Error Reporting (WER) to generate crash dumps even if those above methods don't catch it.

##### Share on other sites
11 hours ago, frob said:

Windows Structured Exceptions you also need SetUnhandledExceptionFilter()

But in my specific case, I need to look into this one? (Because the noexcept could not leak C++ exceptions; i.e. the program would just terminate instead of crash).

##### Share on other sites
12 hours ago, frob said:

There are many other ways to do it because there are many different reasons for programs to get killed.

Ok apparently both

LONG WINAPI UnhandledExceptionFilter(EXCEPTION_POINTERS *exception_record) {
CreateMiniDump(exception_record);
return EXCEPTION_CONTINUE_SEARCH;
}

SetUnhandledExceptionFilter(UnhandledExceptionFilter);
}

and

__except (CreateMiniDump(GetExceptionInformation()), EXCEPTION_EXECUTE_HANDLER) {}

work. Visual Studio was interfering with the exception handling. When I just run the .exe outside Visual Studio, it crashes and generates the dump file. The former does not require __try/__catch and so I can get rid of /EHa, furthermore you get some crash message. The latter does not result in some crash message since the program will catch everything..

Edited by matt77hias

##### Share on other sites

@frob I used a very verbose mini dump including MiniDumpWithFullMemory | MiniDumpWithFullMemoryInfo | MiniDumpWithHandleData | MiniDumpWithThreadInfo | MiniDumpWithUnloadedModules and generated the mini dump (168 MB), but how does one do something useful with the file? The default program for opening the .dmp file is Visual Studio itself. The only "useful" action seems "Debug with Native Only", though that results in the same info Visual Studio gave in the first place?

As a side note, SIMD intrinsics are not the problem, since the program still crashes with the same error after disabling them. This begins to make me really suspicious (again) towards the compiler/linker itself.

Edited by matt77hias

##### Share on other sites

Did you check the call stack that is shown with your minidump? That seems to point to a very concrete location, and should probably allow you to spot the error (which seems to relate to eigther vector pushback or your ModelPart-move-ctor).

##### Share on other sites
1 hour ago, Juliean said:

Did you check the call stack that is shown with your minidump? That seems to point to a very concrete location, and should probably allow you to spot the error (which seems to relate to eigther vector pushback or your ModelPart-move-ctor).

The call stack seems not to make much sense (it also does not match the error location in the editor: at the final } of a not used header). My ModelPart looks something like this:

struct alignas(16) ModelPart final {
AABB m_aabb; // 2x XMVECTOR = 2x __m128; AABB uses alignas(16)
BS m_bs;     // 1x XMVECTOR = 1x __m128; BS uses alignas(16)
F32x3 m_translation;
F32x3 m_rotation;
F32x3 m_scale;
U32 m_start_index;
U32 m_nb_indices;
std::string m_child;
std::string m_parent;
std::string m_material;
}

The __m128 do not impose a problem, if I disable intrinsics and use 4x floats instead of __m128, the crash persists. Furthermore, the copy/move constructors and copy/move assignment operators are all present but made explicitly default.

The AddModelPart method looks like this:

template < typename VertexT >
inline void ModelOutput< VertexT >::AddModelPart(ModelPart model_part, bool create_bounding_volumes) {
m_model_parts.push_back(std::move(model_part));

if (create_bounding_volumes) {
SetupBoundingVolumes(m_model_parts.back());
}
}

which is invoked by:

template < typename VertexT >
ModelPart model_part;
//
//
}

With regard to the std::vector, I use my own allocator to ensure the alignment of 16 bytes. Though, if I just use std::vector, the program crashes at the same location with the same error.

Edited by matt77hias

##### Share on other sites

One thing worth noting, is that on x86 the standard Windows memory allocator only returns 8 bytes aligned memory. On x64 it's 16 byte aligned. That could be causing you problems. I'd suggest overriding global operator new and delete (don't forget the array and nothrow variants) to use _aligned_malloc() to force 16 byte alignment for everything.

Having said that, if you want to debug a crash in optimized code, then what you really need to be looking at is the disassembly.

I'd also recommend using the /Zo compiler option (which requires edit and continue to be disabled).

In addition, you want to disable the "just my code" option. This will let you see more details on what's going on as you have some external code in your crash callstack.

I also wouldn't fully trust the diagnosis of "_that is 0x1". I have seen incorrect messages there before, especially for uncommon crash reasons like data misalignment. Looking at the assembly instruction that it crashed on, and the registers used to calculate the memory address that it's accessing can give a more reliable answer.

If you're not familiar with reading x86 assembly, then I found a basic guide at: http://www.cs.virginia.edu/~evans/cs216/guides/x86.html. You probably won't need to read past the section on data movement instructions to get started with debugging your crash. If need be you could, post the contents of the disassembly window and registers window here for someone else to look at.

##### Share on other sites

One thing worth noting, is that on x86 the standard Windows memory allocator only returns 8 bytes aligned memory. On x64 it's 16 byte aligned. That could be causing you problems. I'd suggest overriding global operator new and delete (don't forget the array and nothrow variants) to use _aligned_malloc() to force 16 byte alignment for everything.

I added these to some TU, but the problem remains.

void *operator new(size_t size) {
void * const ptr = _aligned_malloc(size, 16);
if (!ptr) {
}

return ptr;
}

void operator delete(void *ptr) noexcept {
_aligned_free(ptr);
}

void *operator new[](size_t size) {
return operator new(size);
}

void operator delete[](void *ptr) noexcept {
operator delete(ptr);
}

Edited by matt77hias

##### Share on other sites

I'd also recommend using the /Zo compiler option (which requires edit and continue to be disabled).

I do not find the exact documentation for Visual Studio 2017. Visual Studio 2015:

The /Zo option is enabled by default in Visual Studio 2015 when you specify debugging information with /Zi or /Z7.
Specify /Zo- to explicitly disable this compiler option.

I use /Zi for my release builds, so I presume /Zo will be included.

In addition, you want to disable the "just my code" option. This will let you see more details on what's going on as you have some external code in your crash callstack.

I now use my own Allocator in std::vector. The external calls are now replaced by the construct of my Allocator etc.:

template< typename DataU, typename... ConstructorArgsT >
void construct(DataU *data, ConstructorArgsT&&... args) const {
new ((void *)data) DataU(std::forward< ConstructorArgsT >(args)...);
}

##### Share on other sites
        m_model_output.AddModelPart(std::move(model_part));
00E0B299  lea         ecx,[ebp-1A0h]
00E0B29F  lea         eax,[model_part]
00E0B2A5  push        eax
00E0B2A6  call        mage::ModelPart::ModelPart (0E0BC30h)
00E0B2AB  mov         edi,1
00E0B2B0  mov         dword ptr [ebp-0D8h],eax
00E0B2B6  mov         dword ptr [ebp-0D4h],edi
00E0B2BC  mov         esi,dword ptr [esi+44h]
00E0B2BF  push        edi
00E0B2C0  lea         ecx,[esi+24h]
00E0B2C3  mov         byte ptr [ebp-4],4
00E0B2C7  call        std::vector<mage::ModelPart,mage::AlignedAllocator<mage::ModelPart,16> >::emplace_back<mage::ModelPart> (0E0C340h)  // CALL
00E0B2CC  cmp         byte ptr [ebp-0D8h],0
00E0B2D5  mov         eax,dword ptr [esi+2Ch]
00E0B2D8  mov         ecx,esi
00E0B2DA  sub         eax,0B0h
00E0B2DF  push        eax
00E0B2E0  call        mage::ModelOutput<mage::VertexPositionNormalTexture,unsigned int>::SetupBoundingVolumes (0E0C510h)
00E0B2E5  lea         ecx,[edi+8Ch]
00E0B2EB  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
00E0B2F0  lea         ecx,[edi+74h]
00E0B2F3  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
00E0B2F8  lea         ecx,[edi+5Ch]
00E0B2FB  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
}
00E0B300  lea         ecx,[ebp-44h]
00E0B303  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
00E0B308  lea         ecx,[ebp-5Ch]
00E0B30B  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
00E0B310  lea         ecx,[ebp-74h]
00E0B313  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy_deallocate (0E03BB0h)
00E0B318  mov         ecx,dword ptr [ebp-0Ch]
00E0B31B  mov         dword ptr fs:[0],ecx
00E0B322  pop         ecx
00E0B323  pop         edi
00E0B324  pop         esi
00E0B325  mov         ecx,dword ptr [ebp-14h]
00E0B328  xor         ecx,ebp
00E0B32F  mov         esp,ebp
00E0B331  pop         ebp
00E0B332  mov         esp,ebx
00E0B334  pop         ebx
00E0B335  ret 

Then we get in the std::vector::emplace_back and allocator part:

           _Umove_if_noexcept(this->_Myfirst(), this->_Mylast(), _Newvec);
>> 00E0C3F6  mov         edx,dword ptr [esi+8]
00E0C3F9  xor         al,al
00E0C3FB  push        ecx
00E0C3FC  push        eax
00E0C3FD  push        ecx
00E0C3FE  mov         ecx,dword ptr [esi+4]
00E0C401  push        edi
00E0C402  call        std::_Uninitialized_move_al_unchecked<mage::ModelPart *,mage::ModelPart *,mage::AlignedAllocator<mage::ModelPart,16> > (0E0D5B0h)
_CATCH_ALL
if (_Emplaced)
{
_Alty_traits::destroy(_Al, _Unfancy(_Newvec + _Oldsize));
}

_Al.deallocate(_Newvec, _Newcapacity);
_RERAISE;
_CATCH_END

And apparently, we end up at a location which has nothing to do with all the previous code:

--- d:\users\matthias\documents\visual studio 2017\projects\mage\mage\fps\src\samples\brdf\brdf_scene.cpp
}
00E0BC30  push        ebp
00E0BC31  mov         ebp,esp
00E0BC33  mov         edx,dword ptr [__that]
>> 00E0BC36  movups      xmm0,xmmword ptr [edx] 

I need some more time getting somewhat familiar with x86. But if anyone wants to take a look, be my guest

Edited by matt77hias

##### Share on other sites
8 hours ago, matt77hias said:

00E0BC33  mov         edx,dword ptr [__that]

>> 00E0BC36  movups      xmm0,xmmword ptr [edx]

Those two instructions load the edx register with the contents of the "__that" parameter to the ModelPart move constructor, and then load 16 bytes from there into xmm0. It looks like that's where it has crashed.

That isn't going to be an alignment issue, as movups (as opposed to movaps) won't crash for misaligned addresses. I'm thinking that the contents of __that is actually NULL, or otherwise an invalid pointer. You can check what the pointer is by looking at the contents of the edx register at the point it crashes.

However, since it's a reference, and not a pointer, it really shouldn't be NULL. Could your custom allocator have returned nullptr?

##### Share on other sites

Those two instructions load the edx register with the contents of the "__that" parameter to the ModelPart move constructor, and then load 16 bytes from there into xmm0. It looks like that's where it has crashed.

That isn't going to be an alignment issue, as movups (as opposed to movaps) won't crash for misaligned addresses. I'm thinking that the contents of __that is actually NULL, or otherwise an invalid pointer. You can check what the pointer is by looking at the contents of the edx register at the point it crashes.

However, since it's a reference, and not a pointer, it really shouldn't be NULL. Could your custom allocator have returned nullptr?

In case of nullptr, I would expect a bad_alloc exception earlier. I am going to double check that anyway. The problem, unfortunately, occurs with the default allocator of std::vector as well. I thought a custom one would fix the issue, since std::vector is not guaranteed to ensure the correct alignment.

I also don't get why instructions of brdf_scene.cpp (last snippet) are executed? Based on the code flow, this part should not and cannot be reached at this stage?

Does the possible nullptr also explain the 0x1 address in the error message?

BTW: a merry Christmas to you!

##### Share on other sites

Not sure if thats already been suggested or tested, but this totally sounds like a memory-misaccess happening at some earlier point, resulting at a crash later with a corrupted call stack. Try the usual tools for finding heap-corruptions (ie. microsoft application verifier) or turn on the debug pageheap (https://stackoverflow.com/questions/2470131/visual-studio-how-to-find-source-of-heap-corruption-errors) if all else fails.

##### Share on other sites

I am on a phone on a plane so please forgive my brevity!

Do some research about COMDAT folding. Debug builds hold your hand a LOT and debugging a release build requires giving up on the cushy life of debug land ;-)

Look at your move semantic implementations if you have any constructors, copy/assign/move operators, etc. anywhere near the faulty object types. I see a lot of folks crashing C++11 and later because they didn't do their homework on move semantics. The VC++2015 implementation in particular is extremely unforgiving if you don't follow the spec very exactly. Even if you don't think you are making move mistakes, you may be wrong.

Also listen to Adam_42, he is probably right about you passing a bogus pointer around. Small address values in a read access violation are basically always down to nullptr at some level, at least on Windows. Don't fall into the trap of thinking nullptr means 0.

Also when you get a callstack in the debugger that you don't understand (or really anything any time that you don't understand) please DO NOT ASSUME it is wrong, useless, or not worth including in your post. Challenge your assumptions and learn, don't turn into that person who assumes computers are magic.

##### Share on other sites
On 12/25/2017 at 12:30 AM, Adam_42 said:

That isn't going to be an alignment issue, as movups (as opposed to movaps) won't crash for misaligned addresses. I'm thinking that the contents of __that is actually NULL, or otherwise an invalid pointer. You can check what the pointer is by looking at the contents of the edx register at the point it crashes.

edx is 1.

On 12/25/2017 at 12:30 AM, Adam_42 said:

That isn't going to be an alignment issue, as movups (as opposed to movaps) won't crash for misaligned addresses. I'm thinking that the contents of __that is actually NULL, or otherwise an invalid pointer. You can check what the pointer is by looking at the contents of the edx register at the point it crashes.

On 12/25/2017 at 9:03 AM, matt77hias said:

In case of nullptr, I would expect a bad_alloc exception earlier. I am going to double check that anyway. The problem, unfortunately, occurs with the default allocator of std::vector as well. I thought a custom one would fix the issue, since std::vector is not guaranteed to ensure the correct alignment.

The pointer passed for construction to the allocator is non-nullptr.

Edited by matt77hias

##### Share on other sites
10 hours ago, ApochPiQ said:

Look at your move semantic implementations if you have any constructors, copy/assign/move operators, etc. anywhere near the faulty object types. I see a lot of folks crashing C++11 and later because they didn't do their homework on move semantics. The VC++2015 implementation in particular is extremely unforgiving if you don't follow the spec very exactly. Even if you don't think you are making move mistakes, you may be wrong.

ModelPart [Default]
- AABB [Default]
- XMVECTOR (__m128) [Rely on Microsoft]
- BS [Default]
- XMVECTOR (__m128) [Rely on Microsoft]
- F32x3 [Default]
- std::string [Rely on STD]

[Default] = default copy and move constructor, and default copy and move assignment operator

I also checked my std::move statements.

Not really sure if I need to look for something in particular? In most cases, I pass arguments by value to move assign the argument to a member variable for instance. Or I perform just the "cast" for repassing rvalue references.

Edited by matt77hias

##### Share on other sites
11 hours ago, ApochPiQ said:

Also when you get a callstack in the debugger that you don't understand (or really anything any time that you don't understand) please DO NOT ASSUME it is wrong, useless, or not worth including in your post. Challenge your assumptions and learn, don't turn into that person who assumes computers are magic.

It probably won't have anything to do with the crash location shown in the editor: brdf_scene.cpp is always the file that is automatically opened upon crashing. Visual Studio points to the closing namespace brace of that file mentioning that an exception is thrown. The stack, however, does not mention anything from brdf_scene.cpp, though if I click the topmost call I am always directed to brdf_scene.cpp (but the actual call is not in this file or anywhere close to this file?).

The crash, however, is probably still related to the stack. ModelPart is part of my model loading. If I just do not load any models in a scene, the program runs fine.

I also noticed something odd:

template< typename VertexT, typename IndexT >
ModelPart model_part;

m_model_output.AddModelPart(std::move(model_part)); // Break point at this statement
}

I put a break point at the last statement. model_part seems to have the correct data.

Now, I make sure that the next invocation is not inlined (which as a side effect results in the actual inlining of the previous method):

template< typename VertexT, typename IndexT >
__declspec(noinline) void ModelOutput< VertexT, IndexT >

m_model_parts.push_back(std::move(model_part)); // Break point at this statement

if (create_bounding_volumes) {
SetupBoundingVolumes(m_model_parts.back());
}
}


Here, the data looks completely corrupt? The 3 strings have a <NULL> value and all the U32/F32x3's have random values? This is pretty weird for my F32x3 structs containing 3x floats and have an empty constructor ensuring zero-initialization.

The bigger question, however, is how the ModelPart can become corrupted between these break points?

Edit: If I use a reference, I succeed in importing models and meshes, but then it crashes in the game loop since the mesh data is completely corrupt?

Edited by matt77hias

##### Share on other sites

Did you really make sure thats the exact line where the data is corrupt? You should make sure that its valid every step of the way (ie. before the method is called) until that very line. If thats really sure and with:

Quote

Here, the data looks completely corrupt? The 3 strings have a <NULL> value and all the U32/F32x3's have random values? This is pretty weird for my F32x3 structs containing 3x floats and have an empty constructor ensuring zero-initialization.

Eigther you do have any kind of memory corruption at a firmer point as I suggested, then the only thing you can do is figure that one out.

As an alternative possible explanation, did you try a full clean-build & recompile? Seeing as this is a template method, if you modified it somewhere along the lines, there's some issue in VS2017 that sometimes won't recompile the template-method properly. It will then procceed to use some older variant of the code, while showing you the variables and all, possibly resulting in the random garbage you get to see. I had stuff like this happen multiple time in template-heavy code, which was fixed by a recompile.

Otherwise, use Application Verifier on your programm (just turn it on for the exe and then run the debugger with Basics->Heaps checked). Does this result in a crash/break at some other point in the code?

##### Share on other sites
8 minutes ago, Juliean said:

Otherwise, use Application Verifier on your programm (just turn it on for the exe and then run the debugger with Basics->Heaps checked). Does this result in a crash/break at some other point in the code?

Is this now part of WDK?

8 minutes ago, Juliean said:

Eigther you do have any kind of memory corruption at a firmer point as I suggested, then the only thing you can do is figure that one out.

But how can the input argument be corrupt due to earlier corruption, given that it is not corrupt before passing?

8 minutes ago, Juliean said:

As an alternative possible explanation, did you try a full clean-build & recompile? Seeing as this is a template method, if you modified it somewhere along the lines, there's some issue in VS2017 that sometimes won't recompile the template-method properly. It will then procceed to use some older variant of the code, while showing you the variables and all, possibly resulting in the random garbage you get to see. I had stuff like this happen multiple time in template-heavy code, which was fixed by a recompile.

The error is consistent and still there after a clean+rebuild

Edited by matt77hias

##### Share on other sites
33 minutes ago, matt77hias said:

Is this now part of WDK?

Its part of the Windows SDK, so you should probably have it installed already. Otherwise, here is a direct download:

34 minutes ago, matt77hias said:

But how can the input argument be corrupt due to earlier corruption, given that it is not corrupt before passing?

I can't give you the technical reason, sorry. Had it happen multiple times similarly to what you describe though.

35 minutes ago, matt77hias said:

The error is consistent and still there after a clean+rebuild

Alright, then thats out of the way.