Can you post your complete code, e.g. in a zip file? Unfortunately it's really hard to diagnose something like this (since there are probably multiple contributing problems) by just seeing random snippets.

Update: I believe I might have found the problem. According to this page http://www.asawicki...._violation.html
it has something to do with memory alignment and SSE and that you cannot use XMVECTOR or XMMATRIX inside a dynamically allocated structure (like my class).
I'm currently trying to work my way around it using XMFLOAT3 and XMFLOAT4X4 to store them and I think it works. Still need some time to check.

Update2: It works! It really was the problem.

For reference, that's what I was referring to here:

I suspect the World matrix isn't aligned correctly.

Although in all fairness it's a big leap from that terse quote to the fix you've got.