I have decided to use Sony's Vectormath from Bullet Physics, however it seems I'm having some problems with it.
Firstly, I simply swapped out all my math with Vectormath one, but rendering got broken. Eventually I noticed it's row major, not column major like I had before, so I transposed matrix, however rendering still didn't work. Then I tried to transform single vertex on CPU to see if there's anything wrong with shaders, but I got same value as outputed by shader. Therefore I came here to ask for help and figure out if I've done something wrong, or something is wrong with library. Here's single transformed vertex:
Matrix World = Matrix::identity();
Matrix View = Matrix::lookAt(Point3(1, 500, 1), Point3(0, 0, 0), Vector3(0, 1, 0)); // looking from above
Matrix Proj = Matrix::perspective(PI * 0.5f, 1, 1, 1000);
Matrix WorldViewProj = World * View * Proj;
Vector4 Vertex(-63.5f, 0, -63.5f, 1); // first vertex in my mesh
Vector4 TransformedVertex = WorldViewProj * Vertex; // result is (-88.4768, 1.32573, -31750.1, 63.5)
Mesh consists of 128 * 128 vertices, they all are between (-63.5, 0, -63.5) and (63.5, 0, 63.5) (flat heightmap), but shader output varies a lot; x is between -88 and 91, y between -83 and 80, z between -31000 and 31000, w is just negative of untransformed z.
If anyone could help me out that'd be just nice =)
Thank you in advance.