I'm working on a mesh converter (from OBJ to custom binary format).

I've been trying to get tangent vector calculation to work properly but there were some issues with my code, so I switched to NVMeshMender (though without the "d3d9" dependency, I changed the source slightly to use glm).

Problem is, it's still not working, but not even without the tangents. Whether I instruct NVMeshMender to (re)calculate the normals or not, it introduces artifacts in the raw normals, which I did not have before.

I load the OBJ file line-by-line, collect unique vertices (which have equal V, N, UV indices), then run MeshMender on this data. I store the normals and tangents packed in GL_INT_2_10_10_10_REV format. Strange thing is, only a couple of normals are messed up, and it seems to be random.

Here's a screenshot (note it might seem too dark but that's because I forgot to turn off the tone mapping pass).

The Iron Man model does not seem to have artifacts, but nearly every other model seems to do.

By posting this I'm hoping that somebody will recognize this kind of artifact and can point me in the right direction.

Any help is greatly appreciated!

EDIT: When I also export the normals from Blender in the OBJ, the results are slightly different but the artifacts are still present (e.g. the monkey's eyes have more artifacts horizontally)...

Does all parts of the mesh have unique uv coordinates? To me it looks a bit like a normal map that has been generated with a mesh that has overlapping uv coordinates.

This shouldn't have to do anything with UV-s, if I'm not mistaken. These are ONLY the "raw" view-space normal vectors on the screenshots; they simply come from the vertex attribute array.

Just to be sure, I tried with the same mesh but all UVs removed, and I'm seeing the same thing. :( ... :)

Ok, do you normalize the normals before storing and using them?

I use this code to generate normal, tangent and binormal, perhaps you can make use of it and see if the result is different:

void CalculateNormalTangentBinormal(D3DXVECTOR3* P0, D3DXVECTOR3* P1, D3DXVECTOR3* P2, 
                              D3DXVECTOR2* UV0, D3DXVECTOR2* UV1, D3DXVECTOR2* UV2,
                              D3DXVECTOR3* normal, D3DXVECTOR3* tangent, D3DXVECTOR3* binormal)
{
    D3DXVECTOR3 P = *P1 - *P0;
    D3DXVECTOR3 Q = *P2 - *P0;
    //Cross product
    normal->x = P.y * Q.z - P.z * Q.y;
    normal->y = P.z * Q.x - P.x * Q.z;
    normal->z = P.x * Q.y - P.y * Q.x;

    float s1 = UV1->x - UV0->x;
    float t1 = UV1->y - UV0->y;
    float s2 = UV2->x - UV0->x;
    float t2 = UV2->y - UV0->y;

    float tmp = 0.0f;
    if (fabsf(s1 * t2 - s2 * t1) <= 0.0001f) tmp = 1.0f;
    else tmp = 1.0f / (s1 * t2 - s2 * t1 );

   tangent->x = (t2 * P.x - t1 * Q.x) * tmp;
   tangent->y = (t2 * P.y - t1 * Q.y) * tmp;
   tangent->z = (t2 * P.z - t1 * Q.z) * tmp;

   binormal->x = (s1 * Q.x - s2 * P.x) * tmp;
   binormal->y = (s1 * Q.y - s2 * P.y) * tmp;
   binormal->z = (s1 * Q.z - s2 * P.z) * tmp;

   D3DXVec3Normalize(normal, normal);
   D3DXVec3Normalize(tangent, tangent);
   D3DXVec3Normalize(binormal, binormal);
}

Wow, thanks, your answer made me realize something!

I used to have similar code which I replaced with the MeshMender, so now I tried reverting back (and telling the mender not to recalculate normals) and to my surprise it had the same artifacts! I then realized there's something else I changed at the same time....

Which is the GL_INT_2_10_10_10_REV pack logic... With my old version, it works fine -- unfortunately I wrote that a long time ago and now I realize it relies on undefined behavior. :(

http://stackoverflow.com/questions/3784996/why-does-left-shift-operation-invoke-undefined-behaviour-when-the-left-side-oper

return U32
(
    (((I32)(w)          &    3) << 30) |
    (((I32)(z * 511.0f) & 1023) << 20) |
    (((I32)(y * 511.0f) & 1023) << 10) |
     ((I32)(x * 511.0f) & 1023)
);

Now I'm trying to come up with something which isn't undefined... Stay tuned :) Or if anybody has some code I'd be grateful!

Partial success - I managed to come up with the below, which works with normals :) But now I see issues with tangents, probably because of some other issue I have.

U32 packNormalizedFloat_2_10_10_10_REV(float x, float y, float z, float w)
{
    const U32 xs = x < 0;
    const U32 ys = y < 0;
    const U32 zs = z < 0;
    const U32 ws = w < 0;

    return U32
    (
        ws << 31 | ((U32)(w       + (ws << 1)) &   1) << 30 |
        zs << 29 | ((U32)(z * 511 + (zs << 9)) & 511) << 20 |
        ys << 19 | ((U32)(y * 511 + (ys << 9)) & 511) << 10 |
        xs <<  9 | ((U32)(x * 511 + (xs << 9)) & 511)
    );
}

Here are the tangent vector artifacts:

This is now resolved. The problem wasn't with NVMeshMender of course... :)

I had several mistakes all working together against me.

Hope someone finds my GL_INT_2_10_10_10_REV pack function useful - since I personally couldn't find anything with Google!