# Odd normal transform in Cg

This topic is 3645 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I've decided to try to make a sincere effort to learn shader programming. I was using Cg to write a simple shader that did Blinn-Phong in Eye-Space for one light. It was easy in GLSL, but I've hit a snag with Cg. I had issues where certain regions of the camera transform could cause the lighting to "flicker". I narrowed the problem down to the transformation of my normals using another shader that just displays the normalized, eye-space normal vector as the fragment's color. Sure enough, the normal transformation was working properly for most camera transformations. (And by working properly, I mean that the normals appeared to face a sane direction and rotated as I moved the camera around them, as eye-space normals should). But for some camera transforms, the normals were being garbled. I'm a little weak on my linear algebra, but it almost feels as if my modelview matrix ends up being non-invertible for some transforms and was causing some sort of singularity or discontinuity. As far as I can tell, I am only translating, rotating, and _uniformly_ scaling my modelview matrix. Like I mentioned, I'm weak on the maths, but shouldn't this class of matrix always be invertible? Note the code below where I set the shader parameter for the inverse-transpose-modelview matrix. I was originally using the statement in the upper branch of the preprocessor if-statement. I was letting Cg grab the matrix and perform operations on it automagically. However, as a test, I decided to see what would happen if I inverted and transposed the matrix myself-- The is the code in the lower branch. To my relief, the modelview matrix is indeed invertible. This code block also does not suffer from discontinuities or strange behaviors. However, the resulting normal transformation does not appear to transform the normals into eye-space at all. And by that, I mean that the results are identical to displaying the normals as object-space normals with no transformation. Why might the CG command result in a different matrix? Am I correct that object-space normals need to be transformed by the inverse-transpose of the modelview matrix in order to be projected from object-space into eye-space?
#if ORIGINAL
// How I originally was filling out the normal matrix.
// This causes very strange behavior.
cgGLSetStateMatrixParameter(cgViewIT,
CG_GL_MODELVIEW_MATRIX,
CG_GL_MATRIX_INVERSE_TRANSPOSE);
#else
// Test case to see what happens when I handle the operation personally
// Results in very different behavior from the above.
{
mat4 View, ViewI, ViewIT, ViewITT, I;

I.identity();

glGetFloatv(GL_MODELVIEW, View);

ViewI = View;
ViewI.invert();
ASSERT(ViewI * View == I, "Not proper inverse");

ViewIT = ViewI;
ViewIT.transpose();
ViewITT = ViewIT;
ViewITT.transpose();
ASSERT(ViewITT == ViewI, "Not proper transpose");

cgGLSetMatrixParameterfc(cgViewIT, ViewIT);
}
#endif

Any idea what might be the problem here? Or at least thoughts on what I might want to look into next? I have a suspicion that the solution is either subtle or involves some fundamental misunderstanding. Thanks for taking the time to read this! P.S. I checked my matrix utility classes, and I'm pretty sure they're good. Not that I consider them to be above reproach...

##### Share on other sites
I have no idea if this will work because I do D3D+HLSL, but I notice that there's a cgGLGetMatrixParameter() function. You could try setting it then getting it and comparing it against the matrix you are generating and see how they differ.

HTH.

##### Share on other sites
Today, I was able to get the shader to work flawlessly by performing all lighting in object-space, avoiding the transformation of the normals altogether. My next step is to light in surface-local space (for normal mapping), so I don't really *need* a solution, but this problem is still bothering me.

Quote:
 I have no idea if this will work because I do D3D+HLSL, but I notice that there's a cgGLGetMatrixParameter() function. You could try setting it then getting it and comparing it against the matrix you are generating and see how they differ.

I'll give it a try.

EDIT:

So, I wrote some unit tests and was able to find and fix a small bug in my 4x4 matrix inversion code. Now, the normals are transformed properly when I calculate the inverse-transpose matrix myself -- instead of letting Cg handle the transform -- and lighting in eye-space works fine.

When I use cgGLGetMatrixParameter to retrieve the matrix that Cg is generating, I find it is grossly different from the matrix I am calculating (and assume is correct). Honestly, wth? Could this be a bug in the Cg library?

In any case, it seems that as long as I stay away from the cgGLSetStateMatrixParameter function, I should be fine.

1. 1
2. 2
Rutin
18
3. 3
4. 4
5. 5

• 14
• 12
• 9
• 12
• 37
• ### Forum Statistics

• Total Topics
631428
• Total Posts
3000027
×