Sign in to follow this  

OpenGL TSM freakout

Recommended Posts

godmodder    828
Hello, I've tried to implement TSM shadows really hard for a long time. I've read all the papers and tutorials on the net about the subject (not much). Nvidia has a nice demo with source, but it's all in direct3d and the code really gives me a headache. My question: does somebody now a tutorial or explanation on how to do TSM shadow mapping in OpenGL? Maybe someone can send me there code (I've read on the forums here that some people have done it with succes) I'd really appreciate your help, Thanx in advance, Jeroen

Share this post

Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
Have you already looked at the TSM recipe page

It contains some source code in OpenGL for TSM.

Share this post

Link to post
Share on other sites
godmodder    828

yes, I had already looked at it. It contains source to compute the trapezoidal transformation. (the easier bit)
I'm having problems with the steps before the transformation. The problem is the following: I've computed my 8 points of the eye-frustum, but I need to transform those points into the post-perspective space of the light and scale them to the unit cube in opengl. I thought you one could do it with gluLookAt() and glOrtho(), but frankly, I have no idea on how to do it exactly in opengl.
When I look at the sourcecode in direct3D on the nvidia SDK, I understand every bit they're doing, but I can't reproduce it in opengl.

Can someone help me with this?

Share this post

Link to post
Share on other sites
Yann L    1802
How do they do it in the D3D sample ? They're probably using D3DX ?

Frankly, I wouldn't use builtin OpenGL functions to accomplish this. Just do the matrix math manually, either with a homebrew matrix library, or one of the gazillion third party ones floating around the net.

Share this post

Link to post
Share on other sites
godmodder    828
1st problem: they first calculate a lightspace basis by using: D3DXVec3TransformNormal(), and some other functions like Cross() etc...
What does D3DXVec3TransformNormal() exactly do? Does it transform the vector just by a rotation matrix or by a normal matrix? (normal matrix is the transpose of the inverse of the modelview matrix)

2nd problem: is D3DXVec3TransformCoordArray() the same as multiplying your vector with a rotation matrix?

3rd problem: do I have to calculate the scene's bounding box and transform all of the vertices of my world into eye-space to get all shadow casters (my world) in front of the near plane? That couldn't be true, right? (well... they do it in the sample, while they explicitly state in the TSM paper that NO scene analysis is required in order for TSM to work properly.

4th problem: they use D3DXMatrixOrhtoOffCenterLH() to produce a ortho projection matrix. Can I use glOrtho() to do this in opengl or do I have to calculate that matrix manually and transform the points like in problem 2?

I've been scrathing my head all day,
The rest of the code is cristal clear...

Share this post

Link to post
Share on other sites
paic    645

1) It simply assumes you're transforming a normal. So you won't get the translation information of your matrix (it multiplies (x, y, z, 0) with your matrix, instead of (x, y, z, 1) in the case of D3DXVec3TransformCoord())

2) D3DXVec3TransformCoordArray() multiply each element of the array by the givent matrix. It's exactly the same as :

for (int i = 0; i < size; i++)
D3DXVec3TransformCoord(&output[i], &input[i], &matrix);

3 & 4, I don't know, sorry ^^

Share this post

Link to post
Share on other sites
HellRaiZer    1001

First of all the nVidia demo in on PSM and not TSM. This is an other technique.

Here is the code i used to get it working, but i couldn't find a demo using it. I hope it works (i don't remeber how exactly, but i remember that it had worked)!

void CalcEyeFrustumInObjSpace(GEVector3D* p, GECamera* cam)
float t = cam->GetZNear() * tan(geDegToRad(cam->GetFOV()) * 0.5f);
float b = -t;
float r = cam->GetAspect() * t;
float l = -r;
float n = cam->GetZNear();
float f = cam->GetZFar();
float fn = f / n;
float lFar = fn * l;
float rFar = fn * r;
float bFar = fn * b;
float tFar = fn * t;

p[0].Set(l, b, -n);
p[1].Set(r, b, -n);
p[2].Set(r, t, -n);
p[3].Set(l, t, -n);

p[4].Set(lFar, bFar, -f);
p[5].Set(rFar, bFar, -f);
p[6].Set(rFar, tFar, -f);
p[7].Set(lFar, tFar, -f);

void CalcEyeFrustumInL(GEVector3D* eyeFr, GEVector3D* eyeFrInL, GECamera* eye)
GEMatrix4x4 lookAt, proj, eyeProj, camLookAt_inv;
camLookAt_inv.LookAt(eye->Origin, eye->Dir, eye->UpVector);

Light->LightView.LookAt(Light->GetCamera()->Origin, Light->GetCamera()->Dir, Light->GetCamera()->UpVector);
Light->LightProj.Perspective(Light->GetCamera()->GetFOV(), 1.0f, Light->GetCamera()->GetZNear(), Light->GetCamera()->GetZFar());

eyeProj.Multiply(Light->LightProj, Light->LightView);
eyeProj.Multiply(eyeProj, camLookAt_inv);

// eyeFrustumInLightSpace = P_L * C_L * C_E^-1 * eyeFrustumInObjSpace
// Chapter 3, last par. : The eight corner vertices of E are obtained from
// those corner vertices of the eye's frustum in the object space multiplied
// by PL .CL .CE^-1 where CE^-1 is the inverse camera matrix of the eye. We treat
// E as a flattened 2D object on the front face of the light's unit cube.
IsEyeFrustumInLightFrustum = true;
for(int i=0;i<8;i++)
eyeProj.TransformHomogenPoint(eyeFr[i], eyeFrInL[i]);
eyeFrInL[i].z = 1.0f;

if(fabsf(eyeFrInL[i].x) > 1.0f || fabsf(eyeFrInL[i].y) > 1.0f)
IsEyeFrustumInLightFrustum = false;


glColor3f(1.0f, 0.0f, 0.0f);
glColor3f(0.0f, 1.0f, 0.0f);

void CalcCenterLine(GEVector3D* eyeFrInL, GEVector3D* l)
GEVector3D v1 = eyeFrInL[1] - eyeFrInL[0];
GEVector3D v2 = eyeFrInL[2] - eyeFrInL[3];

v1 = eyeFrInL[0] + v1 * 0.5f;
v2 = eyeFrInL[3] + v2 * 0.5f;

l[0] = v1 + (v2 - v1) * 0.5f;

v1 = eyeFrInL[5] - eyeFrInL[4];
v2 = eyeFrInL[6] - eyeFrInL[7];
v1 = eyeFrInL[4] + v1 * 0.5f;
v2 = eyeFrInL[7] + v2 * 0.5f;

l[1] = v1 + (v2 - v1) * 0.5f;


void CalcTrapezoid(GEVector3D* hull, int numHullPoints, GEVector3D* line, GEVector3D* trapezoid)
// We need an axis orthogonal to center line.
// Because all the points of the hull are on the z = 1.0f plane, we can take
// the cross product of the center line direction with the z axis vector
// as the perpendicular axis.
// FIX : What happens if line[1] = line[0]??? Center line has zero length,
// and we can't find a valid perpendicular axis! This may happen when we
// are near the dualling frusta case.
GEVector3D perpAxis = line[1] - line[0];
if(perpAxis.Length() < 1e-5f)

// This is because we always want the perpAxis to point to +x world axis.
// The normal way to do it is to change the vectors in the cross product,
// but this makes things "symmetrical"!!!
if(perpAxis.x < 0.0f)
perpAxis = perpAxis | GEVector3D(0.0f, 0.0f, 1.0f);
perpAxis = perpAxis | GEVector3D(0.0f, 0.0f, -1.0f);

// If we transform all the points of the hull, so that the above calculated
// perpendicular axis is the x - axis, and the center line is the y - axis,
// we can calculate an axis-aligned bounding box of hull's points. With this
// aabb we can calculate how much we must travel along the center line in case to
// draw the top and the base lines that encloses the whole convex hull.
// We also translate the the hull so the center line starts at (0,0,0).
GEMatrix4x4 rotz, trans;
rotz.MakeRotationMatrix('z', geRadToDeg(-acos(perpAxis.x)));
trans.MakeTranslationMatrix(-line[0].x, -line[0].y, -line[0].z);
rotz.Multiply(rotz, trans);

GEVector3D minv(2.0f, 2.0f, 1.0f), maxv(-2.0f, -2.0f, 1.0f);
GEVector3D transHull[6];

for(int i=0;i<numHullPoints;i++)
rotz.TransformPoint(hull[i], transHull[i]);

// We check only y component, because z is always 1.0f, and we are interested,
// in the y-direction only. This is, we only want the top and the bottom of the
// aabbox.
if(transHull[i].y > maxv.y) maxv.y = transHull[i].y;
if(transHull[i].y < minv.y) minv.y = transHull[i].y;

// We do this (the min/max thing for top_y and base_y) because, we always want to
// have the top line touch the near plane, and the base line touch the far plane.
float topy = min(fabs(minv.y), fabs(maxv.y));
float basey = max(fabs(minv.y), fabs(maxv.y));

GEVector3D clineDir = line[1] - line[0];
float cLineLen = clineDir.Length();

GEVector3D topLine[2];
GEVector3D baseLine[2];
GEVector3D temp;

// We move along the center line, starting from the point on the near plane,
// by a distance of -miny to reach the outer point of the hull. Then with the
// perpendicular axis, we form the top line.
temp = line[0] - clineDir * topy;
topLine[0] = temp + perpAxis;
topLine[1] = temp - perpAxis;

// The same as above, but now we must move +maxy to reach the cross point of
// the center line and the base line.
temp = line[0] + clineDir * basey;
baseLine[0] = temp + perpAxis;
baseLine[1] = temp - perpAxis;

// Calculate distance of point q from the top line ('&#951;' in the paper)
// Chapter 6.2.
float lineFactor = 0.8f; // 80% line
float ksi = 1.0f - 2.0f * lineFactor; // &#958; in the paper
float lamda = cLineLen + fabs(basey) + fabs(topy); // distance between the top and the base line
float delta = 1.0f - ksi;// + topy;
float n = (lamda * delta + lamda * delta * ksi) / (-lamda - 2.0f * delta - lamda * ksi);

// Calculate q
temp = line[0] - clineDir * topy; // temp is the cross between center line and top line.
GEVector3D q = temp + clineDir * n;

// Find the two points of the hull which combined with q give us the side
// edges of the trapezoid. These points, must be points of the hull.

// For every point on the hull, form a line from q to the point. We know that
// all points lie on z = +1 plane. So, from this line, we form a plane, and we
// check every other point on the hull against it. In case to have a winner,
// all other points must lie on the same side of the plane.
int index[2] = {-1, -1};
int curIndex = 0;
GEPlane plane(hull[i], (q - hull[i]) | GEVector3D(0.0f, 0.0f, 1.0f));

int numPos = 0, numNeg = 0;
for(int j=0;j<numHullPoints;j++)
if(i == j)
float dist = plane.ClassifyPointf(hull[j]);
if(dist >= 0.0f)

if((numPos > 0 && numNeg == 0) || (numPos == 0 && numNeg > 0))
index[curIndex++] = i;
if(curIndex > 2)

if(curIndex != 2)
MessageBox(NULL, "Severe Error : Couldn't find to extreme point for calculating the trapezoid.", "ERROR", MB_OK);

GEVector3D pMin, pMax;
if(transHull[index[0]].x > transHull[index[1]].x)
pMax = hull[index[0]];
pMin = hull[index[1]];
pMax = hull[index[1]];
pMin = hull[index[0]];

geIntersectLines2D(trapezoid[0], baseLine[0], baseLine[1], q, pMin);
geIntersectLines2D(trapezoid[1], baseLine[0], baseLine[1], q, pMax);
geIntersectLines2D(trapezoid[2], topLine[0], topLine[1], q, pMax);
geIntersectLines2D(trapezoid[3], topLine[0], topLine[1], q, pMin);

// Check trapezoid points order.
GEVector3D dir1 = trapezoid[1] - trapezoid[0];
GEVector3D dir2 = trapezoid[2] - trapezoid[0];
dir1 = dir1 | dir2;
if(!(dir1 == GEVector3D(0.0f, 0.0f, 1.0f)))
dir1 = trapezoid[0];
trapezoid[0] = trapezoid[1];
trapezoid[1] = dir1;

dir1 = trapezoid[3];
trapezoid[3] = trapezoid[2];
trapezoid[2] = dir1;

// Debug visualization
glColor3f(0.0f, 0.0f, 0.0f);
glColor3f(1.0f, 0.0f, 0.0f);
glColor3f(0.0f, 1.0f, 0.0f);
glColor3f(0.0f, 0.0f, 1.0f);
glColor3f(1.0f, 1.0f, 1.0f);
glColor3f(1.0f, 1.0f, 1.0f);

glColor3f(1.0f, 0.0f, 0.0f);
glColor3f(0.0f, 1.0f, 0.0f);

glColor3f(0.5f, 1.0f, 0.5f);
glColor3f(1.0f, 1.0f, 1.0f);

And the function order is :

GEVector3D eyeFrustum [8];
GEVector3D eyeFrustumInL [8];
GEVector3D line [2];
GEVector3D hull [6];
GEVector3D trapezoid [4];
GEMatrix4x4* GELight::N_T;

CalcEyeFrustumInObjSpace(&eyeFrustum[0], Camera);
CalcEyeFrustumInL(&eyeFrustum[0], &eyeFrustumInL[0], Camera);
CalcCenterLine(&eyeFrustumInL[0], &line[0]);
numHullPoints = CalcConvexHull2D(&eyeFrustumInL[0], &hull[0]);
RenderConvexHull(&hull[0], numHullPoints);
CalcTrapezoid(&hull[0], numHullPoints, &line[0], &trapezoid[0]);
CalcN_T(&trapezoid[0], &Light->N_T, &eyeFrustum[0]);

I think the code is clear enough for you to understand what's going on. If there is any problem with the naming of variables, or something doesn't look right say it.


PS. CalcConvexHull2D() is in Graphic Programming Gems. I couldn't find it in my code, so this isn't included.

Share this post

Link to post
Share on other sites
Guest Anonymous Poster   
Guest Anonymous Poster
The nvidia demo is on PSM, but there is a drop-down where you select TSM within the demo.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Similar Content

    • By povilaslt2
      Hello. I'm Programmer who is in search of 2D game project who preferably uses OpenGL and C++. You can see my projects in GitHub. Project genre doesn't matter (except MMO's :D).
    • By ZeldaFan555
      Hello, My name is Matt. I am a programmer. I mostly use Java, but can use C++ and various other languages. I'm looking for someone to partner up with for random projects, preferably using OpenGL, though I'd be open to just about anything. If you're interested you can contact me on Skype or on here, thank you!
      Skype: Mangodoor408
    • By tyhender
      Hello, my name is Mark. I'm hobby programmer. 
      So recently,I thought that it's good idea to find people to create a full 3D engine. I'm looking for people experienced in scripting 3D shaders and implementing physics into engine(game)(we are going to use the React physics engine). 
      And,ye,no money =D I'm just looking for hobbyists that will be proud of their work. If engine(or game) will have financial succes,well,then maybe =D
      Sorry for late replies.
      I mostly give more information when people PM me,but this post is REALLY short,even for me =D
      So here's few more points:
      Engine will use openGL and SDL for graphics. It will use React3D physics library for physics simulation. Engine(most probably,atleast for the first part) won't have graphical fron-end,it will be a framework . I think final engine should be enough to set up an FPS in a couple of minutes. A bit about my self:
      I've been programming for 7 years total. I learned very slowly it as "secondary interesting thing" for like 3 years, but then began to script more seriously.  My primary language is C++,which we are going to use for the engine. Yes,I did 3D graphics with physics simulation before. No, my portfolio isn't very impressive. I'm working on that No,I wasn't employed officially. If anybody need to know more PM me. 
    • By Zaphyk
      I am developing my engine using the OpenGL 3.3 compatibility profile. It runs as expected on my NVIDIA card and on my Intel Card however when I tried it on an AMD setup it ran 3 times worse than on the other setups. Could this be a AMD driver thing or is this probably a problem with my OGL code? Could a different code standard create such bad performance?
    • By Kjell Andersson
      I'm trying to get some legacy OpenGL code to run with a shader pipeline,
      The legacy code uses glVertexPointer(), glColorPointer(), glNormalPointer() and glTexCoordPointer() to supply the vertex information.
      I know that it should be using setVertexAttribPointer() etc to clearly define the layout but that is not an option right now since the legacy code can't be modified to that extent.
      I've got a version 330 vertex shader to somewhat work:
      #version 330 uniform mat4 osg_ModelViewProjectionMatrix; uniform mat4 osg_ModelViewMatrix; layout(location = 0) in vec4 Vertex; layout(location = 2) in vec4 Normal; // Velocity layout(location = 3) in vec3 TexCoord; // TODO: is this the right layout location? out VertexData { vec4 color; vec3 velocity; float size; } VertexOut; void main(void) { vec4 p0 = Vertex; vec4 p1 = Vertex + vec4(Normal.x, Normal.y, Normal.z, 0.0f); vec3 velocity = (osg_ModelViewProjectionMatrix * p1 - osg_ModelViewProjectionMatrix * p0).xyz; VertexOut.velocity = velocity; VertexOut.size = TexCoord.y; gl_Position = osg_ModelViewMatrix * Vertex; } What works is the Vertex and Normal information that the legacy C++ OpenGL code seem to provide in layout location 0 and 2. This is fine.
      What I'm not getting to work is the TexCoord information that is supplied by a glTexCoordPointer() call in C++.
      What layout location is the old standard pipeline using for glTexCoordPointer()? Or is this undefined?
      Side note: I'm trying to get an OpenSceneGraph 3.4.0 particle system to use custom vertex, geometry and fragment shaders for rendering the particles.
  • Popular Now