# [D3D9] Casting a ray/mouse picking... [SOLVED]

## Recommended Posts

d h k    439
I'm trying to follow this tutorial and I have the function:
int SendRay ( float x, float y )
{
D3DXMATRIX matProj, matView, matWorld;

device->GetTransform ( D3DTS_PROJECTION, &matProj );
device->GetTransform ( D3DTS_VIEW, &matView );
device->GetTransform ( D3DTS_WORLD, &matWorld );

D3DXVECTOR3 v;
v.x =  ( ( ( 2.0f * x ) / ( float ) screen_width ) - 1 ) / matProj._11;
v.y = -( ( ( 2.0f * y ) / ( float ) screen_height ) - 1 ) / matProj._22;
v.z =  1.0f;

D3DXMATRIX m;
D3DXVECTOR3 rayOrigin, rayDir;

D3DXMatrixInverse( &m, NULL, &matView );

// Transform the screen space pick ray into 3D space
rayDir.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;
rayDir.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;
rayDir.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;
rayOrigin.x = m._41;
rayOrigin.y = m._42;
rayOrigin.z = m._43;

// Use inverse of matrix
D3DXMATRIX matInverse;
D3DXMatrixInverse(&matInverse,NULL,&matWorld);

// Transform ray origin and direction by inv matrix
D3DXVECTOR3 rayObjOrigin,rayObjDirection;

D3DXVec3TransformCoord(&rayObjOrigin,&rayOrigin,&matInverse);
D3DXVec3TransformNormal(&rayObjDirection,&rayDir,&matInverse);
D3DXVec3Normalize(&rayObjDirection,&rayObjDirection);

float distanceToCollision;

int model_counter = 0;
std::vector<Model*>::iterator i;
for ( i = model.begin ( ); i < model.end ( ); i++ )
{
std::vector<Mesh*>::iterator j;
for ( j = (*i)->mesh.begin ( ); j < (*i)->mesh.end ( ); j++ )
{
std::vector<unsigned short>::iterator k;
for ( k = (*j)->indices.begin ( ); k < (*j)->indices.end ( ); k += 3 )
{
D3DXVECTOR3 p1 ( (*j)->vertices[(*k)].x, (*j)->vertices[(*k)].y, (*j)->vertices[(*k)].z );
D3DXVECTOR3 p2 ( (*j)->vertices[(*k) + 1].x, (*j)->vertices[(*k) + 1].y, (*j)->vertices[(*k) + 1].z );
D3DXVECTOR3 p3 ( (*j)->vertices[(*k) + 2].x, (*j)->vertices[(*k) + 2].y, (*j)->vertices[(*k) + 2].z );

if ( D3DXIntersectTri ( &p1, &p2, &p3, &rayObjOrigin, &rayObjDirection, NULL, NULL, &distanceToCollision ) )
return model_counter;
}
}

model_counter++;
}

return -1;
}


It always returns -1 and I don't know why, pretty much. I know... - it actually gets to the D3DXIntersectTri ( ), tested it with debug MessageBox calls. - vertices and indices are two std::vectors which contain the respective data in the format D3D9 requires its vertex and index buffers to be in for rendering (ie. vertices is a std::vector<Vertex*> with Vertex being a custom FVF and indices is a std::vector<unsigned short> with indices where each group of three make up one face in a mesh, multiple meshes make up a model of course). I don't quite see where the problem is although I'm definitely no expert, neither at matrix math nor D3D9. Any help is much appreciated! Sorry for the possibly confusing post, it's pretty late at night but I need to get this solved as soon as possible. [Edited by - d h k on March 23, 2010 10:06:50 AM]

##### Share on other sites
marius1930    119
Correct me if I'm wrong, but arnt you testing those intersections in local space?

Also, might want to consider separating the picking ray and actual intersects, as the picking ray code can/will easily go wrong when you're inexperienced.
Helps narrow it down.

##### Share on other sites
d h k    439
I am doing the intersection test in model space (I assume that's what you mean with local space?), the tutorial says you have to. :p

Anyways, I followed your second piece of advice and have added in some debug messages to print out the values and, lo and behold, I think I might be onto something:

D3DXVECTOR3 v;v.x =  ( ( ( 2.0f * x ) / ( float ) screen_width ) - 1 ) / matProj._11;v.y = -( ( ( 2.0f * y ) / ( float ) screen_height ) - 1 ) / matProj._22;v.z =  1.0f;

That one looks correct, in my game I send a ray right through the middle of the screen, the resolution is 1024x768, so the x and y parameters in my SendRay function are 512x384. That means, that v vector above is 0,-0,1 (it actually says -0 for the middle coordinate, I assume that doesn't really matter), so that would be a correct ray pointing into the scene in view space as I would expect it.

HOWEVER, the following:

D3DXMATRIX m;D3DXVECTOR3 rayOrigin, rayDir;D3DXMatrixInverse( &m, NULL, &matView );// Transform the screen space pick ray into 3D spacerayDir.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;rayDir.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;rayDir.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;rayOrigin.x = m._41;rayOrigin.y = m._42;rayOrigin.z = m._43;

rayDir always returns 0,0,1 and rayOrigin 0,0,0. The tutorial says these values should be in world space. I move around the camera, I look around but these values keep returning the above values.

But why do they do that, could it depend on when I call the above function and when I actually do the camera transformations? That's my best guess, currently. Any ideas?

EDIT: Actually, I managed to solve it, I believe. The issue was that I wasn't ever storing my view and projection matrices and just used device->GetTransform ( ) while my camera rebuild these matrices every frame from scratch.

I also forgot to calculate the world matrix for each model and use that instead of a global device->GetTransform ( )...

[Edited by - d h k on March 23, 2010 10:34:55 AM]