Hi guys.

I'm currently trying to create a simpler picker, so constructing a ray from the mouse and colliding it with the scene. Though it's not working as it should, the way I check the results is simply creating a cube, and where the ray collides with a plane is where the cube will be translated, but it remains static. This is my current attempt:

Update Loop:

POINT p;
if (GetCursorPos(&p))
{
	if (ScreenToClient(display.hWnd, &p))
	{
		//p.x and p.y are now relative to hwnd's client area
		intersect.Translate(engine.GetTriAtXY(plane.Vertices, plane.Indices, 6, p.x, p.y));
	}
}

The GetTriAtXY: (Don't mind the name)

CEVector3f C3DEngineObject::GetTriAtXY(CDK::Vertex *vertices, CDK::Index *indices, int indiceCount, int mouseX, int mouseY)
{
	//getting projection matrix
	D3DXMATRIX projMat = *Camera->getProjection();

	//calculating.. mouse ray
	float vx = (+2.0f*mouseX/viewport.Width  - 1.0f)/projMat._11;
	float vy = (-2.0f*mouseY/viewport.Height + 1.0f)/projMat._22;

	//Vector is D3DVECTOR
	CEVector3f Origin(0.0f, 0.0f, 0.0f);

	// I used Z as my UP VECTOR, not sure how it will work for you
	CEVector3f Direction(vx, vy, 1.0f);

	//getting projection matrix
	D3DXMATRIX viewMat = *Camera->getView();

	//inversing projection matrix
	D3DXMATRIX iviewMat;
	D3DXMatrixInverse(&iviewMat, 0, &viewMat);
	D3DXVec3TransformCoord(&Origin, &Origin, &iviewMat);
	D3DXVec3TransformNormal(&Direction, &Direction, &iviewMat);

	//setting variables
	DWORD dwFace;
	FLOAT fBary1, fBary2, fDist;

	//checking intersection
	until(indiceCount/3)
	{
		CEVector3f v0 = XYZ_VEC(vertices[i]);
		CEVector3f v1 = XYZ_VEC(vertices[i+1]);
		CEVector3f v2 = XYZ_VEC(vertices[i+2]);
		if (D3DXIntersectTri( &v0, &v1, &v2, &Origin, &Direction, &fBary1, &fBary2, &fDist ))
		{
			CEVector3f p1p0 = v1 - v0;
			CEVector3f p2p0 = v2 - v0;
			return v0 + p1p0 * fBary1 + p2p0 * fBary2;
		}
	}

	return CEVector3f(0,0,0);
}

But as I mentioned, the cube isn't moved, and there is usually a collision (by putting a break point inside the if(D3DX...), but the returns aren't correct.

Does anyone have an idea why this isn't working as expected?

Thanks, as always.

-MIGI0027

Looking at my picking code, I see two differences:

1. I think that you might want to renormalize the direction vector after you transform it by the inverse view matrix.
2. I think the correct way to generate the intersection point would be something like:

   
   intersect = v0 * (1.0f - fBary1 - fBary2) + v1 * fBary1 + v2 * fBary2;
   

   if I am reading this correctly http://www.scratchapixel.com/lessons/3d-basic-lessons/lesson-9-ray-triangle-intersection/barycentric-coordinates/.

(I'm using SlimDX, which has an overload that returns the intersection point directly, however, I've checked this using the similar Ray.Intersects() method to your D3DXIntersectTri() method and it seems to be correct).

Thank you for your input eric .

The problem was a combination of your suggestions and some other, so I added/modified the things as suggested (thanks for the link btw!), though it still didn't work. Then I realized that I transpose all my matrices for the shaders, and as my projection matrix was pre transposed, incorrect results came out, so transposing the transposed (reverting it) gives me the correct matrix, so yeah, it works.

Thanks.

-MIGI0027