Advertisement Jump to content
Sign in to follow this  

DX11 DX11 - Ray Tri Collision - Mouse Picking

This topic is 1873 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi guys. happy.png


I'm currently trying to create a simpler picker, so constructing a ray from the mouse and colliding it with the scene. Though it's not working as it should, the way I check the results is simply creating a cube, and where the ray collides with a plane is where the cube will be translated, but it remains static. This is my current attempt:


Update Loop:

if (GetCursorPos(&p))
	if (ScreenToClient(display.hWnd, &p))
		//p.x and p.y are now relative to hwnd's client area
		intersect.Translate(engine.GetTriAtXY(plane.Vertices, plane.Indices, 6, p.x, p.y));

The GetTriAtXY: (Don't mind the name)

CEVector3f C3DEngineObject::GetTriAtXY(CDK::Vertex *vertices, CDK::Index *indices, int indiceCount, int mouseX, int mouseY)
	//getting projection matrix
	D3DXMATRIX projMat = *Camera->getProjection();

	//calculating.. mouse ray
	float vx = (+2.0f*mouseX/viewport.Width  - 1.0f)/projMat._11;
	float vy = (-2.0f*mouseY/viewport.Height + 1.0f)/projMat._22;

	//Vector is D3DVECTOR
	CEVector3f Origin(0.0f, 0.0f, 0.0f);

	// I used Z as my UP VECTOR, not sure how it will work for you
	CEVector3f Direction(vx, vy, 1.0f);

	//getting projection matrix
	D3DXMATRIX viewMat = *Camera->getView();

	//inversing projection matrix
	D3DXMATRIX iviewMat;
	D3DXMatrixInverse(&iviewMat, 0, &viewMat);
	D3DXVec3TransformCoord(&Origin, &Origin, &iviewMat);
	D3DXVec3TransformNormal(&Direction, &Direction, &iviewMat);

	//setting variables
	DWORD dwFace;
	FLOAT fBary1, fBary2, fDist;

	//checking intersection
		CEVector3f v0 = XYZ_VEC(vertices[i]);
		CEVector3f v1 = XYZ_VEC(vertices[i+1]);
		CEVector3f v2 = XYZ_VEC(vertices[i+2]);
		if (D3DXIntersectTri( &v0, &v1, &v2, &Origin, &Direction, &fBary1, &fBary2, &fDist ))
			CEVector3f p1p0 = v1 - v0;
			CEVector3f p2p0 = v2 - v0;
			return v0 + p1p0 * fBary1 + p2p0 * fBary2;

	return CEVector3f(0,0,0);

But as I mentioned, the cube isn't moved, and there is usually a collision (by putting a break point inside the if(D3DX...), but the returns aren't correct.


Does anyone have an idea why this isn't working as expected?


Thanks, as always. wink.png


Share this post

Link to post
Share on other sites

Looking at my picking code, I see two differences:

  1. I think that you might want to renormalize the direction vector after you transform it by the inverse view matrix.
  2. I think the correct way to generate the intersection point would be something like:
    intersect = v0 * (1.0f - fBary1 - fBary2) + v1 * fBary1 + v2 * fBary2;

    if I am reading this correctly  

(I'm using SlimDX, which has an overload that returns the intersection point directly, however, I've checked this using the similar Ray.Intersects() method to your D3DXIntersectTri() method and it seems to be correct).

Share this post

Link to post
Share on other sites

Thank you for your input eric happy.png .


The problem was a combination of your suggestions and some other, so I added/modified the things as suggested (thanks for the link btw!), though it still didn't work. Then  I realized that I transpose all my matrices for the shaders, and as my projection matrix was pre transposed, incorrect results came out, so transposing the transposed (reverting it) gives me the correct matrix, so yeah, it works.


Thanks. wink.png


Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using, you agree to our community Guidelines, Terms of Use, and Privacy Policy. is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!