8 replies to this topic

[mind in a box]

I've been working on viewports for my editor and stumbled over a problem regarding Orthographic projections and picking. There were a few threads I found on google with the same problem, but I haven't been able to solve the problem with them.

One of them suggested other code for doing the picking with an ortho-matrix, others said I can just reuse my current code and throw a ortho-matrix in there instead of a projective-matrix (Which does not work).

There is one case where my code works somehow (Not 100% correct, but at least a bit): When the mesh I want to pick is almost at the same location as my camera is (Or so).

Can someone point me into the right direction? Here is my code:

/** Recalculates the projection matrix after e.g. a change of the windowsize */
void SceneRenderer::RecalculateProjectionMatrix()
{
	if(bPerspectiveMatrix)
	{
		// Create perspective matrix
		Aspect=SizeX/SizeY;
		D3DXMatrixPerspectiveFovLH( &ProjectionMatrix, FOV, SizeX/SizeY, NearPlane, FarPlane );
	}else
	{
		// Create ortho matrix
		D3DXMatrixOrthoLH( &ProjectionMatrix, SizeX/OrthoZoom, SizeY/OrthoZoom, NearPlane, FarPlane);
	}
}

// -- other file ---
/** Gets you the coordinates of the mouse cursor in 3D-World space */
void SceneRenderer_ToSwapchain::ComputeMouseWorldPos(D3DXVECTOR3* outLoc, D3DXVECTOR3* outDir, Scene* Scene)
{
	D3DXVECTOR3 vPickRayDir;
	D3DXVECTOR3 vPickRayOrig;

	Scene->SetSceneCamera(SceneCamera);

	D3DXMATRIX* pmatProj = &Scene->GetBasicObjectInfo()->Projection;

	//Get the cursor 2D position
	POINT ptCursor;
	GetCursorPos( &ptCursor );
	ScreenToClient(SwapChainDesc.OutputWindow, &ptCursor );

	RECT r;
	GetClientRect(SwapChainDesc.OutputWindow, &r);

	// Compute the vector of the Mouse in screen space
	D3DXVECTOR3 v;

	v.x = ( ( ( 2.0f * (ptCursor.x ) ) / r.right) - 1 ) / pmatProj->_11;
	v.y = -( ( ( 2.0f * (ptCursor.y ) ) / r.bottom ) - 1 ) / pmatProj->_22;
	v.z = 1;

	// Get the inverse view matrix
	D3DXMATRIXA16 m;
	D3DXMatrixInverse( &m, NULL, &Scene->GetBasicObjectInfo()->View );

	// Transform the screen space Mouse pos into 3D space
	vPickRayDir.x  = v.x*m._11 + v.y*m._21 + v.z*m._31;
	vPickRayDir.y  = v.x*m._12 + v.y*m._22 + v.z*m._32;
	vPickRayDir.z  = v.x*m._13 + v.y*m._23 + v.z*m._33;
	vPickRayOrig.x = m._41;
	vPickRayOrig.y = m._42;
	vPickRayOrig.z = m._43;

	D3DXVec3Normalize(&vPickRayDir,&vPickRayDir);

	//And finally output them
	*outLoc=vPickRayOrig;
	*outDir=vPickRayDir;


	
}

Any suggestions are appreciated.

[Buckeye]

There is one case where my code works somehow (Not 100% correct, but at least a bit): When the mesh I want to pick is almost at the same location as my camera is (Or so).

Do you adjust your pick vectors for the world transform of the mesh you're trying to pick?

For example, if the mesh is translated in world space but you don't translate the pick ray origin, you'll be picking the mesh as if it were still at the origin of the world. So, when the mesh is actually rendered near the origin, the picks appear to be more correct.

EDIT: For example, if the mesh has a world transform of M, you need to multiply the ray vectors by the inverse of M. Note: for the ray direction, you need to zero the translation portion of the transform. That's the difference in D3DXVec3TransformCoord and D3DXVec3TransformNormal.

EDIT2: That may not be particularly clear, so here's some more info that may confuse you entirely.

The mesh data (vertices, etc.) are modeled around the origin. If you translate/rotate the mesh for rendering, that does not actually move the mesh data.

You need to do the pick in the mesh's object space. The ray you're calculating is in world space. By multiplying the ray vectors by the inverse of the mesh's world transform, that transforms the ray from world space into object space.

Further note, if you get a "hit," that hit is in object space. If you want to know where in world space that hit occurred, you need to transform the hit coordinates back into world space by multiplying those coords by M.

[mind in a box]

Thing is, that the code works just fine when I use the matrix which comes out of D3DXMatrixPerspectiveFovLH. I do transform my ray and it's direction by the inverse world-matrix of my meshes, indeed. As I say: The code works, but not with the ortho-matrix.

That is the code where the results of the computation in the OP get used:

void WActor::OnTrace(D3DXVECTOR3 Start,D3DXVECTOR3 Dir,D3DXVECTOR3* Hit,float* Dist,WComponent** HitComp)
{
	D3DXVECTOR3 vNear,vDir;
	D3DXMATRIX invMat;

	if(bContentMarkMode==false)
	{
		D3DXMatrixInverse(&invMat,NULL,&WorldMatrix);
	}else
	{
		D3DXMatrixInverse(&invMat,NULL,ContentWorldMatrix);
	}
	
	D3DXVec3Normalize(&vDir,&vDir);

	BOOST_FOREACH(WComponent * Out,Components)
	{
		
		Out->SetWorld(ContentWorldMatrix);
		if(Out->GetbTransformChanging())
		{
			SetMatricesFor(Out);
			Out->DoTrace(Start,Dir,Hit,Dist);
		}else
		{
			SetMatricesFor(Out);
			D3DXVec3TransformCoord(&vNear,&Start,&invMat);
			D3DXVec3TransformNormal(&vDir,&Dir,&invMat);
			Out->DoTrace(vNear,vDir,Hit,Dist);
			if(bContentMarkMode==false)
			{
				D3DXVec3TransformCoord(&vNear,&vNear,&WorldMatrix);
				D3DXVec3TransformNormal(&vDir,&vDir,&WorldMatrix);
			}else
			{
			
				D3DXVec3TransformCoord(&vNear,&vNear,ContentWorldMatrix);
				D3DXVec3TransformNormal(&vDir,&vDir,ContentWorldMatrix);
			}
		}

		CheckNULL(Hit)*Hit=vNear+(vDir*(*Dist));

		if(*Dist!=-1)
		{
			CheckNULL(HitComp)*HitComp=Out;
		}
	}
}

And this works just fine ... With the perspective matrix.

[Buckeye]

Haven't a clue what all your functions do so I'm making general suggestions.

You might compare your results to D3DXVec3Project/Unproject and D3DXIntersect.

It appears you're making a comparison between float values ( *Dist != -1 ), which is usually bad practice.

[mind in a box]

I tried D3DXVec3Unproject() but I couldn't get it to work. This is how I do it (Quickly hacked in in the last few minutes):

D3DVIEWPORT9 VP;
	D3D11_VIEWPORT VP11;
	UINT NumVPs = 1;
	DXUTGetD3D11DeviceContext()->RSGetViewports(&NumVPs, &VP11);

	VP.Height = VP11.Height;
	VP.Width = VP11.Width;
	VP.X = VP11.TopLeftX;
	VP.Y = VP11.TopLeftY;
	VP.MaxZ = VP11.MaxDepth;
	VP.MinZ = VP11.MinDepth;

	// Get the inverse view matrix
	D3DXMATRIXA16 m;
	D3DXMatrixInverse( &m, NULL, &Scene->GetBasicObjectInfo()->View );

	D3DXVECTOR3 v1;
	D3DXVECTOR3 v2;
	D3DXVECTOR3 Point = D3DXVECTOR3(ptCursor.x, ptCursor.y, 0);

	D3DXMATRIX WorldIdentity;
	D3DXMatrixIdentity(&WorldIdentity);

	D3DXVec3Unproject(&v1, &Point, &VP, &Scene->GetBasicObjectInfo()->Projection, &Scene->GetBasicObjectInfo()->View, &WorldIdentity);

	Point.z = 1;

	D3DXVec3Unproject(&v2, &Point, &VP, &Scene->GetBasicObjectInfo()->Projection, &Scene->GetBasicObjectInfo()->View, &WorldIdentity);

	*outLoc = v1;
	*outDir = v2 - v1;

Is it normal that v2 keeps being at values like 3000 and higher for each component while v1 is around 50?

[Buckeye]

The world matrix parameters for D3DXVec3Project/Unproject should be the world transform for the mesh you want to pick. That assumes you're not using an identity matrix to render the mesh, and your ptCursor is in client coords. Then (after normalizing v2-v1), the ray pos and dir are ready for D3DXIntersect.

However, using an identity for the world matrix in those calls, v1 should be in the vicinity of your world eyepoint. What you're doing, in that particular case, is converting a point on the near projection plane (screen coords with z=0) in "front" of your eyepoint to a world position. The value for v2 (before you change it to a direction) is some position on the far plane (z=1) converted to world coordinates.

For testing purposes (if you want), rather than using the mouse position, use coords at the center of the screen (the ray points "straight ahead".) Then v1 should be the eyepoint plus near-plane distance in the "look" direction, and v2 (before the subtraction) should be v1 plus (far-plane distance minus near-plane) along the "look" direction. The direction v2-v1 should be the same as the "look" direction.

[scgames]

I didn't read all the code, but generally speaking, the same method can in fact be used for both perspective and orthographic projections. The key is to 'unproject' two points, one on the near plane and one on the far plane (for example), and then build the pick ray from these points. This will handle both the orthographic and perspective cases in a uniform manner. (Apologies if that's already what you're doing.)

[mind in a box]

The world matrix parameters for D3DXVec3Project/Unproject should be the world transform for the mesh you want to pick.

Well, that would be a major problem with my engine's design. Currently I am transforming the Origin and Direction calculated by the function I posted above by the inverse-WorldMatrix of the mesh I want to intersect, and then the results by the WorldMatrix of it again (I posted that code here, too).

Can't I somehow do this with D3DXUnproject?

However, using an identity for the world matrix in those calls, v1 should be in the vicinity of your world eyepoint. What you're doing, in that particular case, is converting a point on the near projection plane (screen coords with z=0) in "front" of your eyepoint to a world position. The value for v2 (before you change it to a direction) is some position on the far plane (z=1) converted to world coordinates.

Seems like reading further answered my question from above.

normalize(v2-v1) and it should be ready to use? I'll try that.

I didn't read all the code, but generally speaking, the same method can in fact be used for both perspective and orthographic projections. The key is to 'unproject' two points, one on the near plane and one on the far plane (for example), and then build the pick ray from these points. This will handle both the orthographic and perspective cases in a uniform manner. (Apologies if that's already what you're doing.)

No, I don't. But thank you for that hint. It helped me understanding what is (has to be) going on.

[mind in a box]

And it did indeed work.

I just had to normalize it, meh. I was confused because no other person in all those threads I found was doing this.

Thank you for your help!