I am trying to do picking from a series of fixed billboard models with the following code:

Point2f Universe::ConstructCameraRay( Point2f click, Point3f *rayOriginOut, Point3f *rayDirectionOut )
{
	XMFLOAT4X4 projectionLeft, inverseCamera;
	Point3f direction;

	assert( rayOriginOut != NULL );
	assert( rayDirectionOut != NULL );

	float px, py;

	// Set mouse coordinates to range ( -1.0f to 1.0f, 1.0f to -1.0f )
	px = (( 2.0f * click.x) / projection.ScreenPixelsWidth()) - 1.0f;
	py = ((( 2.0f * click.y) / projection.ScreenPixelsHeight()) - 1.0f) * -1.0f;

	Point3f mouseClickNear( px, py, 0.0f );
	Point3f mouseClickFar( px, py, 1.0f );
	XMVECTOR mouseClickVec, unprojected;
	
	mouseClickVec = mouseClickNear;
	unprojected = DirectX::XMVector3Unproject(
		mouseClickVec,
		0.0f, 0.0f, projection.ScreenPixelsWidth(), projection.ScreenPixelsHeight(),
		0.0f, 1.0f,
		projection[0], camera, XMMatrixIdentity()
	);
	Point3f unprojectedClickNear( unprojected );

	mouseClickVec = mouseClickFar;
	unprojected = DirectX::XMVector3Unproject(
		mouseClickVec,
		0.0f, 0.0f, projection.ScreenPixelsWidth(), projection.ScreenPixelsHeight(),
		0.0f, 1.0f,
		projection[0], camera, XMMatrixIdentity()
	);
	Point3f unprojectedClickFar( unprojected );

	direction = Point3f( mouseClickNear ) - Point3f( mouseClickFar );

	*rayOriginOut = mouseClickNear;
	*rayDirectionOut = direction.Normalize();

	return Point2f( px, py );
}

bool Universe::PickTest( Point3f rayOrigin, Point3f rayDirection, Point3f planeOrigin, Point2f planeSize, Point3f *intersectionPosition )
{
	// Used http://www.rastertek.com/dx11tut47.html as reference
	XMFLOAT4X4 inverseModel;
	XMFLOAT3 origin, direction;
	Point3f planeNormal, rayHit;

	planeNormal = Point3f( 0.0f, 0.0f, -1.0f );

	float halfWidth = (planeSize.x / 2.0f);
	float halfHeight = (planeSize.y / 2.0f);
	
	Point3f points[4] = { // Clockwise, start@top left
		Point3f( planeOrigin.x - halfWidth, planeOrigin.y - halfHeight, planeOrigin.z ),
		Point3f( planeOrigin.x + halfWidth, planeOrigin.y - halfHeight, planeOrigin.z ),
		Point3f( planeOrigin.x + halfWidth, planeOrigin.y + halfHeight, planeOrigin.z ),
		Point3f( planeOrigin.x - halfWidth, planeOrigin.y + halfHeight, planeOrigin.z )
	};

	unsigned int triangle[2][3] = {
		{ 0, 1, 2 },
		{ 0, 2, 3 }
	};

	int intersects = -1;
	for( unsigned int t=0; t < 2; t++ )
	{
		if ( intersects != -1 ) break;
		Point3f e1 = points[ triangle[t][1] ] - points[ triangle[t][0] ];
		Point3f e2 = points[ triangle[t][2] ] - points[ triangle[t][0] ];
		Point3f s1 = rayDirection.Cross( e2 );

		float div = e1.Dot( s1 );
		if ( (div > -0.00001f) && (div < 0.00001f) )
			continue;

		float invDiv = 1.0f / div;

		Point3f d = rayOrigin - points[ triangle[t][0] ];

		float b1 = invDiv * d.Dot( s1 );
		if ( ( b1 < 0.0f ) || ( b1 > 1.0f ) )
			continue;

		Point3f s2 = d.Cross( e1 );
		float b2 = invDiv * rayDirection.Dot( s2 );
		if ( ( b2 < 0.0f ) || ( b1 + b2 > 1.0f ) )
			continue;

		float z = e2.Dot( s2 ) * invDiv;

		rayHit = rayOrigin + (rayDirection * z );
		intersects = t;
	}

	if ( intersects == -1 )
		return false;

	
	float ix = XMVectorGetX( rayHit ), iy = XMVectorGetY( rayHit );

	if ( (ix >= XMVectorGetX( points[0] )) && (ix <= XMVectorGetX( points[1] )) )
	{
		if ( (iy >= XMVectorGetY( points[1] )) && (iy <= XMVectorGetY( points[3] )) )
		{
			if ( intersectionPosition != NULL )
			{
				// Convert to top-left (0,0)
				intersectionPosition->x = ix;
				intersectionPosition->y = iy;
				intersectionPosition->z = XMVectorGetZ( rayHit );
			}
			return true;
		}
	}

	return false;
}

The series of billboards are laid on top of each other, and here is a mini-ASCII visualization of what the scene looks like:

[camera]< ||||||||

Where '|' are the billboards, each with transparent regions. The camera is at a fixed distance and angle (always looking directly at the billboards).

Bigger Edit/Clarifications: The code above does not work. I want to get the point of intersection of a ray to a plane, then test to see if the point of intersection is within a rectangle. Currently, my code does tell me that I am intersecting (however, I am unsure if it does so by accident, or if it's actually correct), but seems to consistently give me strange intersecting coordinates.

My testing planes are all 9600x1080, in between z 0.0f and -100.0f.

My camera pans from left to right of these giant planes, and when the mouse clicks, it should be calculating a ray based on that click, and I find the collision of the ray and each plane. Another section of my code take the intersection and finds whether or not the intersection took place within a given rectangular region, and if so, checks the texture to see if that point is transparent. If it's transparent, it proceeds to check then next furthest plane layer, and if it is not transparent, I check to see if the plane is clickable (only certain planes are) and then proceed with other tasks in the program.

Does anyone have a correction to my code, or a good tutorial on directx 11.1 picking?

There's one called pick10 in the directx sample browser. I know its only a directx 10 example. I didn't even look at it, but I don't think much has changed since then as far as picking goes.

HardlineDigital, on 26 Feb 2013 - 16:10, said:
There's one called pick10 in the directx sample browser. I know its only a directx 10 example. I didn't even look at it, but I don't think much has changed since then as far as picking goes.

Well, yes and no. It's not overly relevant because it uses DirectX 10 Meshes, whereas the ID3DX10Mesh class is not accessible to me in DirectX 11 (afaik). This class handles the function to calculate an intersection, which is primarily what I need.

BCullis, on 27 Feb 2013 - 10:20, said:
Can you explain "strange intersecting coordinates"? Also, if you're using XNAMath, why not use XMVector3Unproject to create your ray? (Full disclosure here, I haven't parsed through your ray creation function long/close enough to tell if that's where the problem is, but when you say "strange coordinates" I immediately want to think it's in your ray calculation, no offense)

Hah, no offense taken - if I didn't think it was my calculation, I wouldn't have posted

I updated my code to my most recent attempts, which does use XMVector3Unproject (found that function yesterday).

Here is a test I performed and the intersection with the following information:

Projection matrix is stereo, both with a znear/far of 0.01/2000.0, and resolution of 1920x1080, and 75.0 degrees for field of vision - they have a minor offset and rotation that is fairly minimal; for unprojection, I am consistently using the "left" projection matrix, I don't forsee this being any sort of issue.
View matrix is translated to -3840.0, 0.0, -700.0

So, when I set a point on my screen (in this case, with a mouse click), to get the ray origin and direction, I do the following:

Point3f rayOrigin, rayDirection;
universe.ConstructCameraRay( Point2f( mousex, mousey ), &rayOrigin, &rayDirection );

When I click near the bottom-left of my screen, I get the following information:

Physical Screen Coordinates: [104.98, 967.97]
Ray Origin: [0.9806, -0.792, 0.0]
Ray Direction: [0.0, 0.0, 1.0]

Clicking on the bottom-right of the screen:

Physical Screen Coordinates: [1893.01, 1041.97]
Ray Origin: [0.971, -0.9295, 0.0]
Ray Direction: [0.0, 0.0, 1.0]

Automatically, I see that my ray origin coordinates do not reflect my camera coordinates (in my ray creation function, I was setting mouseClickNear to the ray origin).

I noticed that mouseClickNear was not the projected coordinates, so I switched to using unprojectedClickNear, with the following results:

Physical Screen Coordinates: [58.98, 1020.96]
Ray Origin: [-3840.01, 0.007, -699.99]
Ray Direction: [0.0, 0.0, 1.0]
Ray/Plane Intersection: [-3840.01, 0.007, -9233]

The ray/plane intersection has to be wrong, as my plane has a z value between 0.0 and -100.0. I assume this means that my direction vector needs to be reversed, but that doesn't seem to fix it.

I'm still fairly stuck on this one =/

Hi,

So, I'm stuck on a very similar issue and I'm interested in finding a solution.

A few details on your code:

As I understand it, XMVector3Unproject takes *screen coordinates* as its initial vector value, you've started converting your mouse click coordinates into 3d space. Which is not quite correct.

You can get the right values directly by using GET_X_LPARAM(lParam) and GET_Y_LPARAM(lParam) from the window msg or you can use the ScreenToClient(...) function to convert to actual window pixels, which is what you'll need.

The other pointer I may offer is to take a look at the DirectXCollision.h file which contains structures for several types of bounding boxes and facilitates intersection tests with those objects.

I'm using a BoundingBox and doing a ray to BoundingBox intersection test in my case. (Which is still failing as my near vector and the location for my BoundingBox still don't match for some reason... :( )

Good luck!