Hello,
I'm writing a map editor for a tile-based 2d sidescroller and for a couple of days I've been stuck with a pretty annoying bug related to picking.
My viewport has the same size as the client area. The camera is at (0, 0, 300), it has clipping planes at 10 and 400, it's FOV is pi/2. The camera looks at (0, 0, 0) and the upvector is (0, -1, 0).
I've tried using different methods of implementing the picking but right now I have one similar to the Pick example from the SDK.
Here's how I do the thing:

D3DXVECTOR3 z;
D3DXMATRIX world, view, proj, m;
D3DVIEWPORT9 vp;
m_pd3dDevice->GetTransform(D3DTS_VIEW, &view);
m_pd3dDevice->GetTransform(D3DTS_PROJECTION, &proj);
m_pd3dDevice->GetTransform(D3DTS_WORLD, &world);
m_pd3dDevice->GetViewport(&vp);

POINT p;
GetCursorPos(&p);
ScreenToClient(m_hwnd, &p);
D3DXVECTOR3 v;

v.x = (((2.0f * p.x) / vp.Width) - 1);
v.y = -(((2.0f * p.y) / vp.Height) - 1);
v.z = 1.0f;
v.x /= proj._11;
v.y /= proj._22;

D3DXMatrixInverse(&m, 0, &view);
D3DXVECTOR3 vPickRayDir, vPickRayOrig;
vPickRayDir.x = v.x * m._11 + v.y * m._21 + v.z * m._31;
vPickRayDir.y = v.x * m._12 + v.y * m._22 + v.z * m._32;
vPickRayDir.z = v.x * m._13 + v.y * m._23 + v.z * m._33;
vPickRayOrig.x = m._41;
vPickRayOrig.y = m._42;
vPickRayOrig.z = m._43;

vPickRayOrig -= vPickRayDir*(m_cam->GetDistance() / vPickRayDir.z);
The results are: when I click at the center of the client area everything is fine. But the farther I click from the center the more is the difference between my transformed coordinates and where should they be. I.e. when I click somewhere in the upper-right corner of the client area the tile is placed above and to the right of the cursor.

I've googled a lot but was unable to find anything that could've helped me. Therefore I ask for your help with this matter.

vPickRayOrig -= vPickRayDir*(m_cam->GetDistance() / vPickRayDir.z);

Are you sure this last line is needed? I have a similar function, but it does not do that last part. I'm also doing the inverse world and view, not just the inverse view. Not sure if either of those are you problem though.

The last line is only to find x and y on the z = 0 plane. And inversing the world isn't needed in my case as I don't use local coordinates at all.

Hi.

I'm thinking that your above pick code is used for a mesh that is allready in world coord(I think)

try doing the invers of the world * view matrix

D3DXMATRIX mWorldView = matWorld * matView;

D3DXMATRIX m;

D3DXMatrixInverse( &m, NULL, &mWorldView );



[color="#008000"][color="#008000"]// Transform the screen space pick ray into 3D space

[color="#008000"][color="#008000"] rayDir.x = v.x * m._11 + v.y * m._21 + v.z * m._31;

rayDir.y = v.x * m._12 + v.y * m._22 + v.z * m._32;

rayDir.z = v.x * m._13 + v.y * m._23 + v.z * m._33;

rayOrigin.x = m._41;

rayOrigin.y = m._42;

rayOrigin.z = m._43;

Of course that's not the case. I've done that during my first attempt when I used Pick.cpp's code. And the only thing I got was the source of the ray moving to (-390, -390) instead of (0, 0) on XY plane.

oops my bad. try this one posted the wrong one

[color="#008000"][color="#008000"]//we need the Identity matrix not the objects world matrix

[color="#008000"][color="#008000"]//this is because unproject needs a reference to work from like the centre of the world I think D3DXMATRIX Identityworld;



D3DXMatrixIdentity( &Identityworld );


D3DXMATRIX ToObjectSpaceMat;

D3DXMatrixInverse( &ToObjectSpaceMat, NULL, &matWorld );



v.x = ptCursor.x;

v.y = ptCursor.y;

v.z = 0.0f;



[color="#008000"][color="#008000"]//unproject the origin to its world location with a z = 0.0f

[color="#008000"][color="#008000"] D3DXVec3Unproject(&rayOrigin,

&v,

&m_ViewPort,

&P,

&matView,

&Identityworld);



[color="#008000"][color="#008000"]// second call has z set to one

[color="#008000"][color="#008000"] v.x = ptCursor.x;

v.y = ptCursor.y;

v.z = 1000.0f;



[color="#008000"][color="#008000"]//unproject the direction to its world location with a z = 1000.0f the z here can fit into any z range you may want maybe?????

[color="#008000"][color="#008000"] D3DXVec3Unproject(&rayDir,

&v,

&m_ViewPort,

pmproj,

&matView,

&Identityworld);







[color="#008000"][color="#008000"]//now transform them to object space

[color="#008000"][color="#008000"] D3DXVec3TransformCoord(&rayOrigin, &rayOrigin, &ToObjectSpaceMat);

D3DXVec3TransformNormal(&rayDir, &rayDir, &ToObjectSpaceMat);

then intercect

Another thing to check: Is your backbuffer the same size as your window's CLIENT area? If you're not calling AdjustWindowRectEx() and you're running in windowed mode, then this is almost certainly the problem.

ankhd, that's just an another way to do the stuff. But still, thanks for the reply.

Evil Steve, of course they are the same! While I had been googling the problem earlier I think I saw you post from long ago about them and checked it immediately. But thanks.

Anyway, the problem was solved. Apparently , the guy who wrote the renderer had all the tiles drawn at z = 5 (don't know the reasons and don't really want to know). So everything works like a charm now and I've got a ton of knowledge concerning transformation matrices in DX. Thanks for all replies!