I have asked, looked up, tried this problem way too many times.
I have followed this site:
Here as well as I have googled countless number of articles, ideas on how to do it.
I cannot for the life of me figure out what's going on.
Here is my code after copying/updating the functions used (I could not find anything on the d3dmath_ functions).
POINT mouseCoords;
GetCursorPos(&mouseCoords);
float dx = TANGENT_OF_FOV * ((mouseCoords.x / WIDTH_DIV_2 - 1.0f) / ASPECT);
float dy = TANGENT_OF_FOV * (1.0f - mouseCoords.y / HEIGHT_DIV_2);
D3DXMATRIX invMatrix, viewMatrix;
renderer.d3ddev->GetTransform(D3DTS_VIEW, &viewMatrix);
D3DXMatrixInverse(&invMatrix, NULL, &viewMatrix);
D3DXVECTOR3 p1;
p1.x = dx * NEAR_VIEW_PLANE;
p1.y = dy * NEAR_VIEW_PLANE;
p1.z = NEAR_VIEW_PLANE;
D3DXVECTOR3 p2;
p2.x = dx * FAR_VIEW_PLANE;
p2.y = dy * FAR_VIEW_PLANE;
p2.z = FAR_VIEW_PLANE;
D3DXVECTOR4 pOut1;
D3DXVECTOR4 pOut2;
D3DXVec3Transform(&pOut1, &p1, &invMatrix);
D3DXVec3Transform(&pOut2, &p2, &invMatrix);
However, both pOut1 and pOut2 dont have any values even close to what I want - or I don't know how to use them to do what I want. I want to be able to test an intersection with 2 triangles (i.e. a square) to see if a mouse click/right click has clicked on a particular tile.
Notes: when I first started this, simply doing:
float dx = ((mouseCoords.x / WIDTH_DIV_2 - 1.0f) / ASPECT);
float dy = (1.0f - mouseCoords.y / HEIGHT_DIV_2);
i.e., the first step on the site, would set a my dy between -1 and 1 and my dx between -.7something and .7something which I think is bad. The site says it should be between -1 and 1 for both. Perhaps that is my issue, but I cannot figure out what I am doing wrong =(
I wasn't sure if that was to be expected or not, so I continued on and needless to say, it didn't go as expected. Any help would be appreciated - this is incredibly frusterating.
(As a side note, how I have been doing clicking is to go through every visible tile/position and project that to screen coordinates and test the mouse coords to be within x/y of the screen coord. this works, but that seems incredibly inefficient and it is also not nearly as precise and does not give me the versatility I would like)
Thanks!