If you are using a 2D mouse cursor on top of a 3D world then the first thing you need to do is find out what 3D objects are underneath the current 2D position of the mouse. This process is sometimes called picking, and your 3D api might support this, e.g. Direct3D Retained Mode has a function Direct3DRMViewport.Pick which takes a 2D pixel coordinate and returns a list of 3D objects underneath it. There''s a similar function for picking faces on a mesh.

If you are using a 3D virtual reality pointer instead of 2D mouse (e.g. a hand) which is moved to physically touch things, then you know what your pointer has touched via standard 3D collision detection techniques.

Unfortunately, since I only know Direct3D Retained Mode, I cannot help you with other 3D APIs, but looking up terms like picking, transform, inversetransform might help.

You could project a ray from camera cordinates (X_Mouse,YMouse,0), direction (X_Mouse,YMouse,1). Then check intersection on this ray with the faces of objects near of the ray, excluding the objects far from the ray via BSP, OCT Tree, etc.

Hope it helps!

Unfortunately I only know how to do it in D3D Retained Mode - IMO the easiest D3D api which for some reason wasn''t very popular. I haven''t got a clue about how to do it in D3D Immediate Mode or D3DX Utility library. Sorry.

I would highly advise a search of the forums, this is a biweekly topic.

If you are using the D3DX Utility library have a look at D3DXVec3Unproject which will project a vector from 2D screen space to 3D object space. Then have a look at D3DXIntersectMesh which takes care of ray intersections with triangles.

As I explained before, I haven''t used D3DX myself so I cannot vouch that these will work, but after looking at the documentation I think they''re a good starting point.

Perhaps one of these days I will progress from Retained Mode to D3DX.

You mentioned that you saw the example in DX8. Indeed this is what you need. You are right, the example also calculates the texture coordinates, but it also calculates the polygons that were picked. You may simply throw away the texture coordinates that were calculated. Just have a look at the sample again. Run it and click on any poly. You will see that it is highlighted. Now make your board, where every square is two triangles, let the algorithm run and test which polygons on the board are hit, and tada, you are ready.
What the example indeed does is casting a ray from the center of your camera through the mouse position by doing an inverse projection (unproject) of the mouse point. Then it calculates the intersection point of this ray with all planes within which triangles lie (except if the ray is parallel to the plane). Now your "texture" coordinates are calculated. It is indeed not the real texture coordinates, but the 2D coordinates of the intersection point in the coordinate system that is spanned by the border vectors of the triangle. If those two coordinates are larger than 0 and their sum is smaller than 1, the ray hits the plane inside the triangle, otherwise outside. That those two coordinates are also returned, is just a bonus, you may throw them away.
The math behind this is rather simple. You only need the cross product (which calculates the normal on two vectors) and the dot product. Where a.b = length(a) * length(b) * sin(angle between a and b). And thus calculates somewhat the length of a in the direction of b. I think nothing more is needed, but it is quite some time, I looked at it.
I hope you are now able to understand the pick example or you can even code it yourself. The thing is much simpler if you calculate this for a board anyways, because you only need to calculate the intersection point with one plane, and then decide on the coordinates you get which field was clicked.
Pictures would help a lot, but I am no good at ascii art.