Hi, I have a very simple terrain map, 256x256 tiles for example, it's divided into tiles (same squares...). every tile have height, slope...

Something like figure below. My default look will be an iso view. (Each tile can be divied into smaller tiles for smooth, i called tesselation)

D3DXMatrixOrthoLH(&matProj, videoWidth , videoHeight , -100000, 100000);

float xPitch=0;

Now I need to select a unit tile onscreen with mouse! I have mouse position Mx,My I need to know what tile it is! If the map is flat, it's very easy! but with height, it's difficult. I'm planning to make the map quite static (not rotate often)... only translation. So I can store all the Vertex coordinate (x,y) on screen. By using D3DXVec3Project. And then search the triangles that contain the mouse position-> the tile we needed! However with this approach we may need to search 5-10 or 20 triangles. Do you know any betterway, more optimized or elegant ? Thanks!

I read about something like RayCasting for detection, maybe it can be used in my case. Because there is no eye pos in my view setup, the view vector is constant!

http://stackoverflow.com/questions/19150215/3d-screenspace-raycasting-picking-directx9

You can get the world position of the click by transforming the screen coordinate and the pixel depth to world space. Then if you have all the triangles stored in a octree you can traverse it and raycast against potential candidates. This way the ray cast is optimized.

Ray casting is exactly what you want to do here - you need to convert your mouse position from screen space in to world space, and then figure out if the ray that it creates intersects with one of your tiles.

When you say there is no eye position... do you mean you're just using the projection matrix to convert your world-space terrain vertices in to screen space? If so that shouldn't matter, you just use the inverse of the projection matrix to convert your two mouse points ( (x,y,0), (x,y,1) ) in to world space.

Yeah, now I'm using this approach in my current code. Using D3DXVec3Unproject 2 times on (x,y,0) and x,y,1) does give me the vector in world space, intersect with z-space give me the right tile, if we are on flat tile(with z=0), (if it is not) use this tile i search around for the right tile (so i dont have to brute force all map)! This way only effective if the tile is not very high, else we have to search quite a lot! Luckily in my map setup, i dont have cliff, every tile is only 1 height above the neighbor!

About the raycasting solution, however to get the right tile, I still have to do intersect check with quite some triangles! So I project some vertex to screen coord using D3DXVec3project , then check which triangle (in screen-space) contain the mouse Position, this is done in 2d so it's quite easy to code (check if a point is inside a triangle or not in 2d)! I know ray casting is quite basic, and searching for the right code is not that hard. Do you think what i'm doing is ok ?