keym

So I guees I have to also do this:

from:

//From [0,1] to [-1,-1] in xyz
float3 ndcPos = (float3(texCoord.x, 1 - texCoord.y, depth2) - 0.5) * 2.0;

to:

//From [0,1] to [-1,1] only in xy, z stays in [0,1]
float3 ndcPos = float3(texCoord.x, 1 - texCoord.y, depth2);
ndcPos.xy = (ndcPos.xy - 0.5) * 2;

But this ^ and changing C to zNear*zFar sadly doesn't produce correct results. : /

Edit: For generating projection matrix I use D3DXMatrixPerspectiveFovRH (right handed coordinate system).
 

Ok, it seems that the problem was related to my part of code responsible for resizing texture that I use for drawing. When I fixed it, it now works as I want (clipping on sides when the window is taller than wider). I don't even have to alter aspect value and perspective matrix in any way. Funny how often you resolve problems after posting ><

Anyways, thanks for looking ya'll.

Yes, I was casting them to floats from the very beginning so that's not the issue. Thanks though.

Solved.

Looks like all my math was ok but I forgot one thing - my rendering WinAPi control has an offset in x,y (cause I have sidebar and other stuff on the side) and I forgot to take that into account when reading mouse position over the viewport. For instance I got [0,0] at the origin of the window, not the rendering control. Now all works well. Thanks for looking.

Well... shouldn't this be that simple:

object space ----[world a.k.a. model matrix]----> world space

world space ----[view a.k.a. camera matrix]----> view space

view space ----[projection matrix]----> clip space

object space <----[inverse world a.k.a. model matrix]---- world space

world space <----[inverse view a.k.a. camera matrix]---- view space

view space <----[inverse projection matrix]---- clip space

?

Anyways, this is how it *seems right* to me, but I'm not a guru here. Maybe I'm being picky ;) about naming and that was not the intention of this topic (but still I wanted to clarify naming before I ask my question(s) and make more confusion).

So, the reason I post is because (obviously) I have a problem with picking. The issue here is that in my renderer I use right hand coordinate system, like in OpenGL (for sake of compatibility, I have OGL renderer in this app too and I don't want to negate every needed value to get the same result, it would only make more future errors).

So I construct my projection matrix using D3DXMatrixPerspectiveFovRH() and view matrix using D3DXMatrixLookAtRH(). Before sending them to HLSL I transpose them (for some reason I have to do this, otherwise I get incorrect results [DX stores matrices in row major, but in HLSL they need to be in column major?]). All is sweet and dandy until picking occurs. I'm pretty sure that I'm doing something wrong, because this is my first attempt with renderer independent picking. I follow what's in the tutorial but intersection test gives incorrect results. For sake of simplicity my sphere is at (0,0,0) so I don't have to care about world and invWorld matrices. I'm guessing that something is wrong with my matrices but it's hard to track down.

Also I'm not sure what's going on here (tutorial):

// Adjust the points using the projection matrix to account for the aspect ratio of the viewport.
m_D3D->GetProjectionMatrix(projectionMatrix);
pointX = pointX / projectionMatrix._11;
pointY = pointY / projectionMatrix._22;
 

and how exactly the unprojecting part works. I mean I have mouse coordinates that I rescale into -1, 1 range but how do I get from vec2 to vec3? Where does the 3rd component come from?