Quote:Original post by Raeldor
Put screen x and y into a 3d vector
Transform vector using inverse of projection matrix

class CProjectionController{   /////////////////////////////////   // Initializing.public:   void SetData(float fovAngleY, int sizeX, int sizeY);   void SetBorders(float fNear, float fFar);   /////////////////////////////////   // Base data.protected:   float  m_fFovAngleY;   int  m_nSizeX;   int  m_nSizeY;protected:   float  m_fNear;   float  m_fFar;   /////////////////////////////////   // Derived data.protected:   float  m_fAspectXY;   float  m_fTanX;   float  m_fTanY;   /////////////////////////////////   // Commands.public:   void ApplyPerspective(void) const;   // From mouse screen coordinates.   xvec GetRayDir(int x, int y) const;};void CProjectionController::SetData(float fovAngleY, int sizeX, int sizeY){   // Consume.   m_fFovAngleY = fovAngleY;   m_nSizeX = sizeX;   m_nSizeY = sizeY;   // Recompute.   m_fAspectXY = ((float)sizeX) / ((float)sizeY);   m_fTanY = tanf(fovAngleY/2);   m_fTanX = m_fTanY * m_fAspectXY;}xvec CProjectionController::GetRayDir(int x, int y) const{   // Screen coordinates to [-1, 1).   float xf = 2.0f * ((float)(x - (m_nSizeX/2))) / ((float)m_nSizeX);   float yf = 2.0f * -((float)(y - (m_nSizeY/2))) / ((float)m_nSizeY);   // To camera coordinates (on the Near plane).   float dx = xf * m_fNear * m_fTanX;   float dy = yf * m_fNear * m_fTanY;   return norm( xvec(dx, dy, m_fNear) );}void CProjectionController::ApplyPerspective(void) const{   glMatrixMode(GL_PROJECTION);   glLoadIdentity();   const float degsY = radsToDegs(m_fFovAngleY);   gluPerspective( degsY, m_fAspectXY, m_fNear, m_fFar);}

Quote:Original post by deffer
Remember, that after projecting, the x,y,z coordinates are divided by w, to make it == 1. Now, it is just a simple rescaling of the vector, so if it is done sooner, or later, it makes no big difference.
So, after:

Quote:Original post by Raeldor
Put screen x and y into a 3d vector
Transform vector using inverse of projection matrix

normalize your vector (by normalize, I mean (xyzw)/w).
(*Not sure that would be all, though*)

Quote:Original post by deffer
Ok, let's assume you used:
D3DXMATRIX* WINAPI D3DXMatrixPerspectiveFovLH
( D3DXMATRIX *pOut, FLOAT fovy, FLOAT Aspect, FLOAT zn, FLOAT zf );
as the projecton matrix. There's no screen width/height mentioned, so for unprojecting you need XY in (-1,1)x(-1,1) with z=0 and w=1. So this you're doing right.

You should also use D3DXVec3TransformCoord, not D3DXVec3Transform, otherwise you would have to normalize (w->1) manually.

Test if this is right, without using other matrices for now (use identities). All step by step.

Quote:Original post by deffer
No no, you are transforming to (-1,1) the right way (look into my code, if you're not sure).

As for w, if it was fixed for 1, what purpose would it serve? For all non-affine transformations it is =1. For projection, it is not. Basically, for projection to perform, we need division. As matrix/vector operations consist only of addition and mulitplication, we are relying on w coordinate, which through multiplication by specific projection matrix gets some specific value (depending on xyz and the matrix, of course). Then, we have to explicitly make the division part by hand: x/=w; y/=w; y/=w; w/=w; (that's what I refer to normalization of a point (Note: it has nothing to do with vector normalization))

Now, normally you would have perform matrix/vecto transformation with D3DXVec3Transform (assuming you are using DX math classes). But there's a special version of that, that is doing also the dividing part: D3DXVec3TransformCoord. If you won't use this function, you will loose the w value after matrix multiplication (unless you're doing all the transformations by hand, or using 4D vectors (as I am)).