This snippet shows how to convert a point on screen to a corresponding point in the world. It borrows heavily from the concept of ray-tracing.

Let x and y be the point on the screen which we want to convert to the world space.

We need to create a ray which passes through the camera and the point(x, y) in worldSpace

This ray needs to satisfy the line equation (1)

where Pl = point along the line.

C = Point on the line. In this case the camera position

D = the Direction of the ray

t = some float

Now we need to get the direction of the ray.

First we transform the point to our View space coordinates. View space ranges from -1 to 1 on the x and y axis and 0 to 1 on the z axis with (0,0) being the center of the screen. We can set viewSpaceZ to 1 since the ray goes into the screen

We just reverse the following formula which is used to scale the the position in the world to fit the viewport size.

x = (ViewSpaceX + 1) * Viewport.Width * 0.5 + Viewport.TopLeftX

y = (1 - ViewSpaceY) * Viewport.Height * 0.5 + Viewport.TopLeftY

z = Viewport.MinDepth + ViewSpaceZ * (Viewport.MaxDepth - Viewport.MinDepth)

Next we adjust the points using the projection matrix to account for the aspect ratio of the viewport.

To get our world matrix we just need to invert our view matrix.

Once we have our world matrix we can get the direction of our ray by transforming the viewspace point with the world matrix and then normalize the resultant vector.

At this stage, we have an infinite number of positions in the world which will fall on this ray(

To get the exact point in the world we need the intersection of this ray with the viewing plane.

The viewing plane be defined by the plane equation

Alternatively, the plane equation can be written as

To get the normal of our viewing plane in world space, we need the combination matrix of our view and projection matrix

Once we have our matrix M, the viewing plane normal is simply the normalized vector of (M._13, M._23, M._33)

When the plane and line intersect ,

Substituting (1) in 2, we get

Using distribution, we get

Solving for t, we get

Once we have t, we can plug it back in equation (1) to get the point in our world.

This ends this mini tutorial of converting our screen coordinates to world coordinates

Let x and y be the point on the screen which we want to convert to the world space.

We need to create a ray which passes through the camera and the point(x, y) in worldSpace

This ray needs to satisfy the line equation (1)

where Pl = point along the line.

C = Point on the line. In this case the camera position

D = the Direction of the ray

t = some float

Now we need to get the direction of the ray.

First we transform the point to our View space coordinates. View space ranges from -1 to 1 on the x and y axis and 0 to 1 on the z axis with (0,0) being the center of the screen. We can set viewSpaceZ to 1 since the ray goes into the screen

We just reverse the following formula which is used to scale the the position in the world to fit the viewport size.

x = (ViewSpaceX + 1) * Viewport.Width * 0.5 + Viewport.TopLeftX

y = (1 - ViewSpaceY) * Viewport.Height * 0.5 + Viewport.TopLeftY

z = Viewport.MinDepth + ViewSpaceZ * (Viewport.MaxDepth - Viewport.MinDepth)

Spoiler

Next we adjust the points using the projection matrix to account for the aspect ratio of the viewport.

Spoiler

To get our world matrix we just need to invert our view matrix.

Once we have our world matrix we can get the direction of our ray by transforming the viewspace point with the world matrix and then normalize the resultant vector.

Spoiler

At this stage, we have an infinite number of positions in the world which will fall on this ray(

*depending on the value of t)*To get the exact point in the world we need the intersection of this ray with the viewing plane.

The viewing plane be defined by the plane equation

*where N = (A, B, C) = the plane normal vector**k = Distance from the origin*Alternatively, the plane equation can be written as

*(2)**where Pp is the point on plane*To get the normal of our viewing plane in world space, we need the combination matrix of our view and projection matrix

Once we have our matrix M, the viewing plane normal is simply the normalized vector of (M._13, M._23, M._33)

When the plane and line intersect ,

Substituting (1) in 2, we get

Using distribution, we get

Solving for t, we get

Once we have t, we can plug it back in equation (1) to get the point in our world.

This ends this mini tutorial of converting our screen coordinates to world coordinates