Jump to content
  • Advertisement

[GLM] Mouse click to create ray.

Recommended Posts

Posted (edited)

What am I doing wrong when trying to calculate a ray from the camera?

glm::vec4 mouseClip = {ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 0, 1};

glm::mat4 inv = glm::inverse(game::camera->projection * game::camera->transform);

glm::vec4 p = inv * mouseClip;
p.x /= p.w;
p.y /= p.w;
p.z /= p.w;

glm::vec3 direction = glm::normalize(glm::vec3{p});

I can guarantee that mouseClip's x and y is correct (from [-1, -1] to [1, 1]).

glm::vec3 intPos, intNor;
if(glm::intersectRaySphere(glm::vec3{game::camera->transform[3]}, direction, glm::vec3{0, 0, 0}, 1, intPos, intNor)) {

intersectRaySphere does as it says, here is it's documentation.

Problem is that.. well.. basically everything is off. There is nothing I can accurately describe, it seems as if the camera's transformation is always exaggerated. I know this is all very vague, please ask questions if you have any.

Edited by midn

Share this post

Link to post
Share on other sites

I tried using glm::unProject instead, but issues still exist:

glm::vec3 direction = glm::normalize(glm::unProject(glm::vec3{ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 1}, game::camera->transform, game::camera->projection, glm::vec4{0, 0, 640, 480}));


Share this post

Link to post
Share on other sites

Issue is common due to window dimensions and mouse relative position.

Where windiw with border can be 640x480 the actual glwindow may be smaller and the mouse coordinate can be calculated from window left top including border if you managed to fix that and have still bad results then you could calculate the ray from near z plane knowing left top corner as 0,0 you can clac the ray knowing mouse relative to window coord sth like this:


inline vec3 GetDirectionFromScreen(mat4 modelview, TSpecCamera * FPP_CAM,  int x, int y, int sw, int sh, float fov, float z_near, float aspect)//, float z_far)

mat4 mvm = modelview;
vec3 dirX, dirY;
	dirX.x = mvm.m[0];
	dirX.y = mvm.m[4];
	dirX.z = mvm.m[8];

	dirY.x =	mvm.m[1];
	dirY.y =	mvm.m[5];
	dirY.z =	mvm.m[9];

//	float aspect = float (SCREEN_WIDTH) / float (SCREEN_HEIGHT);
	float a = fov / 2.0;
float cotangent = 1.0 / tan( a * imopi );

float ax = z_near / cotangent;

float screen_w = 2.0*ax;

float screen_h = screen_w;// * yratio;

screen_w = screen_w * aspect;

float scr_coord_x = float(x) / float(sw);
float scr_coord_y = float(sh - y) / float(sh);

vec3 dir = FPP_CAM->ReturnFrontVector();

//move to lower left corner
vec3 start_pos = (FPP_CAM->pos + dir * z_near) + (-dirX * (screen_w / 2.0)) + (-dirY * (screen_h/2.0));

vec3 start = start_pos + (dirX * (screen_w * scr_coord_x)) + (dirY * (screen_h * scr_coord_y));

return Normalize( vectorAB(FPP_CAM->pos, start) );


Its an old code not sure if fov was fovy imopi stands for pi/180

Coords are.in pixels as i think

Share this post

Link to post
Share on other sites

As I said, I can guarantee the mouse position is correct, that's not the issue.

I apparently have passed a few wrong arguments to glm::unProject, here is the updated code:

glm::vec3 direction = glm::unProject(glm::vec3{ev.cursor.x, 480 - ev.cursor.y, 1}, glm::mat4{1}, game::camera->projection * game::camera->transform, glm::vec4{0, 480, 640, -480});
direction = glm::normalize(direction - glm::vec3{game::camera->transform[3]});

And now it works for almost all cases. It starts to break if there's any Y translation, or any rotation at all, or if I perform glm::translate and glm::rotate in the wrong order. Those values seem to be always exaggerated too much for seemingly no reason.

Share this post

Link to post
Share on other sites

The Y translation making it incorrect? That was a non-issue, I was inverting the mouse's Y coordinate twice, durr.
But rotation in the view matrix is still making it incorrect, said rotation (on any axis) also seems to be exaggerated, as if it's scaled up.

Share this post

Link to post
Share on other sites

Fixed the issue. And, as ALWAYS it's the stupidest issues you can't even imagine.

Check the unProject documentation:


detail::tvec3<T> glm::gtc::matrix_transform::unProject(detail::tvec3< T > const& win,
		detail::tmat4x4< T > const& model,
		detail::tmat4x4< T > const& proj,
		detail::tvec4< U > const& viewport 


The second argument? That's the modelview matrix, not just the view matrix! So I just ended up passing the inverse of the view matrix there, and it works fantastically now.

Share this post

Link to post
Share on other sites

It does not make any sense since model matrix is the object matrix not camera one. Maybe you think about cam_translation_mat*cam_rot_mat?

Share this post

Link to post
Share on other sites

It's GLM's documentation that's misleading. The program now works without issues.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!