Hello, I was hoping someone here could help me identify the issue with my 3D mouse picking mechanic. I have a basic scene and I am trying to pick a mesh (a triangle in my case) and move it around with my mouse. So far I have been able to implement that correctly but I can't seem to be able to pick the object from far away. I'd have to move the camera closer to the triangle bounded by a sphere in order for it to intersect with my ray. What I am trying to do is simply make it so that you can pick the object up and move it around even from far away. I will include code that is relevant to the mechanic below:   

Get the ray direction

// Function that takes mouse position on screen and return ray in world coords
glm::vec3 PhysicsEngine::GetRayFromMouse()
{
    glm::vec2 ray_nds = glm::vec2(mouseX, mouseY);
    glm::vec4 ray_clip = glm::vec4(ray_nds.x, ray_nds.y, -1.0f, 1.0f);
    glm::mat4 invProjMat = glm::inverse(m_Camera.GetProjectionMatrix());
    glm::vec4 eyeCoords = invProjMat * ray_clip;
    eyeCoords = glm::vec4(eyeCoords.x, eyeCoords.y, -1.0f, 0.0f);
    glm::mat4 invViewMat = glm::inverse(m_Camera.ViewMatrix());
    glm::vec4 rayWorld = invViewMat * eyeCoords;
    glm::vec3 rayDirection = glm::normalize(glm::vec3(rayWorld));

    return rayDirection;
}

Check for ray-sphere collision

Here is where I believe I am having the problem. After debugging the code, I noticed that it does pick up an intersection between ray and the sphere even from far away because b^2 - 4ac is greater than 0. However, the real solutions (normally when b^2 - 4ac is > 0 this means we have 2 real solutions) are not returning true for some reason. To be more clear, I have commented the parts where I believe the problem is occurring. 

// Function that checks for ray-sphere intersection and returns true or false
bool PhysicsEngine::ray_sphere(vec3 ray_origin_wor, vec3 ray_direction_wor, float sphere_radius) 
{
    // work out components of quadratic
    vec3 v = glm::vec3(m_Transformation.GetPos().x, m_Transformation.GetPos().y, m_Transformation.GetPos().z) - m_Camera.GetCameraPosition();
    double a = glm::dot(ray_direction_wor, ray_direction_wor); 
    double b = 2.0 * glm::dot(v, ray_direction_wor);
    double c = glm::dot(v, v) - sphere_radius * sphere_radius;
    double b_squared_minus_4ac = b * b + (-4.0) * a * c;
    
    // b^2-4ac is greater than 0 even from far away 
    if (b_squared_minus_4ac > 0)
    {
        // Potential calculation error here 
        double x1 = (-b - sqrt(b_squared_minus_4ac)) / (2.0 * a);
        double x2 = (-b + sqrt(b_squared_minus_4ac)) / (2.0 * a);
        
        // None of these cases test true from far away 
        if (x1 >= 0.0 && x2 >= 0.0)
            return true;
        if (x1 < 0.0 && x2 >= 0.0)
            return true;
    }
    
    return false;
}

I am thankful for any guidance you can give me. 

Here my function for reference:

inline static bool IntersectRaySphere (float &t0, float &t1, const sVec3 &rO, const sVec3 &rD, const sVec3 &sO, const float sqRad) 
	{
		sVec3 L = sO - rO; 
		float tca = L.Dot(rD); 
		float d2 = L.Dot(L) - tca * tca; 

		if (d2 <= sqRad) // hits sphere
		{
			float thc = sqrt(sqRad - d2); 
			t0 = tca - thc; 
			t1 = tca + thc;
			return true;
		}
		else // misses sphere
		{
			t0 = tca; 
			t1 = tca;
			return false;
		}
	}

One intersection can be calculated by rO + rD * t0

If you replace your code with mine and still get the same problems, probably the bug is in ray calculation.

Try stepping through your code with the debugger and look at the floating point value of your individual raycast code's variables. You probably aren't handling large distances very well. Take a look at your v vector. If the distance from the camera to the object is large, the v vector will get really large. Then you do dot(v, v) which can give you huge values. Your variable b_squared_minus_4ac probably has no precision, which is ruining your function altogether.

Here's a thread where I ran into a similar problem: https://github.com/RandyGaul/tinyheaders/issues/30

To solve this you can use code someone else wrote, cross your fingers, and hope it works. Or you can take the time to learn a little more about how floats work and write something that understands numeric sensitivity a little more.

Hello, thank you both for your replies. I think I managed to fix my problem. I changed this:

  if (b_squared_minus_4ac > 0)
    {
        // Potential calculation error here 
        double x1 = (-b - sqrt(b_squared_minus_4ac)) / (2.0 * a);
        double x2 = (-b + sqrt(b_squared_minus_4ac)) / (2.0 * a);
        
        // None of these cases test true from far away 
        if (x1 >= 0.0 && x2 >= 0.0)
            return true;
        if (x1 < 0.0 && x2 >= 0.0)
            return true;
    }

into this:

if (b_squared_minus_4ac == 0)
	{
		// One real root 
		return true;
	}
	else if (b_squared_minus_4ac > 0)
	{
		// Two real roots 
		long double x1 = (-b - sqrt(b_squared_minus_4ac)) / (2.0 * a);
		long double x2 = (-b + sqrt(b_squared_minus_4ac)) / (2.0 * a);
 
		if (x1 >= 0.0 || x2 >= 0.0)
			return true;
		if (x1 < 0.0 || x2 >= 0.0)
			return true;
	}
	
	// No real root 
	return false;

Not sure how reasonable this solution is but at least it seems to be working now. I am able to pick up my triangle and move it around even from afar. 

Thank you once again for all your help guys, enjoy your day.