I used to use ScreenPointToRay function to help me do this in Unity3d. But now I use Ogre and Bullet, I have to do this on my own.

I try to dig into Bullet's example and find out this function getRayTo which convert the point on that click on screen (camera) to the 3d world point with 'farPlane' distance from the camera (that what I think). Then they create the ray (vector AB) by set the A point is the camera position, B point is the point get from the function getRayTo above. I try to re-implement this function in my project (use Ogre for graphic, so I have to use Ogre's camera and world). But I can't get it right because I don't understand the function getRayTo from Bullet.

Please explain the algorithm to convert screen point to ray. If there is a better algorithm please teach me.

Note: the scenario to use this algorithm is to find out the position to go for the character - like in the game TouchLight - a point and click game.

https://github.com/bulletphysics/bullet3/blob/d52fb7510bbe3801431cc0a431e4c1847ee3e0f1/examples/CommonInterfaces/CommonRigidBodyBase.h#L17

With this code

btVector3 rayToCenter = rayFrom + rayForward;
btVector3 dHor = hor * 1.f / width;
btVector3 dVert = vertical * 1.f / height;


btVector3 rayTo = rayToCenter - 0.5f * hor + 0.5f * vertical;
rayTo += btScalar(x) * dHor;
rayTo -= btScalar(y) * dVert;
return rayTo;

Look like they shift the center point to the point that clicked on the camera.

Here is my attempt to convert the function, and it looks pretty ok

Ogre::Camera* m_pCam;

btVector3	EmptyProjectGameState::getRayTo(int x, int y)
{
	float top = 1.f;
	float bottom = -1.f;
	float nearPlane = 1.f;
	float fov = m_pCam->getFOVy().valueRadians();

	btVector3 camPos;

	camPos.setX(m_pCam->getPosition().x);
	camPos.setY(m_pCam->getPosition().y);
	camPos.setZ(m_pCam->getPosition().z);

	btVector3	rayFrom = camPos;
	btVector3 rayForward;
	rayForward.setX(m_pCam->getDirection().x);
	rayForward.setY(m_pCam->getDirection().y);
	rayForward.setZ(m_pCam->getDirection().z);

	rayForward.normalize();
	float farPlane = 10000.f;
	rayForward *= farPlane;

	btVector3 rightOffset;
	btVector3 cameraUp = btVector3(0, 1, 0);

	btVector3 vertical = cameraUp;

	btVector3 hor;
	hor = rayForward.cross(vertical);
	hor.safeNormalize();
	vertical = hor.cross(rayForward);
	vertical.safeNormalize();

	float tanfov = tanf(0.5f*fov);


	hor *= 2.f * farPlane * tanfov;
	vertical *= 2.f * farPlane * tanfov;

	btScalar aspect;
	float width = float(mGraphicsSystem->getRenderWindow()->getWidth());
	float height = float(mGraphicsSystem->getRenderWindow()->getHeight());

	aspect = width / height;

	hor *= aspect;


	btVector3 rayToCenter = rayFrom + rayForward;
	btVector3 dHor = hor * 1.f / width;
	btVector3 dVert = vertical * 1.f / height;


	btVector3 rayTo = rayToCenter - 0.5f * hor + 0.5f * vertical;
	rayTo += btScalar(x) * dHor;
	rayTo -= btScalar(y) * dVert;
	return rayTo;
}

const float aspect = (float)(m_renderbufferWidth) / (float)(m_renderbufferHeight);
const float fx = 2.0f * ((float)(x) / (float)(m_renderbufferWidth - 1)) - 1.0f;
const float fy = 2.0f * ((float)(y) / (float)(m_renderbufferHeight - 1)) - 1.0f;
const float tangent = tan(y_fov_radians / 2.0f);

last_click_float_x = aspect*tangent*fx;
last_click_float_y = -tangent*fy;

Here x, y is the pixel location that was clicked on by the mouse/stylus/finger user, and last_click_float_x, last_click_float_y is the location on the image plane (which is taken to be a distance of 1 from the camera). The y_fov_radians and m_renderBufferWidth, m_renderBufferHeight variables are self-explanatory (hopefully).