# Screen to World Ray

## Recommended Posts

Medo Mex    891
Hi guys,

I'm trying to raycast from the center of the screen towards anything in front of the camera.

My goal is that I make the player able to shoot enemies according to the "+" symbol in the center of the screen.

How can I get D3DXVECTOR3 of the center of the screen?

Any help would be appreciated.

[attachment=12578:ray.jpg] Edited by Medo3337

##### Share on other sites
Tispe    1468
Use D3DXIntersect() maybe?

##### Share on other sites
Medo Mex    891
I use Bullet physics raytest, how do I get the 3D position that appear exactly infront of the camera (at the center of the screen)?

I'm trying to get the 3D position of the symbol "+" so I can raycast from it. Edited by Medo3337

##### Share on other sites
GuyWithBeard    1890
Assuming that in the coordinate system you are using Y is up, X is to the right and Z is forward, the camera forward vector is (X=0, Y=0, Z=1) in view space. You just multiply this by the view-to-world matrix to get the vector in world space. Remember to set W to 0 since this is a direction rather than a point. The origin for the ray would be (X=0, Y=0, Z=0) in view space. To get that into world space you do the same but with W as 1. Then you can just scale the direction vector with something large like 10 000 to get a really long ray vector. If I remember correctly, these two vectors (ie. a ray origin and direction) is what Bullet expects.

##### Share on other sites
Medo Mex    891
@GuyWithBeard: I have the point of the cursor which is x, y, from this point how can I get the x, y, z in the 3D world? the idea is the same like picking 3D mesh, you get the 3D world from the cursor x, y cursor point and raycast from this position to for example 1000.0f forward (1000.0f is the distance). Edited by Medo3337

##### Share on other sites
GuyWithBeard    1890
Is your crosshair in the middle of the screen (or to be more exact in the middle of the view)?. In that case you don't need to work with 2D coordinates at all. The ray will originate from the view space origin (where your camera is) and go straight forward in the direction the camera is facing.

##### Share on other sites
Medo Mex    891
Here is what I want to do, I am trying to do two things:
1. Allow the player to shoot the enemy raycast(centerOfScreenWorld.x, centerOfScreenWorld.y, centerOfScreenWorld.z, 1000.0f); // 1000.0f = distance
How can I get centerOfScreenWorld?

2. This is something completely different (picking mesh), it should be something like pickMesh(cursorPoint.x, cursorPoint.y, 500.0f); // 500.0f = distance
pickMesh should return information about the picked mesh (in other word mesh that was hit by the ray), How can I raycasting from cursorPoint to the distance? Edited by Medo3337

##### Share on other sites
GuyWithBeard    1890
1. The center of the screen = the center of the near plane. This would be (X=0, Y=0, Z=NearPlaneDistance) in view space. Use what I told you before to get into world space.

Update: actually that's not right. The "center of the screen" would probably be the camera position, not the near plane. My bad.

2. http://halogenica.net/ray-casting-and-picking-using-bullet-physics/ That article should give you all you need.

##### Share on other sites
Tispe    1468
Nabbed this from dxtut
[CODE]
// get the current transform matrices
D3DXMATRIX matProjection, matView, matWorld, matInverse;
d3ddev->GetTransform(D3DTS_PROJECTION, &matProjection);
d3ddev->GetTransform(D3DTS_VIEW, &matView);
d3ddev->GetTransform(D3DTS_WORLD, &matWorld);
// use the mouse coordinates to get the mouse angle
GetCursorPos(&MousePos);
float xAngle = (((2.0f * MousePos.x) / SCREEN_WIDTH) - 1.0f) / matProjection(0, 0);
float yAngle = (((-2.0f * MousePos.y) / SCREEN_HEIGHT) + 1.0f) / matProjection(1, 1);
D3DXVECTOR3 origin, direction;
origin = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
direction = D3DXVECTOR3(xAngle, yAngle, 1.0f);
// find the inverse matrix
D3DXMatrixInverse(&matInverse, NULL, &(matWorld * matView));
// convert origin and direction into model space
D3DXVec3TransformCoord(&origin, &origin, &matInverse);
D3DXVec3TransformNormal(&direction, &direction, &matInverse);
D3DXVec3Normalize(&direction, &direction);
[/CODE]

##### Share on other sites
Medo Mex    891
Focusing on number 1 (player shooting raycasting) I tried doing something close to what you said, but it's not working as expected, can you show an example?

##### Share on other sites
Medo Mex    891
@Tispe: This code is dealing with model transformation, I'm not doing anything related to models, instead, I'm trying to raycast from the center of the camera to the forward area.

I'm looking to do something exactly like the following:

##### Share on other sites
GuyWithBeard    1890
[quote name='Medo3337' timestamp='1354548439' post='5006630']
Focusing on number 1 (player shooting raycasting) I tried doing something close to what you said, but it's not working as expected, can you show an example?
[/quote]

This is how I do it in my engine:

[CODE]void Camera::getPickRay(int screenX, int screenY, Vec3& rayOrigin, Vec3& rayDirection, Math::CoordinateSpace space)
{
float vx = (((2.0f * screenX) / (float)mGraphicsEngine->getScreenWidth()) - 1.0f ) / mProjectionMatrix._11;
float vy = (((-2.0f * screenY) / (float)mGraphicsEngine->getScreenHeight()) + 1.0f ) / mProjectionMatrix._22;
switch (space)
{
case Basis::Math::SpaceLocal:
rayOrigin.set(0.0f, 0.0f, 0.0f);
rayDirection.set(vx, vy, 1.0f);
rayDirection.normalize();
break;
case Basis::Math::SpaceWorld:
Mat4& worldMatrix = mParentNode->getLocalToWorldTransform();
Vec4 o(0.0f, 0.0f, 0.0f, 1.0f);
Vec4 d(vx, vy, 1.0f, 0.0f);
o = o * worldMatrix;
d = d * worldMatrix;
rayOrigin.set(o);
rayDirection.set(d);
rayDirection.normalize();
break;
}
}[/CODE]

Basically you give the function the screen coordinates, a ray origin vector, a ray direction vector and the space you want the ray in (world or local). Since this is the camera we are talking about, the local space is effectively the view space.

So, first you get the ray in view space. For that we need the projection matrix of the camera, the screen coordinates and the screen size. If you are cool with with view space you just fill in the vectors and you are good to go.

If you want the ray in world space you need to transform it with the world matrix, aka "local-to-world-matrix", aka "view-to-world-matrix". If you happen to have the "world-to-view-matrix" instead, like most camera implementations, the one you want is just the inverse of that. You fill out two 4D vectors (the origin with W=1, the direction with W=0) and transform them by the matrix. After that you should probably normalize the direction.

Voila, you now have the "start-point" of the ray (which is still just the camera position btw) and a unit-length direction vector of the ray, both in world space. Multiply the direction with whatever huge number you want to get a long ray cast.

Does this help? Edited by GuyWithBeard

##### Share on other sites
Tispe    1468
[quote name='Medo3337' timestamp='1354548862' post='5006639']
@Tispe: This code is dealing with model transformation, I'm not doing anything related to models, instead, I'm trying to raycast from the center of the camera to the forward area.
[/quote]

The matrices are not only for mesh/models. You use them in inverse to get from screen coordinates (your crosshair) to world space, origin and direction vectors.

##### Share on other sites
GuyWithBeard    1890
Yes, if you look closely at Tispe's code you'll notice that it does exactly the same as mine. With the added bonus that it probably compiles for you out of the box, if you use D3DX.

##### Share on other sites
Medo Mex    891
Sorry for the delay, I have tried to implement the idea from the above code but couldn't get it to work, If you can just show example on getting the two values RayFrom and RayTo it would be greatly appreciated:

[CODE]float distance = 1000.0f;
D3DXVECTOR RayFrom; // According to the screen "+" symbol (middle of the screen)
D3DXVECTOR RayTo; // According to distance[/CODE]

##### Share on other sites
Tispe    1468
Since your "+" symbol (cross-hair) is in the middle of the screen xAngle and yAngle are both 0.

You only need this code
[CODE]
D3DXVECTOR3 origin, direction;
origin = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
direction = D3DXVECTOR3(0, 0, 1.0f);
// find the inverse matrix
D3DXMatrixInverse(&matInverse, NULL, &(matWorld * matView)); //D3DXMatrixInverse(&matInverse, NULL, &matView); <- world space instead.
// convert origin and direction into model space
D3DXVec3TransformCoord(&origin, &origin, &matInverse);
D3DXVec3TransformNormal(&direction, &direction, &matInverse);
D3DXVec3Normalize(&direction, &direction);
[/CODE]

You can remove matWorld and you will get the origin and direction in world space. By keeping matWorld you reverse the ray into the model space of the mesh. This is useful if you want to check if the mesh is intersecting with the ray.

##### Share on other sites
Medo Mex    891
Okay, I tried the following:
[CODE]
D3DXVECTOR3 origin, direction;
origin = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
direction = D3DXVECTOR3(0, 0, 1.0f);
// find the inverse matrix
D3DXMATRIX matInverse;
D3DXMATRIX matView = camera->GetViewMatrix();
D3DXMatrixInverse(&matInverse, NULL, &matView); //D3DXMatrixInverse(&matInverse, NULL, &matView); <- world space instead.
// convert origin and direction into model space
D3DXVec3TransformCoord(&origin, &origin, &matInverse);
D3DXVec3TransformNormal(&direction, &direction, &matInverse);
D3DXVec3Normalize(&direction, &direction);
DWORD hitResult = Raytest(origin, direction);
// Raytest() should return the ID of the mesh that was hit when the player is shooting
[/CODE]

What's wrong in the above code? it's not hitting the correct mesh, also where is the distance? I'm trying to get hitResult so I can know which mesh was hit then I will apply damage to that mesh.

##### Share on other sites
Tispe    1468
The Ray just has an origin and a direction, it goes on for infinity. When you do an intersect test you provide the mesh and the ray, and the function tests if the Ray intersects that mesh and at what distance away from the origin. You should use D3DXIntersect() unless you you completly trust your Raytest() function.

I think you can't test if you hit a mesh in World Space. You need to include matWorld when you reverse the ray from screen to model space. That way the Ray will be in the Model space, and only there can you test if the Ray intersect any polygons of the mesh.

So you gotta do this Ray test for each mesh, that means reverse transform the Ray for each model using a new matWorld. Edited by Tispe

##### Share on other sites
Medo Mex    891
[QUOTE]I think you can't test if you hit a mesh in World Space. You need to include matWorld when you reverse the ray from screen to model space.[/QUOTE]

That's alittle confusing, I just use Bullet Physics Raytest which create a line in the world space and I can use it to get the closest hit and it return information about which mesh that was hit, I do nothing other than giving the start and end point and it tells me which mesh was intersected. Edited by Medo3337

##### Share on other sites
GuyWithBeard    1890
[quote name='Tispe' timestamp='1355000512' post='5008600']
I think you can't test if you hit a mesh in World Space.
[/quote]

As far as I know, Bullet expects the ray origin and direction in world space. Also, I don't see why you could not do that in any case. Spaces are just different frames of reference. The important thing is that all data is in the same space.

A few questions for Medo3337:

1. Do your physics meshes have the exact same geometry as your visual meshes? Most games use vastly simplified physics meshes compared to their visual counterparts. This of course means that doing ray casts with the physics engine will not give you the exact face you are seeing on the screen, but the face of the physics mesh. Also, I don't remember if Bullet gives you faces at all, or just the rigid body, but you probably know that better than me.

2. If you have problems getting the right ray, have you tried drawing a line where it is going, to see if it is correctly lined up? Of course you cannot see the ray as you are casting it, since you are effectively looking along the ray itself, so it would appear as just a dot. Instead you can set up your game to cast a ray when you left click, or something like that. The ray info would then be stored in two vectors and drawn on the screen until you cast the next ray. That way you will have time to move your camera a bit to the side so that you can see the ray.

##### Share on other sites
Medo Mex    891
I'm using btBvhTriangleMeshShape for the terrain and btConvexHullShape for some other meshes, one important question, what physics shape should I use for buildings, airstrikes, vehicles and other interiors such as tables, chairs, etc? Remember when the player shoot I have to place a bullet hole billboard on exactly the same point the player is shooting at and since raytest has nothing to do with rendering and it's working according to bullet physics, I have to choose the right shapes for loading physics meshes so when the bullet hole is placed, its position should be realistic.

I'm drawing the line and I see that I'm getting somewhere close to doing the right thing, here is my code:
[CODE]inline D3DXVECTOR3 GetPickRay(float x, float y)
{
D3DXMATRIX m_matProj = camera->projectionMatrix();
D3DXMATRIX m_matView = camera->viewMatrix();

D3DXVECTOR3 coord;
coord.x = ( ( ( 2.0f * x ) / WND_WIDTH ) - 1 );
coord.y = -( ( ( 2.0f * y ) / WND_HEIGHT ) - 1 );
coord.z = 1.0f;
// Back project the ray from screen to the far clip plane
coord.x /= m_matProj._11;
coord.y /= m_matProj._22;
D3DXMATRIX matinv;
D3DXMatrixInverse(&amp;matinv, NULL, &amp;m_matView);
coord*=1000000.0f;
D3DXVec3TransformCoord(&amp;coord, &amp;coord, &amp;matinv);
return coord;
}

POINT m_cursorCoords;
GetCursorPos(&amp;m_cursorCoords);
D3DXVECTOR3 m_rayFrom = camera->GetPosition();
D3DXVECTOR3 m_rayTo = GetPickRay(WND_WIDTH / 2, WND_HEIGHT / 2);
btVector3 btRayFrom = btVector3(m_rayFrom.x, m_rayFrom.y, m_rayFrom.z);
btVector3 btRayTo = btVector3(m_rayTo.x, m_rayTo.y, m_rayTo.z);

if (MouseLeftButtonPressed())
{
DWORD dwHit = BuletRaytest(m_rayFrom, m_rayTo);

// dwHit should store the mesh that got shoot, create bullet hole billboard and apply damage here...
}
[/CODE]

The above code is not shooting from the center of the screen, it's somehow close. Edited by Medo3337

##### Share on other sites
GuyWithBeard    1890
Well, there are a few problems with your approach. First of all it's a lot heavier to physically simulate convex hull shapes than, say, boxes. A lot of objects can be simplified vastly on the physics side. For example, characters are often simulated as capsules until they, for one reason or another become rag dolls.

Another problem is that btConvexHullShape can only represent convex shapes, which means that any concave geometry you may have will be wrong anyway.

Things like this make using the physics engine for placing decals (bullet holes etc.) a bad choice. It's usually better to perform the ray casting on the graphics layer. If you update your D3DX stuff to the new DirectXMath library (which is basically the new version of D3DX) you get a lot of collision detection functionality for free. What was once called xnacollision is now included in the core lib and provides, among other things, ray to triangle intersection which is probably what you want for placing decals.

##### Share on other sites
Medo Mex    891
Do you mean that I should use D3DXIntersect() to make the player shooting instead of using Bullet Physics Raytest? I believe that bullet raytest is used in picking/weapon shooting/etc...

##### Share on other sites
GuyWithBeard    1890
D3DXIntersect is part of D3DX. I suggested that you replace D3DX with the newer and more powerful version, DirectXMath.

You can of course use bullet to perform the ray casts, but remember that there might be situations where you cannot (easily) represent the visual objects using the physics meshes. Bullet (or any other physics engine) is useful for ray casting when you want to know if a bullet shot from a weapon has hit an object, but it's less useful when you want the exact position on a surface of a mesh, for placing a bullet hole decal. Of course, all this depends on how your physics meshes, and indeed all your meshes, are constructed. Some games might very well use the same geometry for both physics and visuals.

http://www.esenthel.com/wiki/index.php?title=Physical_Body

Check out the images in the article above. As you see the physical meshes are often not the same as the visual ones. For example casting a ray onto the capsule around the character and placing a bullet hit billboard at the hit location would make the billboard float in the air, instead of being drawn on the character.

And as you probably know, the character in the image above cannot be represented with a btConvexHullShape.

##### Share on other sites
Medo Mex    891
According to:
[url="http://msdn.microsoft.com/en-us/library/windows/desktop/hh437833(v=vs.85).aspx"]http://msdn.microsof...3(v=vs.85).aspx[/url]

[i]The DirectXMath library is designed for C++ developers working on games and DirectX graphics in Windows Store apps and traditional desktop apps for [b][u]Windows 8 and later[/u][/b].[/i]

I want the game to work on earlier versions of Windows such as Windows XP and Windows 7, so I guess DirectXMath will not be the choice.

Here is the code that I'm using:
[CODE]
// When the user press the left mouse button, this code will be executed on EVERY mesh
// matWorld represent the current mesh world matrix
// Shooting
// get the current transform matrices
D3DXMATRIX matProjection, matView, matWorld, matInverse;
m_d3dDevice->GetTransform(D3DTS_PROJECTION, &matProjection);
m_d3dDevice->GetTransform(D3DTS_VIEW, &matView);
m_d3dDevice->GetTransform(D3DTS_WORLD, &matWorld);
// use the mouse coordinates to get the mouse angle
POINT MousePos;
GetCursorPos(&MousePos);
float xAngle = (((2.0f * MousePos.x) / WND_WIDTH) - 1.0f) / matProjection(0, 0);
float yAngle = (((-2.0f * MousePos.y) / WND_HEIGHT) + 1.0f) / matProjection(1, 1);
D3DXVECTOR3 origin, direction;
origin = D3DXVECTOR3(0.0f, 0.0f, 0.0f);
direction = D3DXVECTOR3(xAngle, yAngle, 1.0f);
// find the inverse matrix
D3DXMatrixInverse(&matInverse, NULL, &(matWorld * matView));
// convert origin and direction into model space
D3DXVec3TransformCoord(&origin, &origin, &matInverse);
D3DXVec3TransformNormal(&direction, &direction, &matInverse);
D3DXVec3Normalize(&direction, &direction);

DWORD fi;
FLOAT u, v;
FLOAT dist;
BOOL hit;
D3DXIntersect(staticMesh,&origin,&direction,&hit,&fi,&u,&v,&dist,NULL,NULL);
if(hit)
{
stringstream ss;
ss << "Mouse over: " << currentMeshId;
SetWindowText(GetForegroundWindow(), ss.str().c_str());
}

[/CODE]

Tried the above code but it's not working exactly as expected, what's wrong with that code? Edited by Medo3337