# inverse world matrix and per-triangle picking

This topic is 1957 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Once again I'm having fun with linear algebra...or so I think right now.

I have picking methods in my level editor that work great for my mesh objects, except when it comes to scaled meshes. I apply an inverse of the mesh's world transform matrix to get the ray from world space into object space, and rotation/translation don't give it any problems. When I scale a mesh up in size, the algorithm no longer lets the ray hit the mesh correctly, and I get a lot of null results on the collision check.

I'm assuming its because the inverse of a scale operation is going to do something wierd to the ray, as I can't grasp at the moment what it's going to do to a single point...I'm assuming scale it relative to (0,0,0) which will...wait, no, that would bring a correctly un-rotated/un-translated ray closer to the object's origin, which makes sense for a scaled up mesh relative to its base 3d coords.

Okay, I don't know then. I'm likely missing something else in the code like the last time I had a question on matrices and vector spaces. But if my guess is off, I'd be happy with any enlightenment.

##### Share on other sites
Hi,

Can you show some code? I use similar method in my project and it works as intended. Before testing each mesh, I transform the pick ray by the inverse world matrix of the model and then test the meshes of the model in the model space.

Cheers!

##### Share on other sites
Here's where I actually calculate the picked prop object (static level mesh object):
 private Prop intersectedProp(Map map) { float? nearestResult = null; Prop returnProp = null; foreach (Prop prop in map.propList) { BoundingSphere bsphere = new BoundingSphere(prop.Model.Midpoint + prop.Position, prop.Model.Radius); if (pickingRay.Intersects(ref bsphere)) { float tempResult = perTriangleModelIntersect(prop.Model, pickingRay); if (tempResult != 0) { if (nearestResult == null || tempResult < nearestResult) { nearestResult = tempResult; returnProp = prop; } } } } return returnProp; } 

And here's the method that does the per-triangle intersection calculation:

 private float perTriangleModelIntersect(Mesh model, Ray ray) { float result = 0; Matrix inverseWorld = Matrix.Invert(model.GetWorld()); Vector3 transRayStart = Vector3.TransformCoordinate(ray.Position, inverseWorld); Vector3 origRayEnd = ray.Position + ray.Direction; Vector3 transRayEnd = Vector3.TransformCoordinate(origRayEnd, inverseWorld); Ray inverseRay = new Ray(transRayStart, transRayEnd - transRayStart); VertexModel[] verts = model.RawVertices; int[] indices = model.RawIndices; for (int i = 0; i < indices.Length; ) { float tempResult; Vector3 v1 = verts[indices[i++]].Position; Vector3 v2 = verts[indices[i++]].Position; Vector3 v3 = verts[indices[i++]].Position; inverseRay.Intersects(ref v1, ref v2, ref v3, out tempResult); if (tempResult != 0) { if (result == 0) { result = tempResult; } else if (tempResult < result) { result = tempResult; } } } return result; } 

I'm going to test and see if it's the boundingsphere intersect check that's tripping things up, I didn't expect it since the bsphere is just built from a centerpoint and radius (which are calculated correctly when the mesh class is built) ...except crap, the radius field doesn't get modified when I scale the object. I might be back soon with a fix, but feel free to look this over anyway.

##### Share on other sites
*sigh* yea, it's the boundingsphere. (if I get rid of the check, picking works pixel-perfect). And the centerpoint is not being calculated correctly, I'm horrible at conceptualizing edge cases, until I walk away from an algorithm for a few days and take another look at it, like now. Going back to the drawing board on that one.

SO, if anyone's having a similar issue and is wondering: applying an inverse of your mesh's world transform to your picking ray WILL put you in the object space of the mesh, so that's still useful.