Once again I'm having fun with linear algebra...or so I think right now.

I have picking methods in my level editor that work great for my mesh objects, except when it comes to scaled meshes. I apply an inverse of the mesh's world transform matrix to get the ray from world space into object space, and rotation/translation don't give it any problems. When I scale a mesh up in size, the algorithm no longer lets the ray hit the mesh correctly, and I get a lot of null results on the collision check.

I'm assuming its because the inverse of a scale operation is going to do something wierd to the ray, as I can't grasp at the moment what it's going to do to a single point...I'm assuming scale it relative to (0,0,0) which will...wait, no, that would bring a correctly un-rotated/un-translated ray closer to the object's origin, which makes sense for a scaled up mesh relative to its base 3d coords.

Okay, I don't know then. I'm likely missing something else in the code like the last time I had a question on matrices and vector spaces. But if my guess is off, I'd be happy with any enlightenment.

Here's where I actually calculate the picked prop object (static level mesh object):

private Prop intersectedProp(Map map)
{
float? nearestResult = null;
Prop returnProp = null;
foreach (Prop prop in map.propList)
{
BoundingSphere bsphere = new BoundingSphere(prop.Model.Midpoint + prop.Position, prop.Model.Radius);
if (pickingRay.Intersects(ref bsphere))
{
float tempResult = perTriangleModelIntersect(prop.Model, pickingRay);
if (tempResult != 0)
{
if (nearestResult == null || tempResult < nearestResult)
{
nearestResult = tempResult;
returnProp = prop;
}
}
}
}
return returnProp;
}


And here's the method that does the per-triangle intersection calculation:


private float perTriangleModelIntersect(Mesh model, Ray ray)
{
float result = 0;
Matrix inverseWorld = Matrix.Invert(model.GetWorld());
Vector3 transRayStart = Vector3.TransformCoordinate(ray.Position, inverseWorld);
Vector3 origRayEnd = ray.Position + ray.Direction;
Vector3 transRayEnd = Vector3.TransformCoordinate(origRayEnd, inverseWorld);
Ray inverseRay = new Ray(transRayStart, transRayEnd - transRayStart);
VertexModel[] verts = model.RawVertices;
int[] indices = model.RawIndices;
for (int i = 0; i < indices.Length; )
{
float tempResult;
Vector3 v1 = verts[indices[i++]].Position;
Vector3 v2 = verts[indices[i++]].Position;
Vector3 v3 = verts[indices[i++]].Position;
inverseRay.Intersects(ref v1, ref v2, ref v3, out tempResult);

if (tempResult != 0)
{
if (result == 0)
{
result = tempResult;
}
else if (tempResult < result)
{
result = tempResult;
}
}
}
return result;
}


I'm going to test and see if it's the boundingsphere intersect check that's tripping things up, I didn't expect it since the bsphere is just built from a centerpoint and radius (which are calculated correctly when the mesh class is built) ...except crap, the radius field doesn't get modified when I scale the object. I might be back soon with a fix, but feel free to look this over anyway.

*sigh* yea, it's the boundingsphere. (if I get rid of the check, picking works pixel-perfect). And the centerpoint is not being calculated correctly, I'm horrible at conceptualizing edge cases, until I walk away from an algorithm for a few days and take another look at it, like now. Going back to the drawing board on that one.

SO, if anyone's having a similar issue and is wondering: applying an inverse of your mesh's world transform to your picking ray WILL put you in the object space of the mesh, so that's still useful.