I'm doing view frustum culling using this fairly standard algorithm
AABBIntersection Intersection(const AABB& aabb, const Mat4& frustumMatrix)
{
const Vec4 rowX(frustumMatrix[0].x, frustumMatrix[1].x, frustumMatrix[2].x, frustumMatrix[3].x);
const Vec4 rowY(frustumMatrix[0].y, frustumMatrix[1].y, frustumMatrix[2].y, frustumMatrix[3].y);
const Vec4 rowZ(frustumMatrix[0].z, frustumMatrix[1].z, frustumMatrix[2].z, frustumMatrix[3].z);
const Vec4 rowW(frustumMatrix[0].w, frustumMatrix[1].w, frustumMatrix[2].w, frustumMatrix[3].w);
AABBIntersection ret(AABBIntersection::Inside);
// left, right, bottom, top, near, far planes
std::array<Vec4, 6> planes = { rowW + rowX, rowW - rowX, rowW + rowY, rowW - rowY, rowW + rowZ, rowW - rowZ };
float d = 0.0f, r = 0.0f;
for (const Vec4& plane : planes)
{
d = glm::dot(Vec3(plane), aabb.mAABBCenter);
r = glm::dot(Vec3(glm::abs(plane)), aabb.mAABBExtent);
if (d - r < -plane.w)
ret = AABBIntersection::Partial;
if (d + r < -plane.w)
return AABBIntersection::Outside;
}
return ret;
}
The frustrumMatrix is the Local-World-View-Projection matrix and the aabb is simply in Local space. This works fine even if the world matrix is scaled or translated. However if a rotation component is added then the test is no longer working.
Why is this? And is there anything I can do to modify the frustrumMatrix to make it play nice with rotations aswell? I could apply a rotation matrix to the aabb but I would very much like to avoid that and only as a last resort if it cannot be done by modifying the frustrumMatrix...
Any ideas?
Thanks
