I've got some issues trying to see if my AABBs are inside the view frustum or not using AABB-plane intersection. No matter what, atleast some plane always returns 'back' during testing of a AABB in clear view.

Here's how I construct a plane and test against AABB:

Plane::Plane(const Vec3& point1, const Vec3& point2, const Vec3& point3) : mNormal(glm::normalize(glm::cross(point2 - point1, point3 - point2))), mDistance(glm::dot(mNormal, point1))
{
}


Plane::PlaneIntersection Plane::Intersection(const AABB& aabb) const
{
	const float radius = aabb.mAABBExtent.x * glm::abs(mNormal.x) +
			     aabb.mAABBExtent.y * glm::abs(mNormal.y) +
			     aabb.mAABBExtent.z * glm::abs(mNormal.z);

	const float distance = glm::dot(mNormal, aabb.mAABBCenter) - mDistance;

	if (glm::abs(distance) <= radius)
		return PlaneIntersection::Intersect;

	return distance > radius ? PlaneIntersection::Front : PlaneIntersection::Back;
}

Getting the six planes of a view frustrum, I first calculate the corners and then use three points to get each plane:

// std::array<Vec3, 8> - 8 corners of the view frustum
CameraFrustrum frusturm = CalculateCameraFrustrum(mRenderQueue.mCamera.mCameraViewProjectionMatrix);

// ntl, ntr, nbr
Plane nearP(Vec3(frusturm.at(1)), Vec3(frusturm.at(2)), Vec3(frusturm.at(3)));
// fbl, ftl, ftr
Plane farP(Vec3(frusturm.at(4)), Vec3(frusturm.at(5)), Vec3(frusturm.at(6)));

// nbl, fbl, ftl
Plane leftP(Vec3(frusturm.at(0)), Vec3(frusturm.at(4)), Vec3(frusturm.at(5)));
// ntr, ftr, fbr
Plane rightP(Vec3(frusturm.at(2)), Vec3(frusturm.at(6)), Vec3(frusturm.at(7)));

// ntl, ftl, ftr
Plane topP(Vec3(frusturm.at(1)), Vec3(frusturm.at(5)), Vec3(frusturm.at(6)));
// nbr, fbr, fbl
Plane bottomP(Vec3(frusturm.at(3)), Vec3(frusturm.at(7)), Vec3(frusturm.at(4)));

Then for each plane, use intersection() above. As I understand it, if even one of the plane intersection tests is returned 'Back', the AABB is not considered inside the frustrum.

Look at the alien, its AABB center about about Vec3(0, 1, -2) in the picture, camera standing at origo, looking down Z:

[attachment=27430:post-182913-0-44780300-1426029853.png]

Apparently the Far and Bottom planes says the alien is 'Back' , i.e not in the frustrum.

What am I doing wrong?

As a quick guess, the winding order of the points you have used for the frustum planes are inconsistent. Check that either all plane normals point inward on the volume, or outward (depending on how you like to test inside/outside)

The Bottom planes normal is Vec3(0.02, 0.8, -0,57) and its D is 2.45.

The far plane normal is Vec3(0.003, -0.003, -0.9999) and its D is 100.

Both are reported as being outside.

I find visualizing planes somewhat difficult. How do I know if the normal is pointing inward on the frustrum?

I guess part of the confusion is I dont fully understand what the D parameter really is. Sources keeps refering to it as 'distance' - in the direction of the normal? If so, how could the far plane ever point inwards?

Its distance from the origin (0,0,0) to the plane in the direction of the planes normal.

So, assuming you are facing down +Z (0,0,1) the normal for the near plane would be +Z (0,0,1) while the normal for the far plane would be -Z (0,0,1) in order to make them both point inward. As you move forward or backward along the Z plane, those normals don't change, but the D parameter does.

Completely ignore the fact that the coordinates are 3d and think of values along a ruler. If you have a section of a ruler, then from the left side to the right side you are incrementing values (1), from the right to the left you are decrementing values (-1). The distance along the ruler * either 1 or -1 depending on the side of the is the D value.

OK so I got the plane-from-three-points working fine.

Then I tried the fast extract-planes-from-transform method suggested here - http://gamedevs.org/uploads/fast-extraction-viewing-frustum-planes-from-world-view-projection-matrix.pdf

I got the a,b,c coefficients alright and got the correct normal vector from it.

But the distance D from the origin looks completely bonkers. From wolfram (http://mathworld.wolfram.com/Plane.html) the equation to get D is:

D = d / (sqrt(a^2) + sqrt(b^2) + sqrt(c^2))

where a,b,c are x,y,z of the normalized normal vector of the plane, like this:

// xyzw = abcd
Plane::Plane(const Vec4& planeCoefficients) :
	mNormal(glm::normalize(Vec3(planeCoefficients.x, planeCoefficients.y, planeCoefficients.z))),
	mDistance(planeCoefficients.w / (glm::sqrt(mNormal.x * mNormal.x + mNormal.y * mNormal.y + mNormal.z * mNormal.z)))
{
}

Suprise suprise, the mDistance looks bonkers. For the nearplane it computes into -0.1f and for the far plane it is 0.1f - (where proj matrix is near=0.1f and far 100.0f).

What am I missing?

(Also - a note of advice for anyone using the linked pdf to get frustum planes... the matrix layout is confusing and counter-intuitive, instead just use the table at the end of chaper 2)