I'm trying to implement frustum culling, however I can't figure out where/how it's breaking.

Frustum culling on
How it should look

It's almost like it's rendering the opposite of what is expected, however that's not constantly the case. It's highly dependent on the view rotation and position.

Frustum code

Frustum viewFrustum(renderContext.projectionMatrix()*renderContext.viewMatrix());

Frustum::Frustum(const glm::mat4& matrix)
{
    // Extract the frustum faces from the projection*view matrix
    // http://web.archive.org/web/20120531231005/http://crazyjoke.free.fr/doc/3D/plane%20extraction.pdf
    mPlanes[Face::LEFT].n.x = matrix[3][0] + matrix[0][0];
    mPlanes[Face::LEFT].n.y = matrix[3][1] + matrix[1][0];
    mPlanes[Face::LEFT].n.y = matrix[3][2] + matrix[2][0];
    mPlanes[Face::LEFT].d = matrix[3][3] + matrix[3][0];

    mPlanes[Face::RIGHT].n.x = matrix[3][0] - matrix[0][0];
    mPlanes[Face::RIGHT].n.y = matrix[3][1] - matrix[1][0];
    mPlanes[Face::RIGHT].n.y = matrix[3][2] - matrix[2][0];
    mPlanes[Face::RIGHT].d = matrix[3][3] - matrix[3][0];

    mPlanes[Face::DOWN].n.x = matrix[3][0] + matrix[0][1];
    mPlanes[Face::DOWN].n.y = matrix[3][1] + matrix[1][1];
    mPlanes[Face::DOWN].n.y = matrix[3][2] + matrix[2][1];
    mPlanes[Face::DOWN].d = matrix[3][3] + matrix[3][1];

    mPlanes[Face::UP].n.x = matrix[3][0] - matrix[0][1];
    mPlanes[Face::UP].n.y = matrix[3][1] - matrix[1][1];
    mPlanes[Face::UP].n.y = matrix[3][2] - matrix[2][1];
    mPlanes[Face::UP].d = matrix[3][3] - matrix[3][1];

    mPlanes[Face::FAR].n.x = matrix[3][0] - matrix[0][2];
    mPlanes[Face::FAR].n.y = matrix[3][1] - matrix[1][2];
    mPlanes[Face::FAR].n.y = matrix[3][2] - matrix[2][2];
    mPlanes[Face::FAR].d = matrix[3][3] - matrix[3][2];

    mPlanes[Face::NEAR].n.x = matrix[3][0] + matrix[0][2];
    mPlanes[Face::NEAR].n.y = matrix[3][1] + matrix[1][2];
    mPlanes[Face::NEAR].n.y = matrix[3][2] + matrix[2][2];
    mPlanes[Face::NEAR].d = matrix[3][3] + matrix[3][2];

    for (int i = 0; i < 6; i++)
    {
        mPlanes[i].normalize();
    }
}

bool Frustum::sphereIntersects(const glm::vec3& pos, float radius)
{
    // cheack against each plane
    for (int i = 0; i < 6; i++)
    {
        if (glm::dot(mPlanes[i].n, pos) + mPlanes[i].d + radius <= 0)
        {
            return false;
        }
    }

    return true;
}

Where I check each point

            glm::vec4 point = renderContext.projectionMatrix()*renderContext.viewMatrix() * 
                glm::vec4((float)x*World::CHUNK_SIZE, (float)c*World::CHUNK_SIZE, (float)z*World::CHUNK_SIZE, 0);

            
            if (!viewFrustum.sphereIntersects(glm::vec3(point.x, point.y, point.z), 16.f))
            {
                continue;
            }

EDIT: Looks like I had some errors in my matrix calculation for the frustum. My current issue is still if I rotate the camera upward (+y) it culls what should be visible. If the camera is horizontal (y ~= 0) or if the camera is rotated downward (-y) it renders as expected.

Latest code: https://github.com/rcapote/Survivox/tree/master/src

Original text:

I did some more research and testing, still no success but here is what I have gathered. I also fixed the issue where I was setting y twice o.o

Multiplying the point by the proj*view before testing it against the frustum is incorrect. According to the examples I've seen, I should be able to just pass in the world position of the test point...I think. I've removed that from my code.

It seems as if the problem is that coordinate spaces are incorrect, such that the point I'm testing against is different from where it's getting rendered. The culling seems to happens with a small area near the origin, instead of the position that the meshes are being rendered at. I can't pin-point this, however. The position I'm rendering and the position I'm testing are exactly the same, save the proj*view transform. My guess would be that multiplying the test point against the matrix would solve the issue...but it only exacerbates it. I've tried increasing the testing positions by several magnitudes, virtually no difference in results.

Another symptom is that if I look "downward" (towards -y) there is no culling, however looking up causes aggressive culling. This only happens from one direction (+x, +z, looking down -z). If I move to the other end and look down +z...the opposite is true: looking up causes no culling, while looking down causes aggressive culling.

Updated code:

Frustum::Frustum(const glm::mat4& matrix)
{

    
    // Extract the frustum faces from the projection*view matrix
    // http://web.archive.org/web/20120531231005/http://crazyjoke.free.fr/doc/3D/plane%20extraction.pdf
    // GLM is column major
    mPlanes[Face::LEFT].n.x = matrix[0][3] + matrix[0][0];
    mPlanes[Face::LEFT].n.y = matrix[1][3] + matrix[1][0];
    mPlanes[Face::LEFT].n.z = matrix[2][3] + matrix[2][0];
    mPlanes[Face::LEFT].d = matrix[3][3] + matrix[3][0];

    mPlanes[Face::RIGHT].n.x = matrix[0][3] - matrix[0][0];
    mPlanes[Face::RIGHT].n.y = matrix[1][3] - matrix[1][0];
    mPlanes[Face::RIGHT].n.z = matrix[2][3] - matrix[2][0];
    mPlanes[Face::RIGHT].d = matrix[3][3] - matrix[3][0];

    mPlanes[Face::DOWN].n.x = matrix[0][3] + matrix[0][1];
    mPlanes[Face::DOWN].n.y = matrix[1][3] + matrix[1][1];
    mPlanes[Face::DOWN].n.z = matrix[2][3] + matrix[2][1];
    mPlanes[Face::DOWN].d = matrix[3][3] + matrix[3][1];

    mPlanes[Face::UP].n.x = matrix[0][3] - matrix[0][1];
    mPlanes[Face::UP].n.y = matrix[1][3] - matrix[1][1];
    mPlanes[Face::UP].n.z = matrix[2][3] - matrix[2][1];
    mPlanes[Face::UP].d = matrix[3][3] - matrix[3][1];

    mPlanes[Face::FAR].n.x = matrix[0][3] - matrix[0][2];
    mPlanes[Face::FAR].n.y = matrix[1][3] - matrix[1][2];
    mPlanes[Face::FAR].n.z = matrix[2][3] - matrix[2][2];
    mPlanes[Face::FAR].d = matrix[3][3] - matrix[3][2];

    mPlanes[Face::NEAR].n.x = matrix[0][3] + matrix[0][2];
    mPlanes[Face::NEAR].n.y = matrix[1][3] + matrix[1][2];
    mPlanes[Face::NEAR].n.z = matrix[2][3] + matrix[2][2];
    mPlanes[Face::NEAR].d = matrix[3][3] + matrix[3][2];

    for (int i = 0; i < 6; i++)
    {
        mPlanes[i].normalize();
        
    }
}

bool Frustum::sphereIntersects(const glm::vec3& pos, float radius)
{
    // cheack against each plane
    for (int i = 0; i < 6; i++)
    {
        
        if (glm::dot(pos, mPlanes[i].n) + mPlanes[i].d + radius <= 0.f)
        {
            return false;
        }
    }

    return true;
}

            glm::vec4 point = /*renderContext.projectionMatrix()*renderContext.viewMatrix() */
                glm::vec4((float)x*World::CHUNK_SIZE, 
                          (float)c*World::CHUNK_SIZE, 
                          (float)z*World::CHUNK_SIZE, 1);

            
            if (!viewFrustum.sphereIntersects(glm::vec3(point.x, point.y, point.z), 16.f))
            {
                continue;
            }

Interesting....that seemed to have fixed all my problems. Resulting code:

            glm::vec4 point = renderContext.projectionMatrix()*renderContext.viewMatrix() *
                glm::vec4((float)x*World::CHUNK_SIZE+8, 
                          (float)c*World::CHUNK_SIZE-8, 
                          (float)z*World::CHUNK_SIZE+8, 1);

            point = (glm::normalize(point)) + glm::vec4(0.5f);

            Frustum viewFrustum(renderContext.projectionMatrix()*renderContext.viewMatrix()*transform);

            if (viewFrustum.sphereIntersects(glm::vec3(point.x, point.y, point.z), 14.f))

Screenshots

Yea, I have no idea what I was doing. It's working, most of the meshes are being culled, but I think there are too many false positives. For example, I'm looking at a section that should only be rendering 4 meshes...but it's rendering around ~20. That's out of 200 meshes in the scene, so it's a significant improvement. That 14.f is the minimum radius such that the culling isn't visible on the screen.

Basically I took what you said about multiplying the point by the matrices, bringing it down to 0...1 range and checking if it's within that range. That was an improvement on what I had before. It only rendered where the camera of looking, which was much better than what I had before...however it was only rendering a small number of meshes in that direction...and none of the meshes that were in the peripheral vision.

So I just started poking around, changing values, moving stuff around...until I got the above which seems to be working decently enough, but I don't think it's strictly correct. I'm still open to alternative fixes