I ran into some interesting bugs and issues while debugging it :
- My code initially didn't correctly handle two frustums pointing from cell A to cell B. If an object was only visible through cell B and A was processed first, the object disappeared.
- The code finds the cells containing the camera point, but culls portals based on the camera's view frustum. See a problem? The space between the near plane and the camera location was the issue. If a portal lay between the camera & the near plane, it was culled, and large parts of the level would disappear from view. The fix was to add a new culling method to my frustum class that skipped the near plane.
- Negative w values. When transforming the portals to Normalized Device Coordinate space, 0 w values might wreak havok, and dividing a vertex position by a negative w puts it in front of the camera but mirrored. One solution is to actually clip the portal to the near plane, but a simpler one is, for vertices behind the near plane - force them to the edges of the screen. If the x < 0, make x = -1, else x = 1; etc.
The last thing we accomplished was to make the anti-portals work. These are quads that have a frustum constructed from the viewpoint and the face and edges quad itself. For this I needed to add another frustum culling method - this one that would skip the far plane.
After all potentially visibly cells are processed, I find all anti-portals in the view frustum, and then cull them vs the cell / portal system. Any surviving ones get added to a per-frame list of active anti-portals.
When an object survives the view frustum & cell checks, it then gets tested vs each anti-portal. If one anti-portal frustum completely encloses the box, the box is considered culled.
I need to a check so that if the camera is too close to the plane of the quad to skip anti-portal checks with it, b/c the frustum created will be invalid.
In the shots below, the anti-portal is red, and is embeded in a solid wall.