I'm making a map editor that uses brushes like in BSP.

How do you determine which polygons are covered up by one or several brushes, either because of coplanar brushes or because they face a volume covered up from the entity-side of the map, forming an enclosure.

It's my understanding that brushes are defined as planes forming a convex hull, as opposed to polygons, and it is the intersection of these planes that produces polygons from the brush. It shouldn't be an issue eliminating polygons in an editor, because the BSP compiler itself will properly intersect all the planes and deduce which polygons form the visible surfaces on the inside of the map.

If the issue is rendering the brushes in the editor, you might have to perform some simple BSP-style intersection, maybe building a sort of crude BSP from in the editor that is used specifically for rendering, re-building (or updating) it when the user moves/changes/creates/deletes a brush.

EDIT: If you're talking about optimizing editor rendering, I would just do a screenspace occlusion query for each brush, sorting them front to back, and with each brush creating a minimum occluding rectangle on the screen, until there is no more space on the screen that any brushes intersect with their maximum bounding rectangle.

Not sure if this is exactly what you're after but here's a link to a .pdf which describes how to perform constructive solid geometry on brushes to remove coplanar surfaces.

http://mattn.ufoai.org/files/MAPFiles.pdf

The start of the .pdf describes thw .map file format (Quake engine games). Further in it explains how to perform CSG.

Matt

This is exactly what I was looking for. Thanks.