I currently have a similar problem. I have an app that draws some fractals, one of them is Menger Sponge. To keep things easy to code I made it out of many cubes. At later iterations there is a staggering amount of faces that will never be seen. At the 3rd iteration there are 8,000 cubes which my phone handles fine (Nexus 5, 50+ fps) but the next iteration there are 160,000 cubes which makes the fps drop to about 20 - which was sort of ok but now I've attempted to add shadows which it really can't deal with with that much stuff going on.

Depending on your model you can optimize it a bit. I briefly tried a voxel based version of my own using points instead of cubes and for that I was able to work out if a voxel is visible or not and if it isn't I removed it from being drawn. I effectively hollowed it out. My model was 8 million points but once hollowed it was simply the points on the surface and it runs really well (until I started carving it to higher iterations of the Menger Sponge.

If you have neighbouring information you could work out how many visible neighbours there are and choose an appropriate 'cube' - e.g. a 1 faces cube when there are 5 neighbours, 5 faces when there are 1 and so on as you suggested.

Your thread inspired me to try and improve the performance of my own app. I got it to use 65% less vertices/faces, which when I start with 5.7 million that's a great improvement. The way I did it was to have 'cubes' as before but not in the sense that they have faces etc, just abstract cubes. then I generated neighbour information for them. Finally pass them a vector of vertices and just ask them to add their own vertices to the list. They can then check if they have a neighbour in a direction, if so they just don't add vertices for that side. I will take a look at that greedy mesh method too.

The primary drawback with that method, is that while the number of vertices drawn goes down, the number of fragments/pixels processed goes up (assuming you do hidden-face elimination which is the most important optimization here!)

Reducing to boxes never decreases number of fragments drawn. In practically all situations, it will drastically increase their amount.

Doing this would only make sense if doing this vertices-for-fragments trade results in net performance increase.

One possible scenario would be far-away chunks - where each voxel takes just few pixels on screen, shading is cheap because you dont even need textures or proper lighting at extreme distances, and further you dont need to be able to easily edit those chunks because theyre too far for such things to matter.

So, the disadvantages in a normal voxel grid scenario with high fragments-to-vertices ratio:

-Overall performance significantly decreases because theres vastly more fragments to shade (even with early depth eliminations and whatnot)

(the problem is hidden face elimination. Reducing to boxes is a limited form of hidden face elimination. Its better than the naive "just render many boxes" approach, but much less effective than complete per-voxel hidden surface elimination.)

-Cost of modifying object increases (need to recalculate the 'optimized' boxes)

-No flexibility with per-voxel/per-vertex changes (like how minecraft renders its lighting with some per-face shadow texture things IIRC). Also you cant make nonlinear transformations on the voxels because this will detach box faces and vertices from each other spatially (with normal voxels there are only vertex-vertex connections as you say)

The first one (that its a highly less effective form of hidden surface removal) is the most relevant here (discussing performance).

The conclusion would be that AS LONG AS you do not 'break' normal hidden surface elimination, this kind style of optimization helps (still the other disadvantages apply):

-you can apply it to the 2D surfaces surviving hidden surface elimination only, instead of 3D voxel regions (like you seem to have decided to do)

-you can apply it to 3D voxel regions, if most of the resulting box is visible (not hidden)

And if you DO decide to still use it, heres a performance tip for making it practical:

Perform hidden surface removal on the boxes! Only replace a voxel region with a box, when some faces of the box can be removed such that most of the resulting box surface is visible (not hidden).

Like you could split your pyramid such that each layer is formed by 4 thin and long boxes, with the 'inside' faces removed/hidden. That would be very close to an optimal mesh.