I've implemented instance culling using instance cloud reduction described here:
This technique works by storing the instance positions in a VBO, then using a vertex shader to cull them (basically render them), then using the transform feedback mechanism and a geometry shader only store those in an output vbo that passed the visibility testing. Now this way the geometry shader outputs GL_POINTS (vec4). This way only position is considered.
Now in my app I'd like to use rotation and scaling as well. As far as I know, this can be best described by a matrix. My question is: how can I output these matrices with the geometry shader? Should I output more than one vertices (more EmitVertex();)? How do they get stored in the output vbo?
Plus how would the culling be done? should I just transform the bounding box by this matrix? then check that?
Edited by Yours3!f, 04 June 2013 - 05:31 AM.