What about a neural network for this job, could be more adapted than GA ? I''m not sure about the outputs that it should produce..It looks like one would need need an extra large hidden layer and tons of inputs.

I''m not an expert in 3D rendering techniques, but I am fairly experienced with GA''s so I''ll put in my .02 here.

If I understand correctly, efficient culling basically boils down to being able to split your scene up into groupings of triangles that you will use so you can cull entire groups at once instead of testing each individual triangle. This is then extended to support a hierarchy of triangle groupings (ie. quadtrees, octrees, etc) [not really sure what an ABT is, but I''m guessing its some kind of alternative to quadtrees].

You could definately use a GA to find an efficient "non-regular" grouping of triangles that produces efficient culling. The way I would do it is first create the geometry for your scene. Then create a playable version of the game that either uses per-triangle culling and will only run on very beefy computers, or use lower-quality geometry to create this playable version. Have a bunch of people play-test this version, and have code that will log the camera position and orientation for every frame. Once you have enough of these logs you can create a weighted map of your scene that determines what camera locations occur most often and least often. You can then design a GA that will generate a hierarchical grouping of the scenes geometry and use this weighted map of camera locations to choose say 100+ viewpoints to use to render frames from and recording how efficient this specific triangle grouping is (this will be your fitness function).

Yes, this will take a very long time to run. You will probably need to set it up on a top of the line PC and let it churn away for a month or so. If you develop it in a distributed manner (which this specific GA lends itself to very naturally) you can use the idle time of all the computers in your company to do the computations. I would expect a properly designed GA to be able to generate a triangle grouping that is drastically better than any generic approach such as quadtrees or octrees.

Of course you could always develop your fitness function so that it rewards framerate instead of culling efficiency. That would alleviate your concerns about that. But now I am curious as to how much improvement a GA generated "triangle tree" (not sure what the appropriate term is) will have versus an optimized traditional method. Sounds like a good masters project.