I simply want to add a check before a triangle gets transformed, so that certain triangles never hit the vertex shader.

If I'm understanding your intentions correctly, the geometry shader comes after the vertex shader - not before it - so you can't use it to cull triangles based on some arbitrary check.

As mark ds says, the geometry shader runs after the vertex shader.

If I understand correctly you want to do some checking before running them through an expensive vertex shader, am I right?

I think you'd have to use a trivial vertex shader, use your geometry shader to check which triangles should be culled, and store the triangles that pass the test in the transform feedback buffer. You can then use that buffer as input to your expensive vertex shader and run the rest of the pipeline.

Being that the GS comes after the VS shader and takes primitives as input, it can be used for culling in a general sense. In your use case, you want to cull primitives before they are transformed, which like many already pointed out, a geometry shader won't work for that purpose, unless you are doing the actual transformation in the geometry shader. However, I don't think this approach will give you whatever performance increase you are looking for. You will need to profile you application to see where the bottleneck is before attempting any form of optimization.

It might be sane, but the problem is as stated there is no stage before the VS that you could do this in; at best you could do a pass through-VS and do the work in the GS but that'll hurt as you'll end up spilling out to gfx memory in and introduce ordering.

Amusingly the whole VS->[tess]->GS thing is just an artifact of the way the pipeline was setup; current hardware could, quite happily, put a 'GS'-like stage first to operate on a primitive (IA is done in ALU cores these days anyway) before the VS/Tess stages... in fact VS wouldn't really be needed at all in that situation as you could test and transform 3 points just fine.

In short; the idea might have merit but the way the software pipeline I arranged means that while the hardware could operate in that manner you aren't going to be able to do it.