So,here is my prob. I use the internal product of two vectors to find the angle between them to make frustum culling and normal checking for my geometry. So for example if the normal of a triangle doesn't point at my direction I doesn't draw the triangle. It works but when the angle computed is very close to the check angle artifacts occur(one frame the triangle is drawn and the next it wont), I am almost sure that this happens because the precision of the float is not enough. Is there any way to fix this without using doubles? Thanks.

Using vertex normals for backface culling may not be the best thing to do, as normals are usually interpolated between several triangles, so one normal doesn't tell much about a particular triangle.
Also, the hardware can perform backface culling from the winding direction automatically, just glEnable it, and that's it.

Regarding frustum culling, using the dot product is a bit unusual, but certainly works (although it's not really optimal, since screens are not circular). It may be better to use clipping planes instead, as these better match a rectangular screen.

A simple and obvious solution to get rid of popping triangles is to
a) remove complete triangles only (not individual vertices) if all 3 vertices are outside the frustum
b) make the frustum a little bit larger (this will discard a few less triangles, but will remove the popping)
c) not cull triangles at all, but instead cull entire objects (or hierarchies of objects) when they are completely off-screen

Thanks for the reply.

1)When I am talking about normals I mean the vector that are perpendicular to the specific triangle(without any interpolation). I don't want to do backface culling from hardware I just want the information whether or not one triangle is visible to me to decide if I will split it or not. So any hardware culling is useless for this case.

2)I will try what you proposed(to remove complete triangles only if all 3 vertices are outside the frustum) it may make things better but I dont think it will solve the problem...

The point is that I want to do culling per triangle, I use an algorithm that subdivides a surface so I need to know every frame if a triangle is in my view or out.

I just want a way to get rid of those artifacts.

Well from recent discovery you can not rely on the hardware to cull if you are handling a very large seen.

However as a very general thought for you. You may not want to cull exactly to your view frustum anyway. If you do you are very likely to have the issue you are encountering.

I would suggest making your culling frustum a little large than the view one. This will prevent this issue. You will need to experiment with the difference. If you looked at your view frustum as a cone a very cheap and easy way to do this is to create a culling frustum that was the same size but starts X distance behind the viewpoint. What this does is create a cone that parallels the other cone and gives the slight larger boundary you may need.

My guess is you are working on a some sort of terrain system. Which is why you are wanting to cull triangles verses the objects.

However, if you are going to have a very large area you will need to use some sort of object culling that works with your terrain culling system. Otherwise you will find objects in the distance even when behind one another bleed through it is an issue with the Z-buffer which I found out myself recently.

I thought this, but I dont know if it will work.

Every frame I store the computed cos and the next frame I compute something like (cosNew-CosOld)/CosOld. If this is < e then I keep the previous state of the triangle.

any ideas?

What you should do is maybe store a flag for each triangle which is set to TRUE when the triangle should be culled, otherwise FALSE. Start with all triangles set to TRUE. Then you need an upper and lower culling bounds (lets call them UB for upper bound and LB for lower bound) which are based on the value you are using now (ie you want them to be set like this: LB < current cull value < UB).
The idea is that during each frame, the dot of a triangle which is flaged TRUE is evaluated against LB and a triangle which is flaged FALSE is evaluated against UB.

The loop should look more or less like this:

for each t in triangles do
dot = dot(t.normal, cam.normal)

if t.cullFlag == TRUE && dot < LB then
t.cullFlag = FALSE
else if t.cullFlag == FALSE && dot > UB then
t.cullFlag = TRUE
end if

if t.cullFlag == TRUE then
// Do something with the triangle
else
// Do nothing with the triangle.
end if
end for

You will have to find the exact values of LB and UB.
Hope this helps
Doron