@Tim: I actually do something similar with the clipping to viewport bounds and stuff but I was trying to implement actual vertex clipping if a vertex is outside the guard bands.
Anyhow I've managed to get it working (at least flawlessly with a single triangle ).
For those who might wanna know and/or found this years after , this is how I do it :
I'm now translating my post transform coordinates to basically get [-1,1] coordinates and then check these with the guard band boundaries.
Then I count how many vertices are outside the guard band. If there's two vertices outside then it's an easy case where I just use the cohen-sutherland algorithm to find the two intersection points with the guard band boundary and just change the original position with the intersection point.
If there's just one vertex outside it gets a little trickier, because you now have to basically generate an additional triangle and "reconnect" the original vertices with the new ones from the intersections.