If I use a filter, anisotropic or tri/bilinear, I only have the pixels of the current triangle to work with in my triangle mapper. If a pixel is near an edge, the filter kernel may go outside the triangle, how to get these pixels outside of the current triangle ? Artifacts do occur if they are not taken into account, so how to treat them ? how is it done in games ? may sound curious to some people, but I'm writing my renderer completely from low level..

Maybe i dont understand your question correctly but anisotropic or tri/bilinear filtering has nothing to do with triangle, it's just the way how neightbour texels taken into account when texel taken from texture is not mapping perfectly to current pixel on screen.

If I understand your question, there are a number of ways you can handle this. Assume you have a sample [5694738] and a kernel (1/9)[12321].

1) You can clamp the edge values so that every pixel past the edge has the same value as the pixel at the edge.
Visually (| is the boundary of the sample):
sample---555|5694738|888
kernel---------------12|321

2) You can reflect the edge values.
965|5694738|837
-----------12|321

3 You can disregard the parts that cross the boundary and renormalize the kernel.
|5694738|
-------12|
(Except now the kernel weight is 1/3 instead of 1/9)

I hope that makes sense.

Ok, it keeps removing spaces that I put in. Disregard the '-'s and line up the '|'s on the right edge of the sample.