I've been working on building a custom GUI system for the past few months. The engine I'm using does not have a very good solution for rendering scalable text so I decided to write my own based on the technique described by Valve Software in the white paper found here. I was able to successfully implement their technique for rendering scalable fonts. My GUI text object now has functionality for displaying, aligning, and wrapping text but I'm completely stuck on the last piece of functionality I want to add: Clipping text glyphs that are outside a set clip boundary. I've been struggling with this for several days now so I decided to post here to see if anyone with any experience in this type of thing could give me some advice to push me in the right direction.

The 'Text' class works by combining glyphs (characters). Each 'Glyph' contains vertex data and texture coordinates that correspond to a specific character. The texture coordinates specify the location around the character on a texture atlas (I.E. The texture for each glyph is packed into a single texture atlas). Each Glyph also has fields for specifying an amount to 'clip' the glyph. The idea being: when the glyph is positioned partially outside of the clip bounds I can set the clip amount to render only the part of the glyph that is within the clip boundaries. The code that each glyph uses to calculate vertex and texture coordinates is below:

    public final List<Vector3f> getVertices(final float xPosition, final float yPosition, final float scale) {
        List<Vector3f> verts = new ArrayList<Vector3f>();
        verts.add(new Vector3f((xOffset * scale) + xPosition + vertClipLeft, (font.getBaseLine() - yOffset) * scale + yPosition - vertClipTop, 0f)); //Top Left Vertex
        verts.add(new Vector3f((xOffset * scale) + xPosition + vertClipLeft, (font.getBaseLine() - height - yOffset) * scale + yPosition + vertClipBottom, 0f)); //Bottom Left Vertex
        verts.add(new Vector3f((width + xOffset) * scale + xPosition - vertClipRight, (font.getBaseLine() - height - yOffset) * scale + yPosition + vertClipBottom, 0f)); //Bottom Right Vertex
        verts.add(new Vector3f((width + xOffset) * scale + xPosition - vertClipRight, (font.getBaseLine() - yOffset) * scale + yPosition - vertClipTop, 0f)); //Top Right Vertex
        return verts;
    }
   
    public final List<Vector2f> getCoordinates() {
        float pixWidth = 1f / atlasTextureWidth;
        float pixHeight = 1f / atlasTextureHeight;
        
        List<Vector2f> coords = new ArrayList<Vector2f>();
        coords.add(new Vector2f((xLoc + coordClipLeft) * pixWidth, ((atlasTextureHeight - yLoc)- coordClipTop) * pixHeight)); //Top Left Coordinate
        coords.add(new Vector2f((xLoc + coordClipLeft) * pixWidth, ((atlasTextureHeight - yLoc) - height + coordClipBottom) * pixHeight)); //Bottom Left Coordinate
        coords.add(new Vector2f((xLoc + width - coordClipRight) * pixWidth, ((atlasTextureHeight - yLoc) - height + coordClipBottom) * pixHeight)); //Bottom Right Coordinate
        coords.add(new Vector2f((xLoc + width - coordClipRight) * pixWidth, ((atlasTextureHeight - yLoc) - coordClipTop) * pixHeight)); //Top Right Coordinate
        
        return coords;
    }

In the main 'Text' class I use the following code to determine if a Glyph needs to be clipped and if so, calculate how much the Glyph needs to be clipped. (This is for clipping a glyph when it is outside the North (top) clip boundary. I choose top because the problem is more apparent on the north and south boundaries than it is on the east and west.The code for clipping Left, Right, and Bottom is very similar to this but they have been omitted because: redundancy. If I can get one side to work I can get the others to work with minimal effort.

        if (yPosition + yTextOffset + (font.getBaseLine() * size - glyph.getYOffset() * size) > clipHeight && vClip) {
            float amountToClip = (yPosition + yTextOffset + (font.getBaseLine() * size - glyph.getYOffset() * size)) - clipHeight;
            glyph.setVerticalVertexClipping(amountToClip, 0f);
            glyph.setVerticalCoordinateClipping(amountToClip, 0f);
        }

The end result is that the vertices clip properly but the coordinate clipping only works when the character is the exact same size as it is on the texture atlas (in this case, the texture was rendered at a font size of 72pt). I'm almost certain that the solution will involve calculating the coordinate clipping amount separate from the vertex clipping amount, but I cannot, for the life of me, figure out how to calculate the clipped coordinates so that the characters don't look as if they are being compressed (I.E. scaled down) at the clip bounds. See the below screenshot for a visual example of the problem:

First, the unclipped text. Rendered at 72pt size. The gray box around the text represents the clip boundaries.

When I move the Y-Offset of the text up so that the characters exceed the boundaries it works perfectly with 72pt size text.

Unfortunately, when I try the same thing using a smaller text size: some of the clipped glyphs become compressed at the clip boundary. Notice how the bridge of the lower-case 'h' is no longer even with the rest of the line. The 'i', 'b', 'j', and 'd' characters are also deformed. (The deformity is subtle and may be difficult to notice at first, but it is certainly there).

I've been trying for several days to fix this issue but haven't had any luck. If someone with more experience in this kind of thing (and with better math skills than me) could point out where I am miscalculating I would be eternally grateful. I'm almost certain the problem has to do with the coordinate clippings. Thanks in advance for any assistance!

I'm not sure how you're rendering your glyphs, but would it be feasible for you to achieve clipping at the hardware level by using the stencil buffer?

I'm not sure how you're rendering your glyphs, but would it be feasible for you to achieve clipping at the hardware level by using the stencil buffer?

The engine handles most of the OpenGL stuff (I just wrote the shader). Each glyph is basically a textured quad - the coordinates are mapped to the location of each individual character within the texture atlas. In the end, the entire text is rendered as a single object with a single material (with 4 vertices and coordinates per character). I have started looking into using a stencil buffer (from your suggestion). The engine I use doesn't have any functionality for setting the stencil buffer so I would have to write it in myself which may not be feasible with my current technical abilities unless it's something that I can do inside the shader.

The engine handles most of the OpenGL stuff (I just wrote the shader). Each glyph is basically a textured quad - the coordinates are mapped to the location of each individual character within the texture atlas. In the end, the entire text is rendered as a single object with a single material (with 4 vertices and coordinates per character). I have started looking into using a stencil buffer (from your suggestion). The engine I use doesn't have any functionality for setting the stencil buffer so I would have to write it in myself which may not be feasible with my current technical abilities unless it's something that I can do inside the shader.

According to this tutorial, working with the stencil buffer is fairly straightforward (assuming Java has the same API). However you do need to manipulate global OpenGL state for it to work, so it's not something you could contain to just shader code.

The engine handles most of the OpenGL stuff (I just wrote the shader). Each glyph is basically a textured quad - the coordinates are mapped to the location of each individual character within the texture atlas. In the end, the entire text is rendered as a single object with a single material (with 4 vertices and coordinates per character). I have started looking into using a stencil buffer (from your suggestion). The engine I use doesn't have any functionality for setting the stencil buffer so I would have to write it in myself which may not be feasible with my current technical abilities unless it's something that I can do inside the shader.

According to this tutorial, working with the stencil buffer is fairly straightforward (assuming Java has the same API). However you do need to manipulate global OpenGL state for it to work, so it's not something you could contain to just shader code.

Thank you, the information you've provided and that link have been very helpful. I haven't much experience with the global OpenGL state but now is as good a time to learn as any. I'm going to try this and see if I can make it work as I feel like I've tried everything else.

Resolved, if anyone else runs into a similar problem:

I was able to discover a very simple solution. I abandoned my previous method of working with the coordinates directly and instead starting looking for a solution in the fragment shader. I was able to add code to discard any fragment that falls outside of the clip boundaries (which are now being set as a vec4 material parameter that gets passed to the shader, not as fields inside the text object as I was doing before). This was extremely easy to do in the fragment shader, I don't know why I didn't think of it before.

Here is a sample of the code I used in the vertex and fragment shaders:

font.vert (Vertex Shader)

    #ifdef HAS_CLIP_BOUNDS
        position = g_WorldViewMatrix * vec4(inPosition, 1.0);
    #endif

font.frag (Fragment Shader)

    //Discard fragment if its position is outside of the clip boundaries.
    #ifdef HAS_CLIP_BOUNDS
        if (position.x < m_Clipping.x || position.x > m_Clipping.z || position.y < m_Clipping.y || position.y > m_Clipping.w) {
            discard;
        }
    #endif

m_Clipping is a uniform vec4 (and is passed to the shader from the material)

position is a varying vec4 and is the position of the fragment.