whisp

I figured out a possible solution for the problem.

The data passed from a vertex program to a fragment program always gets interpolated. It's not possible to pass custom values that don't get interpolated, thus the interpolation needs to be removed in the fragment shader.

In this case i need the non-interpolated model-space-coordinates of the vertices in order to look up the tile-map texture.

In the vertex shader i calculated the texture coordinates:
// Texture Coordinates, 1/8 of the tileset
  OUT.texCoord  = OUT.pageVertXZ.xy * 0.125;

With this coordinates the vertex position can be calculated in the fragment shader:
// set texCoord relative to 0,0
   float2 pageVertXZ = floor(indata.texCoord * 8);

The interpolation-value just gets cropped, what's left is the vertex coordinate in model space.