I have been trying to implement rendering of the BMFont format in DirectX 11 based upon the sample code and I am running into issues once I started to use the channel information. Below I have two images, one where I do not use the channel information in the pixel shader and one where I do. The test being displayed is "testna" using the Comic24 binary file from the example code. The text at the top is with alpha blending off, the second is with it on.

Before using the channel information

With using the channel information

When I use the do not use the channel information the pixel shader is just

float4 PS( VS_OUTPUT input) : SV_Target {
    return textureDiffuse0.Sample(textureSampler0, input.Tex);
}

When I use the channel information I do it the same way the example code does (except for using color)

float4 PS( VS_OUTPUT input) : SV_Target {
	float4 pixel = textureDiffuse0.Sample(textureSampler0, input.Tex);

    // Are we rendering a colored image, or 
    // a character from only one of the channels
    if( dot(vector(1,1,1,1), input.channel) )
    {
        // Get the pixel value
		float val = dot(pixel, input.channel);
		
        pixel.rgb = 1;
        pixel.a   = val;
    }
    return pixel;
}

Does anyone know why when I use the channel information that this happens? I inspected the channel information while debugging and it is valid. I also tried to hard code it in the shader with the value that I see that it typically is (256) and I get the same result

I have my vertex data defined as

struct Vertex {
    float x, y, z, u, v;
    DWORD channel;
};
Vertex m_VertexData[400];

?

?

?

?

?

?

Then I assign the chanel with

m_VertexData[id].channel = ch->chnl

In the vertex shader it is

struct VS_INPUT {
    float4 Pos : POSITION;
    float2 Tex : TEXCOORD0;
	uint4 channel : BLENDINDICES0;
};
struct VS_OUTPUT {
    float4 Pos : SV_POSITION;
    float2 Tex : TEXCOORD0;
	uint4 channel : BLENDINDICES0;
};
VS_OUTPUT VS( VS_INPUT input ) {
    VS_OUTPUT output = (VS_OUTPUT)0;
    output.Pos = mul( float4( input.Pos.xyz, 1.0f), WorldViewProjMatrix);
    output.Tex = input.Tex;
    output.channel = input.channel;
	
    return output;
}

All of that seems to be exactly how you have it, along with the following vertex format

D3D11_INPUT_ELEMENT_DESC layout[] =
{
	{ "POSITION", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, 0, D3D11_INPUT_PER_VERTEX_DATA, 0 },
	{ "TEXCOORD", 0, DXGI_FORMAT_R32G32_FLOAT, 0, 12, D3D11_INPUT_PER_VERTEX_DATA, 0 },
	{ "BLENDINDICES", 0, DXGI_FORMAT_R8G8B8A8_UINT, 0, 24, D3D11_INPUT_PER_VERTEX_DATA, 0 },
};

I recommend changing 'uint4 channel' to 'float4 channel' and DXGI_FORMAT_R8G8B8A8_UINT to DXGI_FORMAT_R8G8B8A8_UNORM. That way you'll get values in range [0;1] and dot() should return some meaningful values. (at least AA will work) As for wrong selection, like Andreas said, check what are the real values in input.channel (just move rbg directly to pixel.rbg)

In the pixel and vertex shader the value will be incorrect it seems. It will end up being "input.channel x = 0, y = 128, z = 196, w = 195 uint4". When I hard code it in the shader to be "input.channel = uint4(0, 1, 0, 0)", then it will be correct. The texture is being read as RGBA as DirectX11 does not seem to have an ARGB format. Which I suspect may also be leading to my issue where when I use alpha testing all my character are showing up as boxes.

I suppose something that I do not understand is how are you setting the channel to a DWORD in VtxData and VtxPos, but then the shader knows it is a int4(0, 1, 0, 0)? How does it end up taking the DWORD value of 256 and then converting it into that? Perhaps there is something else I am missing, but I stepped though the code and looked at the values in PIX to see that

I figured out what the issue was. In the BLENDINDICIES D3D11_INPUT_ELEMENT_DESC format I was saying they should start at 24, instead of 20. I look at it dozens of times and it finally stood out for me.

Thank you for the help