Yet another HLSL issue for me. Why are these things never simple? Anyway, it seems that whenever I use frac() something goes horribly wrong. I tried using it in my code and it wasn't giving the expected results, so I tried using it on my TexCoords to check I was doing it right and I got very strange results. Pos.xy has the range [0... 256], OneOverTexSize is 256. Normally I calculate my TexCoords as follows: Which gives values in the range [0... 1] obviously. To test that frac() worked as expected I changed this to: Surely this should also give values in the range [0... 1]? However, instead of sampling my texture correctly I just get a solid colour (as if it's sampling one point all over the texture). Am I misunderstanding what frac() does here? I'm using SM3.0 in case it matters.

I worked out the problem... and feel stupid now that I've realised what was wrong.

Previously I was using multiple quads to cover my area [0... 256], but to reduce draw calls I've moved to just using 1. Which of course means that the only values I'm passing to my vertex shader are 0 and 256, which with frac() both come out as 0.

I'll move the code using frac() into the pixel shader tonight and all should be fixed. I was trying to improve efficiency by keeping it in the vertex shader, but I'm not GPU bound yet and I'm sure I have better places to optomise.

Just to check, the expected result of frac(-0.75) is 0.25? (As with -1.75, -2.75...?)

[Edited by - adt7 on May 29, 2009 4:35:02 AM]

See, that is what I thought, but comparing float4(0.25, 0, 0, 1) and (float4(frac(-0.75), 0, 0, 1) as a pixel shader output they appear the same, whilst float4(0.75, 0, 0, 1) appears to be noticably brighter.

I'll use the reference device to make sure it's not my hardware when I get time.

Edit: I've just thought about this (I should do this more often) and frac(x) is defined as follows:

Therefore, for x = -0.75:


Just thought I would post that for other people's reference.

Thanks for the help Jason.