# HLSL frac() Issues

This topic is 3429 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Yet another HLSL issue for me. Why are these things never simple? Anyway, it seems that whenever I use frac() something goes horribly wrong. I tried using it in my code and it wasn't giving the expected results, so I tried using it on my TexCoords to check I was doing it right and I got very strange results. Pos.xy has the range [0... 256], OneOverTexSize is 256. Normally I calculate my TexCoords as follows:
Pos.xy * OneOverTexSize


Which gives values in the range [0... 1] obviously. To test that frac() worked as expected I changed this to:
frac(Pos.xy * OneOverTexSize)


Surely this should also give values in the range [0... 1]? However, instead of sampling my texture correctly I just get a solid colour (as if it's sampling one point all over the texture). Am I misunderstanding what frac() does here? I'm using SM3.0 in case it matters.

##### Share on other sites
Are you certain that Pos.xy is varying over the surface of the screen? Try simply outputting the Pos.xy as a color to make sure.

You could also just try calling frac( float4(0.5,1.5,2.5,3.5) ) and you should get gray out everywhere. I have used frac quite a few times, and it appears to work as advertised.

If those things don't shed light on the topic, try running with the reference rasterizer to see if you get the same result. If you are, then it is your program that is doing something incorrectly!

##### Share on other sites
I worked out the problem... and feel stupid now that I've realised what was wrong.

Previously I was using multiple quads to cover my area [0... 256], but to reduce draw calls I've moved to just using 1. Which of course means that the only values I'm passing to my vertex shader are 0 and 256, which with frac() both come out as 0.

I'll move the code using frac() into the pixel shader tonight and all should be fixed. I was trying to improve efficiency by keeping it in the vertex shader, but I'm not GPU bound yet and I'm sure I have better places to optomise.

Just to check, the expected result of frac(-0.75) is 0.25? (As with -1.75, -2.75...?)

[Edited by - adt7 on May 29, 2009 4:35:02 AM]

##### Share on other sites
AFAIK, the result of frac(-0.75) should be 0.75. I think it just removes the sign and integer portion. I would try it out to make sure though...

##### Share on other sites
See, that is what I thought, but comparing float4(0.25, 0, 0, 1) and (float4(frac(-0.75), 0, 0, 1) as a pixel shader output they appear the same, whilst float4(0.75, 0, 0, 1) appears to be noticably brighter.

I'll use the reference device to make sure it's not my hardware when I get time.

Edit: I've just thought about this (I should do this more often) and frac(x) is defined as follows:
frac(x) = x - floor(x)

Therefore, for x = -0.75:
floor(x) = -1frac(x) = -0.75 - (-1)frac(x) = 0.25

Just thought I would post that for other people's reference.

Thanks for the help Jason.

1. 1
2. 2
Rutin
25
3. 3
4. 4
5. 5

• 10
• 10
• 13
• 20
• 14
• ### Forum Statistics

• Total Topics
632946
• Total Posts
3009363
• ### Who's Online (See full list)

There are no registered users currently online

×