Sign in to follow this  
adt7

HLSL frac() Issues

Recommended Posts

Yet another HLSL issue for me. Why are these things never simple? Anyway, it seems that whenever I use frac() something goes horribly wrong. I tried using it in my code and it wasn't giving the expected results, so I tried using it on my TexCoords to check I was doing it right and I got very strange results. Pos.xy has the range [0... 256], OneOverTexSize is 256. Normally I calculate my TexCoords as follows:
Pos.xy * OneOverTexSize

Which gives values in the range [0... 1] obviously. To test that frac() worked as expected I changed this to:
frac(Pos.xy * OneOverTexSize)

Surely this should also give values in the range [0... 1]? However, instead of sampling my texture correctly I just get a solid colour (as if it's sampling one point all over the texture). Am I misunderstanding what frac() does here? I'm using SM3.0 in case it matters.

Share this post


Link to post
Share on other sites
Are you certain that Pos.xy is varying over the surface of the screen? Try simply outputting the Pos.xy as a color to make sure.

You could also just try calling frac( float4(0.5,1.5,2.5,3.5) ) and you should get gray out everywhere. I have used frac quite a few times, and it appears to work as advertised.

If those things don't shed light on the topic, try running with the reference rasterizer to see if you get the same result. If you are, then it is your program that is doing something incorrectly!

Share this post


Link to post
Share on other sites
I worked out the problem... and feel stupid now that I've realised what was wrong.

Previously I was using multiple quads to cover my area [0... 256], but to reduce draw calls I've moved to just using 1. Which of course means that the only values I'm passing to my vertex shader are 0 and 256, which with frac() both come out as 0.

I'll move the code using frac() into the pixel shader tonight and all should be fixed. I was trying to improve efficiency by keeping it in the vertex shader, but I'm not GPU bound yet and I'm sure I have better places to optomise.

Just to check, the expected result of frac(-0.75) is 0.25? (As with -1.75, -2.75...?)

[Edited by - adt7 on May 29, 2009 4:35:02 AM]

Share this post


Link to post
Share on other sites
See, that is what I thought, but comparing float4(0.25, 0, 0, 1) and (float4(frac(-0.75), 0, 0, 1) as a pixel shader output they appear the same, whilst float4(0.75, 0, 0, 1) appears to be noticably brighter.

I'll use the reference device to make sure it's not my hardware when I get time.

Edit: I've just thought about this (I should do this more often) and frac(x) is defined as follows:
frac(x) = x - floor(x)


Therefore, for x = -0.75:
floor(x) = -1
frac(x) = -0.75 - (-1)
frac(x) = 0.25



Just thought I would post that for other people's reference.

Thanks for the help Jason.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this