Jump to content
  • Advertisement
Sign in to follow this  
adt7

HLSL frac() Issues

This topic is 3364 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Yet another HLSL issue for me. Why are these things never simple? Anyway, it seems that whenever I use frac() something goes horribly wrong. I tried using it in my code and it wasn't giving the expected results, so I tried using it on my TexCoords to check I was doing it right and I got very strange results. Pos.xy has the range [0... 256], OneOverTexSize is 256. Normally I calculate my TexCoords as follows:
Pos.xy * OneOverTexSize

Which gives values in the range [0... 1] obviously. To test that frac() worked as expected I changed this to:
frac(Pos.xy * OneOverTexSize)

Surely this should also give values in the range [0... 1]? However, instead of sampling my texture correctly I just get a solid colour (as if it's sampling one point all over the texture). Am I misunderstanding what frac() does here? I'm using SM3.0 in case it matters.

Share this post


Link to post
Share on other sites
Advertisement
Are you certain that Pos.xy is varying over the surface of the screen? Try simply outputting the Pos.xy as a color to make sure.

You could also just try calling frac( float4(0.5,1.5,2.5,3.5) ) and you should get gray out everywhere. I have used frac quite a few times, and it appears to work as advertised.

If those things don't shed light on the topic, try running with the reference rasterizer to see if you get the same result. If you are, then it is your program that is doing something incorrectly!

Share this post


Link to post
Share on other sites
I worked out the problem... and feel stupid now that I've realised what was wrong.

Previously I was using multiple quads to cover my area [0... 256], but to reduce draw calls I've moved to just using 1. Which of course means that the only values I'm passing to my vertex shader are 0 and 256, which with frac() both come out as 0.

I'll move the code using frac() into the pixel shader tonight and all should be fixed. I was trying to improve efficiency by keeping it in the vertex shader, but I'm not GPU bound yet and I'm sure I have better places to optomise.

Just to check, the expected result of frac(-0.75) is 0.25? (As with -1.75, -2.75...?)

[Edited by - adt7 on May 29, 2009 4:35:02 AM]

Share this post


Link to post
Share on other sites
AFAIK, the result of frac(-0.75) should be 0.75. I think it just removes the sign and integer portion. I would try it out to make sure though...

Share this post


Link to post
Share on other sites
See, that is what I thought, but comparing float4(0.25, 0, 0, 1) and (float4(frac(-0.75), 0, 0, 1) as a pixel shader output they appear the same, whilst float4(0.75, 0, 0, 1) appears to be noticably brighter.

I'll use the reference device to make sure it's not my hardware when I get time.

Edit: I've just thought about this (I should do this more often) and frac(x) is defined as follows:
frac(x) = x - floor(x)


Therefore, for x = -0.75:
floor(x) = -1
frac(x) = -0.75 - (-1)
frac(x) = 0.25



Just thought I would post that for other people's reference.

Thanks for the help Jason.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!