Sign in to follow this  
rumble

HLSL % what am I misunderstanding

Recommended Posts

In the following pixel shader, I apply the % operator to two numbers. But often I get unexpected result for large numbers. I.e.

4294967281%4294967285 evaluates to 0 (wrong) and I get black screen.

1%4294967285 I get white screen
11%4294967285 I get black screen
111%4294967285 I get white screen
1111%4294967285 I get black screen
11111%4294967285 I get white screen

I had no such problem for the equivalent .cgfx shader on the same Geforce 8700m and another Geforce 4600.

Is this due to different floating point precision?

Tried ps_2_0 up to ps_4_0.


float4 std_PS(vertexOutput IN) : COLOR {

unsigned int val = 11%4294967285;
return float4(val,val,val,1);
}

Share this post


Link to post
Share on other sites
If you compiles those from the command line and compare the shader asm that is printed to the console, do you see any differences? My guess is that compiler is producing different results for the static calculation.

Share this post


Link to post
Share on other sites
Thanks. I did further testings:

uint test = 2147483647; //this is the largest signed int
float3 c = (float)test;
return float4(c,1);

produces

// Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
ps_3_0
def c0, 2.14748365e+009, 1, 0, 0
mov oC0, c0.xxxy

but

uint test = 2147483648; //line 120
float3 c = (float)test;
return float4(c,1);

produces

error X3000: syntax error: unexpected token ';' // at line 120
compilation failed; no code produced


Both are compiled with fx_2_0.
This seems to tell me uint isn't supported, or the c0 register doesn't accept uint. (It accepts float)

Anyone else have success using uint?

Share this post


Link to post
Share on other sites
Then, under SM3, can I assume uint is the same as int? Is 'unsigned int' meaningful in either SM3 or SM4?

Seems fxc.exe generates no warning when I stuff number larger than range of int in this uint.

My card Quadro 4800 supports SM4. What flag should I pass to fxc.exe? fx_3_0 doesn't work

Thanks

Share this post


Link to post
Share on other sites
You'll have to use sm4.0 to get ints or uints. Any earlier shader model will just run the math as floats if it compiles at all and likely cause the troubles you're having here. fx_4_0 will work fine for you if you have a dx 10 capable card.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this