Jump to content
  • Advertisement
Sign in to follow this  
ehmdjii

HLSL weird matrix behaviour

This topic is 2748 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

hello,

i have some problem using an int 3x3 matrix as the convolution filter to do edge detection. (i know int matrices are not very meaningful since there is only float on the gpu)

here is the code:



float2 stepSizeX = (1.0/512.0, 0.0);
float2 stepSizeY = (0.0, 1.0/512.0);


for(int y = -1; y <= 1; y++)
{
for(int x = -1; x <= 1; x++)
{
float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);
endColorX.rgb += c.rgb * theMatrix[x+1][y+1];
}
}


the matrix is set to
-1 -2 -1
0 0 0
1 2 1
in the application, but it seems the values in the HLSL shader are different, though i cant figure out how and why?

thanks a lot!

Share this post


Link to post
Share on other sites
Advertisement
You should probably post the code that declares the [font="'Courier New"]theMatrix[/font] variable, and the DX code that sets it on the application side. Also can you describe what 'it seems the values are different' means.

And yeah, you should probably just use a float matrix smile.gif

Share this post


Link to post
Share on other sites

You should probably post the code that declares the [font="Courier New"]theMatrix[/font] variable, and the DX code that sets it on the application side. Also can you describe what 'it seems the values are different' means.


thanks for your reply! well actually the whole setup is done with rendermonkey., so i cant post the DX code, however let's assume it is correct.

i tried to test the values with a function like this and it returns false


bool eq(float a, float b)
{
return abs(a-b) < 0.1;
}


also i am using ps2.0

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!