Sign in to follow this  
ehmdjii

HLSL weird matrix behaviour

Recommended Posts

ehmdjii    238
hello,

i have some problem using an int 3x3 matrix as the convolution filter to do edge detection. (i know int matrices are not very meaningful since there is only float on the gpu)

here is the code:


[code]
float2 stepSizeX = (1.0/512.0, 0.0);
float2 stepSizeY = (0.0, 1.0/512.0);


for(int y = -1; y <= 1; y++)
{
for(int x = -1; x <= 1; x++)
{
float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);
endColorX.rgb += c.rgb * theMatrix[x+1][y+1];
}
}[/code]

the matrix is set to
-1 -2 -1
0 0 0
1 2 1
in the application, but it seems the values in the HLSL shader are different, though i cant figure out how and why?

thanks a lot!

Share this post


Link to post
Share on other sites
Hodgman    51324
You should probably post the code that declares the [font="'Courier New"]theMatrix[/font] variable, and the DX code that sets it on the application side. Also can you describe what '[i]it seems the values are different[/i]' means.

And yeah, you should probably just use a float matrix [img]http://public.gamedev.net/public/style_emoticons/default/smile.gif[/img]

Share this post


Link to post
Share on other sites
ehmdjii    238
[quote name='Hodgman' timestamp='1302355737' post='4796325']
You should probably post the code that declares the [font="Courier New"]theMatrix[/font] variable, and the DX code that sets it on the application side. Also can you describe what '[i]it seems the values are different[/i]' means.
[/quote]

thanks for your reply! well actually the whole setup is done with rendermonkey., so i cant post the DX code, however let's assume it is correct.

i tried to test the values with a function like this and it returns false


[code]bool eq(float a, float b)
{
return abs(a-b) < 0.1;
}[/code]

also i am using ps2.0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this