Jump to content
  • Advertisement
Sign in to follow this  
Juliean

HLSL color output bug

This topic is 2051 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Oups, hit the post button when meaning to close this. Well, I'm too damn tired, correcting this then going to bed. So I'm trying to convert my old SSAO shader to the new engine, stepping from fx files to seperate vertex and pixel shader. Now the result isn't as expected, so I'm trying to ouput different stages of the pixel shader, for example if the textures are even sampled correctly. Now, look at this shader:

 

float2 cScreenSize : register(c0);
float4 cParams : register(c1); // scale, bias, intensity, sample rad
float4x4 cView : register(c2);
texture2D cNormal : register(s0);
texture2D cPosition : register(s1);
texture2D cNoise : register(s2);

sampler cNormalSampler = sampler_state
{
	Texture = <cNormal>;
};

sampler cPositionSampler = sampler_state
{
	Texture = <cPosition>;
};

sampler cNoiseSampler = sampler_state
{
	Texture = <cNoise>;
};

struct PS_INPUT
{
	float4 vPos             : POSITION0;
	float2 vTex0            : TEXCOORD0;
};

float3 getNormal(in float2 uv)
{
	return normalize( mul( tex2D(cNormalSampler, uv).xyz, cView) * 2.0f - 1.0f );
}

float2 getRandom(in float2 uv)
{
	const float random_size = 64.0f;
	return normalize(tex2D(cNoiseSampler, cScreenSize * uv / random_size).xy * 2.0f - 1.0f);
}

float doAmbientOcclusion(in float2 tcoord,in float2 uv, in float3 p, in float3 cnorm)
{
	float3 diff = getNormal(tcoord + uv) - p;
	const float3 v = normalize(diff);
	const float d = length(diff)*cParams.x;
	return max(0.0,dot(cnorm,v)- cParams.y)*(1.0/(1.0+d))*cParams.z;
}

float4 mainPS(PS_INPUT i) : COLOR0
{
	const float2 vec[4] = {float2(1,0),float2(-1,0),
				float2(0,1),float2(0,-1)};

	float3 p = mul( tex2D(cPositionSampler, i.vTex0), cView);
	
	float3 n = getNormal(i.vTex0);
	
	float2 rand = getRandom(i.vTex0);

	float ao = 0.0f;
	float rad = cParams.w/  p.z;

	//**SSAO Calculation**//
	int iterations = 4;
	for (int j = 0; j < iterations; ++j)
	{
	  float2 coord1 = reflect(vec[j], rand)*rad;
	  float2 coord2 = float2(coord1.x*0.707 - coord1.y*0.707,
				  coord1.x*0.707 + coord1.y*0.707);
	  
	  ao += doAmbientOcclusion(i.vTex0, coord1*0.25, p, n);
	  ao += doAmbientOcclusion(i.vTex0, coord2*0.5, p, n);
	  ao += doAmbientOcclusion(i.vTex0, coord1*0.75, p, n);
	  ao += doAmbientOcclusion(i.vTex0, coord2, p, n);
	}
	ao/=(float)iterations*4.0;
	//**END**//

	//Do stuff here with your occlusion value “ao”: modulate ambient lighting, write it to a buffer for later //use, etc.
	
	return tex2D(cNoiseSampler, i.vTex0); //now here is the interesting stuff
	return ao;
}

If I return the sample from the noise sampler, it will instead return the normal texture. Same if I sample the position texture. Now, this happens for all of my shaders, even those that are working correctly. So uhm, any ideas why this is happening, how I can solve this? That way I don't know if the whole sampling is broken or just the output :/

Share this post


Link to post
Share on other sites
Advertisement

I had this issue a while ago, what I learned is that you cannot assume how HLSL will arrange your texture registers, and even forcing the order using the "register" directive did not help. Furthemore, debugging made it even worse because HLSL dynamically moves the textures around depending on whether you are using them or not in the shader!! Eventually I had to use shader reflection to work out where which texture was.

 

A real headache indeed.

Share this post


Link to post
Share on other sites
You sure ? If I drop the (effect style) sampler_state stuff completely I can force the layout with the register keyword (using sampler instead of texture2D):

sampler cNormalSampler : register(s0);
sampler cPositionSampler : register(s1);
sampler cNoiseSampler : register(s2);
Assembly output (checked with the command line compiler (fxc /T ps_3_0 /E mainPS /Fx shader.asm shader.fx

// Registers:
//
//   Name          Reg   Size
//   ------------- ----- ----
//   cNoiseSampler s2       1
Or, here (now returning the ao value):

sampler cNormalSampler : register(s0);
sampler cPositionSampler : register(s4);
sampler cNoiseSampler : register(s1);
Result:

// Registers:
//
//   Name             Reg   Size
//   ---------------- ----- ----
//   cScreenSize      c0       1
//   cParams          c1       1
//   cView            c2       4
//   cNormalSampler   s0       1
//   cNoiseSampler    s1       1
//   cPositionSampler s4       1
//

Share this post


Link to post
Share on other sites

You sure ? If I drop the (effect style) sampler_state stuff completely I can force the layout with the register keyword (using sampler instead of texture2D):

 

Ah, yeah, you are right, using you "sampler" instead of sampler_state and texture2D obviously fixes the issue. Thanks, also means less code, so, yippie :D

 

Furthemore, debugging made it even worse because HLSL dynamically moves the textures around depending on whether you are using them or not in the shader!!

 

Yeah, I noticed that one. Kind of a pain in the ass, specifally when looking in Nvidia PerfHUD, since you can't tell whether a texture hasn't been bound correctly or just isn't used for some reason. I think I really gotta learn to better understand the assembly and how it is produced, should make things a bit easier.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!