Direct3D weird error (heap corruption when loading pixel shader)

Started by
8 comments, last by Adam_42 8 years, 4 months ago

Hi,

when I initialize Direct3D 11 and call CreatePixelShader() for my post processing light pixel shader and then Sleep(20000), I get (Game).exe triggered a breakpoint and something like Critical error detected c0000374 (within these 20 seconds). The call stack only shows external code and there are a few threads running, but none of them is started by me and the main thread is sleeping. The callstack begins in igd10ium64.dll and ends in ntdll.dll. The heap cannot be corrupted before the call to CreatePixelShader(), I commented out every piece of code I don't need to create the pixel shader, so there is only the Direct3D initialization and the loading of the compiled shader before the call to CreatePixelShader(). The error occurs after creating said pixel shader - when I Sleep(20000) before this call, I get the error after 20 seconds. The pixel shader loading function successfully loads other pixel shaders before, so this function is working correct I think. (I commented out the other shader loads but the error still occurs.)

When I don't Sleep() at all, I get some bad_alloc exceptions when using std::wstring, so I think something corrupts my heap.

The same error occurs in release and in debug build.

This error occurs only on my Dell laptop (Windows 8.1), not on my desktop computer (Windows 7) and it doesn't occur when I start the game using Ctrl+F5 (start without debugger) or when I start the (Game).exe directly, but when I do so, I get a "(Game).exe has stopped working" error when entering the world and I would like to debug this error, but I can't because I don't get this far with the debugger.

I don't have any idea how to solve this, I don't know if it is even possible to solve it because it looks like a driver error. but I don't know.

My post processing light PS is "very big" (50 kb compiled), but this should be no problem I think?

Below is the PS code, which compiles without errors for shader model 4.0:


#define MAX_NUM_DIR_LIGHTS 4
#define MAX_NUM_TEXTURES_PER_LIGHT 8

#define MAX_NUM_SPOTLIGHTS_WITH_SHADOW 8
#define MAX_NUM_SPOTLIGHTS 32

#define MAX_NUM_POINT_LIGHTS_WITH_SHADOW 8
#define MAX_NUM_POINT_LIGHTS 32

Texture2D colorTexture : register(t0);
Texture2D normalTexture : register(t1);
Texture2D positionTexture : register(t2);

Texture2DArray<float> pointLightTexture : register(t3);
Texture2DArray<float> spotLightTexture : register(t4);

Texture2D<uint> LightLevelTexture : register(t5);

Texture2D<float> DirectionalLightTextures[MAX_NUM_DIR_LIGHTS*MAX_NUM_TEXTURES_PER_LIGHT];


static const float PI = 3.14159265f;


SamplerState SampleTypePoint : register(s0);
SamplerState SampleTypeClamp : register(s1);

struct LightDirectionAndNumTextures{
	float3 Direction;
	uint NumTextures;
};

struct DiffuseColorAndBias{
	float3 DiffuseColor;
	float Bias;
};

struct LightColorAndDistance_t{
	float3 Color;
	float MaxLightRange;
};

struct LightPositionAndFlags_t{
	float3 Position;
	uint Flags;
};

struct SpotLightDirAndAngle_t{
	float3 DirNormalized;
	float Angle;
};

cbuffer LightBuffer
{
	matrix DirectionalLightMatrices[MAX_NUM_DIR_LIGHTS*MAX_NUM_TEXTURES_PER_LIGHT];
	
	matrix PointLightMatrices[MAX_NUM_POINT_LIGHTS_WITH_SHADOW];
	matrix PointLightOrientationMatrices[6];
	
	matrix SpotLightMatrices[MAX_NUM_SPOTLIGHTS_WITH_SHADOW];

	float3 ambientColor;
	uint NumDirectionalLights;

	DiffuseColorAndBias DiffuseColorsAndBias[MAX_NUM_DIR_LIGHTS];

	LightDirectionAndNumTextures LightDirsAndNumTextures[MAX_NUM_DIR_LIGHTS];

	LightPositionAndFlags_t pointLightPositions[MAX_NUM_POINT_LIGHTS];
	LightColorAndDistance_t pointLightColors[MAX_NUM_POINT_LIGHTS];
	
	uint pointLightCount;
	float ScreenWidth;
	float ScreenHeight;
	float padding0;

	LightPositionAndFlags_t spotLightPositions[MAX_NUM_SPOTLIGHTS];
	SpotLightDirAndAngle_t spotLightDirs[MAX_NUM_SPOTLIGHTS];
	LightColorAndDistance_t spotLightColors[MAX_NUM_SPOTLIGHTS];
	uint spotLightCount;
	float3 padding1;
};


struct PixelInputType
{
    float4 position : SV_POSITION;
};







float4 main(PixelInputType input) : SV_TARGET
{
    float4 normals;
	float4 positions;
    float lightIntensity;
    float4 OutputColor;
	float3 distanceVector;
	float4 pointLightShadow;
	uint i, j;
	float2 DirectionalLightTexCoords;
	float2 SpotLightTexCoords;
	
	uint3 TexCoords = uint3(input.position.xy, 0);

    positions = positionTexture.Load(TexCoords);

	OutputColor.rgb = ambientColor;
	OutputColor.a = 1.0f;

	if (positions.w == 1.0f){ //positions.w has been set to 1.0f in other pixel shaders if light calculations should be done for this pixel
		
		normals = normalTexture.Load(TexCoords);


		//////////////////////////
		/// DIRECTIONAL LIGHTS ///
		//////////////////////////
		
		for (i = 0; i < NumDirectionalLights && i < MAX_NUM_DIR_LIGHTS; i++){

			for (j = 0; j < LightDirsAndNumTextures[i].NumTextures && j < MAX_NUM_TEXTURES_PER_LIGHT; j++){

				float4 LightPosition = mul(float4(positions.xyz, 1.0f), DirectionalLightMatrices[i*MAX_NUM_TEXTURES_PER_LIGHT + j]);
				DirectionalLightTexCoords.x = LightPosition.x / LightPosition.w / 2.0f + 0.5f;
				DirectionalLightTexCoords.y = -LightPosition.y / LightPosition.w / 2.0f + 0.5f;

				if (DirectionalLightTexCoords.x >= 0.0f && DirectionalLightTexCoords.x <= 1.0f && DirectionalLightTexCoords.y >= 0.0f && DirectionalLightTexCoords.y <= 1.0f){

					float Depth = DirectionalLightTextures[i*MAX_NUM_TEXTURES_PER_LIGHT + j].Sample(SampleTypeClamp, DirectionalLightTexCoords);

					float LightDepth = LightPosition.z / LightPosition.w;

					LightDepth -= DiffuseColorsAndBias[i].Bias;

					if (LightDepth < Depth){

						OutputColor.rgb += (DiffuseColorsAndBias[i].DiffuseColor.xyz * saturate(dot(normals.xyz, LightDirsAndNumTextures[i].Direction)));


					}
					if (LightDepth <= 1.0f){
						break; //use only the most detailed depth texture of the light's textures for those the world position is in range of this texture
					}
				}

			}
		}
	

		OutputColor.rgb *= LightLevelTexture.Load(TexCoords) / 128.0f * 0.75f + 0.25f;
		 

		////////////////////
		/// POINT LIGHTS ///
		////////////////////

		
		for (i = 0; i<pointLightCount && i<MAX_NUM_POINT_LIGHTS_WITH_SHADOW; i++){
			distanceVector = positions.xyz - pointLightPositions[i].Position;
			float distance2 = dot(distanceVector, distanceVector);

			if (distance2 == 0.0f){
				OutputColor.rgb += pointLightColors[i].Color;
			}
			else if (distance2 < pointLightColors[i].MaxLightRange*pointLightColors[i].MaxLightRange){
				OutputColor.rgb += pointLightColors[i].Color * pow(sqrt(distance2) / (pointLightColors[i].MaxLightRange) - 1, 2);
			}

		}


		for (i = MAX_NUM_POINT_LIGHTS_WITH_SHADOW; i < pointLightCount && i < MAX_NUM_POINT_LIGHTS; i++){
			distanceVector = positions.xyz - pointLightPositions[i].Position;
			float distance2 = dot(distanceVector, distanceVector);

			if (distance2 == 0.0f){
				OutputColor.rgb += pointLightColors[i].Color;
			}
			else if (distance2 < pointLightColors[i].MaxLightRange*pointLightColors[i].MaxLightRange){
				OutputColor.rgb += pointLightColors[i].Color * pow(sqrt(distance2) / (pointLightColors[i].MaxLightRange) - 1, 2);
			}

		}

		///////////////////
		/// SPOT LIGHTS ///
		///////////////////

		
			for (i = 0; i<spotLightCount && i<MAX_NUM_SPOTLIGHTS_WITH_SHADOW; i++){

				float SpotLightIntensityShadow = 1.0f;
				if (spotLightPositions[i].Flags & 1 != 0){

					float4 LightPosition = mul(float4(positions.xyz, 1.0f), SpotLightMatrices[i]);
						SpotLightTexCoords.x = LightPosition.x / LightPosition.w / 2.0f + 0.5f;
					SpotLightTexCoords.y = -LightPosition.y / LightPosition.w / 2.0f + 0.5f;

					if (SpotLightTexCoords.x >= 0.0f && SpotLightTexCoords.x <= 1.0f && SpotLightTexCoords.y >= 0.0f && SpotLightTexCoords.y <= 1.0f){

						float Depth = spotLightTexture.Sample(SampleTypeClamp, float3(SpotLightTexCoords, i));

						float LightDepth = LightPosition.z / LightPosition.w;

						LightDepth *= 0.9996f;

						if (LightDepth >= Depth){
							SpotLightIntensityShadow = 0.5f;
						}
					}
				}
				distanceVector = positions.xyz - spotLightPositions[i].Position;

				float distance2 = dot(distanceVector, distanceVector);

				float SpotLightIntensity = acos(dot(normalize(distanceVector), spotLightDirs[i].DirNormalized));

				if (SpotLightIntensity > spotLightDirs[i].Angle) continue;

				SpotLightIntensity = 1 - pow(SpotLightIntensity / spotLightDirs[i].Angle, 2);

				if (distance2 < spotLightColors[i].MaxLightRange*spotLightColors[i].MaxLightRange){
					OutputColor.rgb += spotLightColors[i].Color * pow(sqrt(distance2) / (spotLightColors[i].MaxLightRange) - 1, 2) * SpotLightIntensity * SpotLightIntensityShadow;
				}

			}


			for (i = MAX_NUM_SPOTLIGHTS_WITH_SHADOW; i<spotLightCount && i<MAX_NUM_SPOTLIGHTS; i++){
				distanceVector = positions.xyz - spotLightPositions[i].Position;

				float distance2 = dot(distanceVector, distanceVector);

				float SpotLightIntensity = acos(dot(normalize(distanceVector), spotLightDirs[i].DirNormalized));

				if (SpotLightIntensity > spotLightDirs[i].Angle) continue;

				SpotLightIntensity = 1 - pow(SpotLightIntensity / spotLightDirs[i].Angle, 2);

				if (distance2 < spotLightColors[i].MaxLightRange*spotLightColors[i].MaxLightRange){
					OutputColor.rgb += spotLightColors[i].Color * pow(sqrt(distance2) / (spotLightColors[i].MaxLightRange) - 1, 2) * SpotLightIntensity;
				}

			}

		/*if(lightColor.r != 0.0f || lightColor.g != 0.0f || lightColor.b != 0.0f){
			lightColor=lightColor/max(lightColor.r,max(lightColor.g,lightColor.b));
		}*/
		


		OutputColor.rgb=saturate(OutputColor.rgb);
		

	}
	else{
		OutputColor.rgb = ambientColor;
	}

	
    return float4(OutputColor.rgb,1.0f);
}

Advertisement

It sounds like there's two possibilities here:

1. The heap is corrupt before you create the shader, and the compilation process just exposes the problem. If that's what's going wrong Windows has special heap debug modes
that you can enable with a tool called gflags which should help find the cause.

2. The shader compilation process corrupts the heap. Given that igd10ium64.dll is on the call stack that tells me it's an Intel driver. One simple way to rule out the driver is to try it on a PC with different hardware. You could also try different driver versions.

1. Thank you, I will try it.

2. It works on my desktop computer so it is likely that it's a driver bug. But why is it working when starting the game without a debugger - at least until I try to enter the ingame world?

But why is it working when starting the game without a debugger - at least until I try to enter the ingame world?


This tells me that you have an uninitialized pointer somewhere on the stack. Under the debug build the compiler will NULL it for you, under the release build it's pointing somewhere random. It most likely therefore a "your code" bug rather than a driver bug. It would help if you posted your shader loading and compiling code, since that is more likely to be where the bug is than in the actual HLSL code itself.

Direct3D has need of instancing, but we do not. We have plenty of glVertexAttrib calls.

Starting a program through the debugger also enables various heap debugging features, which means it's more likely to detect heap corruption.

You can disable those features by setting the environment variable _NO_DEBUG_HEAP, although that's probably not the ideal solution if it's a driver bug. Heap corruption bugs will tend to cause more problems (including crashes) later on.

You should contact Intel to report the bug, if it's still there on their latest drivers.


It works on my desktop computer so it is likely that it's a driver bug.

Unfortunately, it's not that straightforward.

Intel drivers typically don't maintain the strict separation between GPU memory and main memory that other vendors do (mostly because Intel chips don't have dedicated GPU memory). Which means that heap corruption can have all sorts of fun and interesting effects on the GPU, and GPU buffer overruns can in turn cause heap corruption.

It's often helpful to enable page heap checking to detect egregious instances of heap corruption.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

How could a GPU buffer overflow/overrun? Is there maybe a maximum byte size for compiled shaders? Or is the Intel driver assuming it has a maximum byte size? Maybe they use short for the size, which means 50 kB would result in a negative value... But I hope not because this would be ridiculous.


How could a GPU buffer overflow/overrun?

I don't think compiling a shader would do that, no. But we can't see what else you initialise before this point. Heap corruption tends to not manifest until much later in the program than it actually occurred.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

I know. Actually, I am not compiling the shader at runtime, it is compiled at compile time and at runtime the cso file is loaded.

And if there is a heap corruption, it should also be there on my desktop computer, but I never had any heap corruption on my desktop computer of which I didn't find the code causing it...

I will try gflags as soon as I have time for this - maybe in a few days. Thank you for your help.


I know. Actually, I am not compiling the shader at runtime, it is compiled at compile time and at runtime the cso file is loaded.

The driver for the graphics card will do another compile of the shader at runtime, to convert from the generic D3D bytecode, to a hardware specific format. To improve performance the driver will also do some optimizations on the shader at the same time. If you want to see the result of that process for ATI/AMD cards you can use http://developer.amd.com/tools-and-sdks/graphics-development/gpu-shaderanalyzer/

It could be that Intel have a bug in their code that's triggered by your shader, which is quite big and complicated. It could also be your other code that's causing the heap corruption, and the driver code just runs into it.

This topic is closed to new replies.

Advertisement