Sign in to follow this  

problem with using stencil

Recommended Posts

while I'm attempting to make a shadow volume demo, I'm experimenting how the stencil buffer works. I think I have understood the concept but my code doesnt work. the basic algorithm: 1: render a terrain with the natural light on, 2: use the stencil technique to mask part of the terrain by drawing an object (which will be replaced by the shadow volume) on the terrain. 3: update the fragments of the terrain which are inside of the object with the shadow light on. the goal is to render a terrain with some part in shadow.
void render_frame(void)
	d3ddev->Clear(0, NULL, D3DCLEAR_TARGET|D3DCLEAR_ZBUFFER, D3DCOLOR_XRGB(0, 0, 0), 1.0f, 0);
	d3ddev->Clear(0, NULL, D3DCLEAR_STENCIL, D3DCOLOR_XRGB(0, 0, 0), 1.0f, 0);


	// assume the #1 light is the natural light, #0 is the light used for drawing shadowed region.

	// *********************** pass 1  ***********************
	// pass 1
	d3ddev->SetRenderState(D3DRS_ZFUNC, D3DCMP_LESS); 

	d3ddev->SetRenderState(D3DRS_LIGHTING, TRUE);
	d3ddev->LightEnable(1, TRUE);  // turn on the normal light
	d3ddev->LightEnable(0, FALSE);// turn off the shadow light

	mesh_landscape->UpdateAll(); // update the matrices

	objects[0]->UpdateAll(); // draw some random object
	// *********************** shadow volume ***********************

		d3ddev->SetRenderState(D3DRS_LIGHTING, FALSE); // not bothering with light when drawing the shadow volume.


		d3ddev->SetRenderState(D3DRS_ZWRITEENABLE, FALSE);

		d3ddev->SetRenderState( D3DRS_COLORWRITEENABLE , FALSE); // turn off the color buffer.

		// stencil configurations
		d3ddev->SetRenderState(D3DRS_STENCILFUNC,        D3DCMP_ALWAYS); // stencil test always passes

		d3ddev->SetRenderState(D3DRS_STENCILPASS,       D3DSTENCILOP_INCR);
		d3ddev->SetRenderState(D3DRS_CULLMODE,D3DCULL_CCW); // render the front face of the shadow volume.
		objects[1]->draw(); // draw the front side of the shadow volume
		d3ddev->SetRenderState(D3DRS_STENCILPASS,       D3DSTENCILOP_DECR);
		d3ddev->SetRenderState(D3DRS_CULLMODE,D3DCULL_CW); // render the back of the shadow volume.
		objects[1]->draw();// draw the back side of the shadow volume.
		d3ddev->SetRenderState(D3DRS_CULLMODE,D3DCULL_CCW); // back to normal face culling.

		d3ddev->SetRenderState( D3DRS_COLORWRITEENABLE , 0xff); // turn the color buffer back on.
	// *********************** pass 2  ***********************
	// pass 2: draw the region that is covered by shadow.

		d3ddev->SetRenderState(D3DRS_LIGHTING, TRUE); // turn the light back on, but only the shadow light source to draw the shadowed region.
		d3ddev->LightEnable(0, TRUE);  // turn on the shadow light
		d3ddev->LightEnable(1, FALSE);// turn off the normal light

		d3ddev->SetRenderState(D3DRS_ZWRITEENABLE, TRUE); // turn z-buffer write back on.
		d3ddev->SetRenderState(D3DRS_ZFUNC, D3DCMP_EQUAL);

		// stencil configurations
		//d3ddev->SetRenderState(D3DRS_STENCILENABLE,      TRUE);
		d3ddev->SetRenderState(D3DRS_STENCILFUNC,        D3DCMP_NOTEQUAL); // for this part even i change it to D3DCMP_NEVER everything still gets drawn
		d3ddev->SetRenderState(D3DRS_STENCILREF,         0x0);

		d3ddev->SetRenderState(D3DRS_STENCILFAIL,        D3DSTENCILOP_KEEP);
		d3ddev->SetRenderState(D3DRS_STENCILZFAIL,       D3DSTENCILOP_KEEP);
		d3ddev->SetRenderState(D3DRS_STENCILPASS,        D3DSTENCILOP_KEEP);
		// redraw the landscape

	/*************************** end  *********************/


    d3ddev->Present(NULL, NULL, NULL, NULL);


the result is the whole terrain mesh is redrawn in the second pass. it seems the stencil testing doesnt reject the pixels that have a zero value. I have spent a ridiculous amount of time on this already. I greatly appreciate it for reading this.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this