Hi.

I want to write a cell shaded renderer. Im breaking this into 2 parts. As far as I can see to get a proper cell shaded effect you need to limit the tones. For example a smooth gradient would become one solid colour. And secondly to add black outlines to the rendered result like you would see in a hand drawn "cartoonish" or "anime'ish" picture.

The first step seems the easiest. For diffuse lighting I was thinking of simply rounding (or maybe ciel) the NDotL to the nearest 0.1 or 0.2 or whatever. Asuming that the models textures dont contain any gradients this should be it for step 1

Now step 2. for creating the black outlines i was thinking of writing depth values to an FBO in the first pass while also drawing the entire screen to the backbuffer (simple forward rendering since i dont think alot of light sources would work with cell shading anyway). Then il just render a fullscreen quad directly onto the backbuffer using the depth fbo and for every fragment il check the difference in depth between it and surrounding fragments and based on that either make the fragment black or disgard it (or maybe even blend between if it will look better for edge cases).

So far this seems pretty straight forward. Is there any other methods I should be considering? or maybe even more steps altogether.

Thnx in Advance!

For diffuse lighting I was thinking of simply rounding (or maybe ciel) the NDotL to the nearest 0.1 or 0.2 or whatever. Asuming that the models textures dont contain any gradients this should be it for step 1

Use the texture for color informations only (albedo), what you need to do is to limit the light. The NDotL approach is ok, when not using many lights you can use a lookup texture to define a smooth gradient map.

For the outlines use the depth and normals in combination with a sobel filter.

[quote name='Wilhelm van Huyssteen' timestamp='1343303114' post='4963262']
For diffuse lighting I was thinking of simply rounding (or maybe ciel) the NDotL to the nearest 0.1 or 0.2 or whatever. Asuming that the models textures dont contain any gradients this should be it for step 1

For the outlines use the depth and normals in combination with a sobel filter.
[/quote]

Do you mean seperately? as in use the depth+normals to calculate a "blackness" value and use sobel filter to calculate another seperate "blackness" value and then add the 2 together?

For direct cell shading control use a 1D texture of all the tones you want. Since dot product is in range 0 to 1, use that as a 1D UV coord into the texture. You can then at any time change from 2 tones to 4,8....and so on.

For outlines it depends how specific you want to outline. There is a case where say a camera is in front of you and you have your hand over your chest. You either want just the silhouette of the entire body, you want to draw outlines around the hand. (The hand is inside the silhoutte of the chest/body).

For just silhouette, draw your model 2x, once with GL_LINES and changing the GL_LINE_WIDTH bigger depending on your outline thickness.
For the second case of all outlines, just take the normal relative to the eye/screen. As it approaches 0, that means the normal is starting to point away from you.
if( dot(normal, vec3(0,0,1) < VALUE)
{
gl_FragColor = black;
}
Where value would be between 0 and maybe .3 depeding on how thick you want it to be.

For the second case of all outlines, just take the normal relative to the eye/screen. As it approaches 0, that means the normal is starting to point away from you.
if( dot(normal, vec3(0,0,1) < VALUE)
{
gl_FragColor = black;
}
Where value would be between 0 and maybe .3 depeding on how thick you want it to be.

O nice. This way it can even all be done in one pass as well O and using a 1D texture for the tones is nifty as well.

Thnx!

Do you mean seperately? as in use the depth+normals to calculate a "blackness" value and use sobel filter to calculate another seperate "blackness" value and then add the 2 together?

It is more like a OR, either depth or normals differs enough to draw an outline (it is not exactly a outline any longer ), think of a corner or an transition from floor to wall, there is not really a difference between the depth value, but the normal changes.

When considering the normal, you should take care of normal mapped geometry, this could introduce unwanted lines. In this case you should use two normal buffers, one for the normal mapped surfaces and one for the standard surface normals.

Here's a screenshot of the outlines in my game, using a single pass post-processing step to draw them (sobel/depth/normal). The effect is toned down and colored, so it is hard to read all the outlines from the screenshot, but you can download the pre-alpha version and see it in action (follow my signature).

Ah. Thnx! making use of both normals and depth makes sense to get the correct "outline" in all (most?) cases. ("outline" being the lines that a 2D artist would draw before doing the colouring and not just a line that goes around the outskirts of the model.)

But you still add a sobel filter afterwards? If its not too much effort could you perhaps show me that same scene without the sobel filter? or any other scene if you allready have a screenshot like that.

But you still add a sobel filter afterwards?

This could be a missunderstanding. The sobel filter is actually the outline detection algorithm. It is used to detect edges between values (normal/depth in my case), by retrieving the 8 neighbor pixels (3x3 sobel kernel), summing the values up (weighted) and compare the weighted sum to the current pixel value.

or any other scene if you allready have a screenshot like that.

Here a journal entry about my outline implementation including some comparision shots (outline on/off). An other option is to check it in my game by turning it off in the configuration settings (set post processing lowest setting to turn outlines off).

For the second case of all outlines, just take the normal relative to the eye/screen. As it approaches 0, that means the normal is starting to point away from you.
if( dot(normal, vec3(0,0,1) < VALUE)
{
gl_FragColor = black;
}
Where value would be between 0 and maybe .3 depeding on how thick you want it to be.

A method that provides better results is to draw the model a second time with culling reversed and use the vertex shader to extrude the vertices outwards by some amount. The pixel shader just returns black (or whatever color you want).
The provides an outline that does not cover any of the original image (unless of course the object overlaps itself) and remains a constant size.


L. Spiro