Archived

This topic is now archived and is closed to further replies.

Cel Shading... Using ProcessVertices

This topic is 5303 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I was looking at techniques for cel shading using vertex shaders. They manually calculate lighting in the shader as the fixed pipline is replaced. Surely this is slower than leaving the hardware to do it? (not to mention messy when you want to combine cel shading with other techniques). So I was looking at ProcessVertices (ID3DDevice9). My idea is to process the vertices with transforms, lighting and whatever and then to have a small vertex shader simply to do the texture mapping required for cel shading. Firstly, being a novice at this I cannot tell from the documentation whether this can use graphics cards or not. Second, I was wondering if this was a good approach at all. Would it be faster, for example? Or would it use up too much memory (requiring two copies of everything)? Also post if you have a better way to do cel shading other than using a vertex shader to swap lighting for a discreet texture.

Share this post


Link to post
Share on other sites
Do you mean that your card doesn''t support hardware vertex shaders?vertex shader can be calculated in hardware sometimes faster than fixed function pipeline on cards that support hardware vertex shaders.also cel shading requires special techniques for light rendering and applying colors from a lookup textre that consist of one color with varying color depth,so you can''t apply a texture in cel shading (as a wooden texture for example).only you can apply colors not textures.

Share this post


Link to post
Share on other sites
This is slightly off-topic because I''m unfamiliar with the "ProcessVertices" function. I just wanted to point out that it IS possible to have textures when you cel-shade. I created the following example based on the "SkinnedMesh" sample in dx8, using multi-texturing and a shader (which is executed in the hardware if you have a GeForce 256 or newer card):



Share this post


Link to post
Share on other sites
quote:
Original post by sadwanmage
Surely this is slower than leaving the hardware to do it?


I'm not sure what you mean by that. If you have a newer graphics card then shaders are totally done in hardware, and on some chips is actually faster than the fixed function pipeline (it'd only be slower with a very complex shader).


quote:
So I was looking at ProcessVertices (ID3DDevice9). My idea is to process the vertices with transforms, lighting and whatever and then to have a small vertex shader simply to do the texture mapping required for cel shading.

Firstly, being a novice at this I cannot tell from the documentation whether this can use graphics cards or not.


ProcessVertices() ALWAYS runs in software - there's no way to "read back" post transformed vertices from a hardware T&L chip.

On a fixed function hardware T&L chip, the shader will be done in software too.

Then finally you'll be passing the vertices to the hardware for rasterisation only.

Generally, that's not a good plan.




quote:
Second, I was wondering if this was a good approach at all. Would it be faster, for example? Or would it use up too much memory (requiring two copies of everything)?


Nope, see above, not a good plan.

1) older graphics chips that came out before GeForce256 (aka GF1) can't do ANY vertex processing in hardware so:
- vertex shaders are processed in software (by the CPU),
- fixed function draw calls are processed in software,
- and ProcessVertices is in software.

2) newer graphics chips such as the GeForce3 and above do all write only vertex processing in hardware:
- vertex shaders are processed in hardware,
- fixed function vertex processing draw calls are done in hardware,
- and ProcessVertices is in software.

3) the first generation T&L chips such as the GeForce256 can do some vertex processing in hardware:
- vertex shaders are processed in software.
- fixed function vertex processing draw calls are done in hardware
- and ProcessVertices is in software.


So you're doing a lot of worrying just for type 3 - i.e. stuff that only does some hardware vertex processing. Although there's a lot of those cards out there, people are already starting to upgrade to shader capable cards.


Personally I'd just use shaders directly - with software VP for any device which didn't do shaders in hardware (i.e. type 3) - you lose a small bit of hardware T&L benefit by doing it.

quote:
Also post if you have a better way to do cel shading other than using a vertex shader to swap lighting for a discreet texture.


I'm not sure what you mean by "swap lighting for a discreet texture" - the usual way is to plug the result of (L.N) [etc] summed for all the lights into a 1D texture coordinate which is used to look up into a "shades" texture. In the pixel pipeline that's blended with the base texture.

Comparisons in a pixel shader could avoid that extra texture lookup if its bothering you.

[edit: fixed quote tags]

--
Simon O'Connor
Creative Asylum Ltd
www.creative-asylum.com

[edited by - S1CA on June 6, 2003 9:19:14 AM]

Share this post


Link to post
Share on other sites
Khalie: Could you please tell me how you got Cel Shading working with a skinned mesh? i''ve been trying it but when i animate the character it gets all weird.

Do you have some example code? Thanks.

Share this post


Link to post
Share on other sites
Dreamcatcher:

You know how to cel-shade a model with a 1d-texture, right? This is actually the same thing, except you place the 1d-texture as a second texture and use it as a "lightmap":


m_pd3dDevice->SetTextureStageState( 1, D3DTSS_COLORARG1, D3DTA_TEXTURE );
m_pd3dDevice->SetTextureStageState( 1, D3DTSS_COLORARG2, D3DTA_CURRENT );
m_pd3dDevice->SetTextureStageState( 1, D3DTSS_COLOROP, D3DTOP_MODULATE );


Then you have to calculate the texture coordinates in the vertex shader (except you''re calculating oT1 instead of oT0)... I just added this in the end of the skinned mesh shaders:


dp3 r1, r5, c1
mov oT1.x, r1
mov oT1.y, c0.y


r5 is vertex normal (already calculated in the vertex shader if you check the skinnedmesh sample), c1 is light direction, c0.y is 0.5 . Note: To make this work you need to set the renderstate to use clamp texture mode (or else the shades will "loop"):


m_pd3dDevice->SetTextureStageState( 1, D3DTSS_ADDRESSU, D3DTADDRESS_CLAMP );
m_pd3dDevice->SetTextureStageState( 1, D3DTSS_ADDRESSV, D3DTADDRESS_CLAMP );


Hope that makes sense.

Share this post


Link to post
Share on other sites