I'm having my game use a simple material system similar to Doom 3 where I can specify that the material has certain diffuse map, specular, normal, glow, so on... I want to be able to turn these on and off on a per material basis. There would also be additional parameters like blend mode, depth mask, billboarding, things like that...

On top of that I also need to have skeletal animations and particle effects working with shaders on a per object basis, meaning I need to do additional vertex shaders only if the object being drawn needs it.
[/quote]

If you can wrap your vertex shader in an if clause, or find some other way to have it noop if you don't need it, that's best. If you want to mix-n-match different vertex shaders you're just going to end up with different draw calls, which is usually best avoided.

So basically I might create a material for a brick wall using a diffuse map, normal map, and specular map and have it be texturing some wall.
Then I might create another material for a shiny metal with a specular map, some diffuse map, and a random glow map for some glowy bits.
Does this require different permutations of shaders compiled?
[/quote]

Not necessarily. For monolithic shaders usually the idea is something like (in HLSL psuedocode):


float4 pixelShader
{
float4 glowColor;
if(GLOW)
{
glowColor = ...
}

float4 diffuseColor;
if(DIFFUSE)
{
diffuseColor = ...
}

float4 finalColor = glowColor + diffuseColor;

return finalColor
}


You don't automatically have to use ifs though. You can do something like this instead:

float4 pixelShader
{
float4 glowColor = ...
float4 diffuseColor = ...
float4 finalColor = glowColor * GLOW + diffuseColor * DIFFUSE;

return finalColor
}


That is, always assume everything is turned on, and just multiply the components by 0 or 1 at the very end to turn on or off that feature.

Let's say I also decide to use the brick material to texture an animated skeletal model. Does that same brick material now need another shader that renders the brick both normally and one that is combined with a vertex shader that is used when rendering the model?
[/quote]

Not necessarily. Ideally your shader could handle both cases, but you might not have enough instruction count to have a shader that big.

Would this be where an uber shader comes in to play with ifdefs enabling/disabling features? I can't find any actual concrete tutorials on GLSL and uber shaders so I can only guess.
[/quote]

You can use ifdefs, but then each combination of features is its own different shader, and therefore a totally different draw call, which defeats the purpose of an uber shader. You would have less instruction count, though.

I'm also a bit confused about texture units. Let's say I only have 4 texture units available on the card. I want to pass diffuse, normal, specular, height, and glow maps. That's 5 textures. Does that mean one shader can't do it? Would I have to have multiple shaders render the image with multiple passes and then overlay them to get the final image?
[/quote]

Yeah, you can only address up to 4 textures in a single pass/draw call if you only have 4 texture units on the card. You could either split your shader into a few different passes (with one draw call per pass), or mung your 5 textures into 4 or less textures. eg: Maybe your diffuse texture is 4 times larger than the others. Then just combine the 4 smaller textures into a single texture. Then you are only using 2 texture units.

Also let's say I added Screen Space Ambient Occlusion. As I understand something like SSAO requires a separate pass where you only render the depth data. Does this mean effects like SSAO that need separate passes have separate shaders per pass per material? So I might potentially have 5 fragment shaders for a brick wall if I have to do 5 passes of some sort?
[/quote]

Each pass can have its own arbitrary vertex and pixel (fragment) shaders. But each pass is another draw call, so you try to combine passes if at all possible. Whether you need a separate pass for SSAO or not depends on how clever you are with it. I haven't implemented SSAO before, so I don't know what data it needs or why it might need a separate pass. But often times you can steal the alpha channel on your render target to store extra data that would otherwise require another pass.

Heh, also on a side note I'm trying to figure out how I will support specular maps. I'm thinking RGB will be specular and the alpha component will be the gloss. Is this a good idea? This way I combine the components in to one texture instead of having one texture be specular, and one be gloss. Is there ever a time specular maps actually need alpha for the transparency?
[/quote]

That's good thinking, yeah. Whether you need the alpha on specular or not depends on what you're doing in your shader. Does your shader even look at the alpha? If not, you don't need it.

The only tricky bit is that you'll need a tool side step that combines them. Don't expect artists to make the textures like that for you. It's annoying for artists, and it means you can't easily undo it if you decide specular textures need alpha in the future.

Same with glow maps. Does it make sense for a glow map to have an alpha channel for transparency? As far as I understand, the glow works like additive blending, black is no glow which would act like transparency.
[/quote]

Again, look at your shader.

While there are some commonly accepted norms for how glow, diffuse, etc. work, keep in mind that they aren't set in stone. With modern shaders you can invent any wacky system you want, as long as it works for artists, and you can spit out a final RGB(A) from your fragment shader. Don't feel hamstrung trying to implement someone else's graphics model; you have huge freedom with programmable shaders.

Generally, I'd start from specific graphical effects you want and work backwards through the shader back to the toolchain and what artists actually produce.