So I've read its best to write dedicated shaders for specific purposes rather than bloat them with too many conditionals. And it makes sense. But I look at how many options are possible and it quickly becomes overwhelming:

1. Normal mapping

2. GPU Tesselation / Displacement mapping

3. Shadow mapping

4. Animation/Skinning

I could go with or without the each of these for any particular shader, resulting in 16 possible shaders. When you consider I have to write shaders for shadow passes for different types of lights (point, spot, directional, etc). the number of shaders quickly becomes ridiculous.

What's a smart way to cluster these and keep it reasonable?

You don't need to write a shader for everything from scratch.

Both DirectX and OpenGL are offering to merge multiple vertex/fragment shaders to one.

I guess this has also a performance impact as only parts of a shader get changed but I don't know.

In OpenGL (GLSL) you can basically define two shaders with the main method as well as some shaders defining methods used by both main-shaders.

On creation you link the additional shaders together with the main shaders to one program and you are ready to go.

This problem is usually solved by using shader permutations.

Basically you write a single shader containing *all* techniques you want to support (all the ones you mentioned and more). But separate each technique inside preprocessor #if X -> #endif sections.

Eg:

 
float3 position;
float3 normal;
#if SKINNING
    skin(position)
#endif
 
#if NORMAL_MAPPING
    process(normal)
#endif
 
//ETC

Then you generate multiple permutations of one shader file by compiling it multiple times with different defines (4th argument of D3DCompile function).

At runtime you select the appropriate shader permutation based on the requirements of the object you are drawing.

More info here

This way you only have to manage a shader source file and each permutation only contains the required code (no branches at runtime).

You can also try to analyse this shader for Bitsquid engine (start at line 1389 for some easy to understand code) (it combines multiple techniques like I explained)

Keep in mind that a system like this looks simple but can become quite complex once you add dependencies between techniques, keep generated permutations optimized, etc.

(For my engine I wrote a tool that manages the different permutations)

Also, each new technique can potentially double the number of permutations, so it's probably best to keep a database of the permutations your game actually need and only generate those. More info in this post

PS: Above, when I said a single shader file, I actually mean one shader file per type of object. For example I have a shader file for 3D actors, one for the terrain, a few for particles, etc. I could put them all in the same file but it doesn't make much sense since they don't share techniques (terrain and particles dont support skinning, etc)

"How many dedicated Shaders to use?"

as many as you need! Just try to minimize the work as suggested and try to see the repeating parts - don't write the same code multiple times, use include files to store common functions.

Also, store all the compiled shaders to disk - as the project grows bigger the number of shaders increase and the length of shader files too, so compiling gets slower and slower. I have a shader manager which at start up loads the shader cache file (if it exists). After it checks if the non-compiled shader on disk is newer than the shader cache version - if so, the shader from the disk is loaded and compiled. This method supports also recompilation of the shaders while the program is running (very useful feature). The shader cache stores (for each shader) the shader file name, shader entry point, shader file date/time and the shader blob data.

Cheers!