Hello,

I've decided to unroll my own shader permutation system using an uber-shader and macro-defines. Every permutation option is represented as a bit of a 32/64 bit bitfield. In order to make startup-time bearable, I want to precompile all definitions that are known to be used in a game based on the games material. So far so good, but I'm still missing a good way of checking whether a certain option is even supported by the shader. E.g. if I bind a material with a normal map texture, but the shader does not support normal mapping, whats a good way to see that from the code and skip compilation of the obvious unecessary permutation? What do you guys use for that case, or do you rely on the user to asign the materials in a senseful manner?

My shader system worked pretty much exactly that way as well and using GLSL. Startup times are just fine. I reuse shader programs between materials that are already compiled. If I have 10 materials that all use diffuse, normal, and specular, and 3 materials that are only diffuse, I only compile 2 shaders on demand. Performance is pretty good.

And if you try to use an unsupported uniform or attribute for that shader, GlGetUniform will return an error, so it makes it easy to debug during development.

When I load up a shader I have my C++ code look at the bit mask.

Then I say:

if(shaderBitMask & NORMALS) {

defines += "#define NORMALS\n";

}

Then that defines string is passed in along with the shader text to the GLSL compiler. My shader text has

#ifdef NORMALS

code that uses normal goes here

#endif

At the moment though I'm writing an even more flexible system but this has worked for my school project for the last year or so. I was able to render some really huge environments with many shader permutations very nicely without any noticeable performance losses.

Yep, this is almost as my system works. However one think that you didn't mention, which was my original question, is what happens if there is a bitmask that defines an option that is not used in the shader. Say you have a bitmask with the normal option defined, used for a shader that does not utilizes this options. If this gets compiles, you have at least one redundand version of the shader (its the same as the one with the same bitmask without the normal setting, since there is no #ifdef for the NORMAL setting). How do you detected and resolve that case?

In GLSL parts of a shader get optimized out so even if you mistakenly keep an unecessary define it'll not be part of the compiled code. I find this makes debugging REALLY hard if I randomly comment out a portion of code that uses a diffuse texture for example. I then have to comment out the C++ code that also passes in the diffuse texture sampler or else it says unknown uniform.

I found using

min(0, sampler2D(diffuseTexture)) for example, would force a zeroing out without optimizing the diffuseTexture uniform out of the shader, making me not have to recompile the C++ if I'm debugging.

Also I have a material resource where I specify things like what normal map I use, what diffuse texture I use, etc... So these bitmasks are generated automatically for me and there's no human error. If I provide a normal map, my system automatically says:

bitmask |= NORMAL_MAP

bitmask |= TEX_COORDS

bitmask |= TANGENTS

If I just provide a diffuse texture I'd say

bitmask |= DIFFUSE_MAP

bitmask |= TEX_COORDS

since tangents aren't necessary unless you're doing normal mapping or other tangent space calculations in the shader. Textures all need UVs passed in so TEX_COORDS is passed in.

Since I have a deferred shader I always pass in normals for the deferred shading stage.

If I have a full bright unlit material, that doesn't need normals and is rendered in the forward shading stage. If I want to render a lit forward shaded material I can set the Fullbright flag to false in my material, meaning this material will be affected by lights. Then I do:

bitmask |= NORMALS

Then if I need skeletal animations for the material I also have the user say this material will be used on skinned meshes and then do:

bitmask |= SKELETAL_ANIMATIONS

It's possible to generate 2 shaders instead for the material, one with skeletal animations, and one without. But I figured I'd most likely almost never try to texture a character with a brick wall texture, and I'd definitely never try to texture a wall with a character skin. If I really wanted to I can just create 2 separate materials and the textures would be shared between them with my resource management system anyway. I'd still have 2 separate shaders.

Then the renderer can also have some asserts and other debug build only checks that make sure valid things are passed in so i don't mistakenly have a mesh without tangents in its vbo being used with a material that needs tangents.

I annotate my shader files with information about which options they support, for use by my shader compiler:

That seems practical, thanks! However, what are the different types (bool, int) used for? I always thought shader permutations are about simple binary "used/not used" scenarios?

Also, as a final catch-all, in the D3D version, my compiler (which runs ahead of time, pre-building every possible permutation) does a byte-by-byte comparison of every compiled binary shader program, and merges any duplicates that are generated.

Do you simply generate every available permutation, or do you do some content-processing to determine if a permutation does not need to be built because it can't be used? I quess otherwise the data-size could get quite out of hands...

Talking about it, yet another question that pops up, whats the best way for storing the permutations? I can't simply use an array / std::vector with the permutation-id as the key because of the sheer amount of possibilities. Is a std::map with the id as key more appropriate? Or an vector/list with a std::pair of key and permutation ptr?

A scenario where you might want to be able to use a range of values rather than on/off is when you're passing a variable sized array of something into a shader, but want to be able to unroll the loop that operates on that array (i.e. you want the array size known at compile time).
e.g. you might have a shader that supports 1-4 spotlights.

Ah, didn't consider that before, I wouldn't even really known how I could unroll such a thing before, but with the annotations, it now makes perfect sense. In retrospective, I was also wondering how you would e.g. tie in the different names for the defines, like OPTION_NORMAL, etc.. where different types of shaders would use different annotations (it doesn't make sense to have a OPTION_SKINNING-field for a water-shader), but that seems doable with that :D

At the moment I build all permutations. I don't have any shaders with 232 permutations, etc... I think the largest I've got at the moment is 25 permutations, and the largest I've had in the past was about ~240 valid permutations... so my data hasn't forced me to solved that problem yet biggrin.png

Ah, so if you haven't used a much larger amount of permutations, I doubt I will eigther. I don't have a clue how much it will turn out to be right now, since I'm in an experimental stage and have to rewrite my shaders to support this, but I quess it won't reach that high of numbers eigther. Still, since I'm writing an engine that could probably be used by different people with different requirements, I'd like to account for that possible problem too. So I quess I'll start with taking the materials into consideration for pre-compiling only valid shaders, as to answer your last question...

Inside your rendering code, how do you intend to select a set of actual shader programs (i.e. pick the right permutation) for each object? Do you just get the material's permutation bitfield, and then find the programs that match it exactly?
Do you have any concept of an "engine option" like I mentioned above?

Right now, I don't have such a thing as engine option, and I'd just go with the materials bitfield. I'll definately plan on adding it in the future, as things like glow on highligh are features I wouldn't want to miss out at some point. Still, I do generate some materials in code rather than through tools right now, since I don't have a wrapper for e.g. fullscreen-effects like for blurring, hdr-luminance downsampling, etc... which not all can be generated from a material-definition-file (in my current design (?)). I quess that marking these shaders options as unpredictable engine options isn't the best choice here, since they are only unkown because my engine can't handle it otherwise right now. Would simply allowing the engine to compile missing effect permutations at runtime be a viable solution here?

I went with Deferred shading to avoid some of the crazier permutations of lights and other options, and also because it's awesome and makes a whole lot of other things suddenly much easier, like decals.

The most used case ends up being solid objects that are renderable with deferred shading, and then you can make a few sacrifices on things that aren't deferred shadeable. Then again, if you're doing this on a mobile phone that may not be an option yet. In that case, the hardware may not be powerful enough to support some options anyway.

I just used an unordered_map to get permutation mask to shader.