Jump to content

  • Log In with Google      Sign In   
  • Create Account

Shader permutation: permutation support detection


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
8 replies to this topic

#1 Juliean   GDNet+   -  Reputation: 2607

Like
0Likes
Like

Posted 29 August 2013 - 10:25 AM

Hello,

 

I've decided to unroll my own shader permutation system using an uber-shader and macro-defines. Every permutation option is represented as a bit of a 32/64 bit bitfield. In order to make startup-time bearable, I want to precompile all definitions that are known to be used in a game based on the games material. So far so good, but I'm still missing a good way of checking whether a certain option is even supported by the shader. E.g. if I bind a material with a normal map texture, but the shader does not support normal mapping, whats a good way to see that from the code and skip compilation of the obvious unecessary permutation? What do you guys use for that case, or do you rely on the user to asign the materials in a senseful manner?



Sponsor:

#2 ill   Members   -  Reputation: 320

Like
1Likes
Like

Posted 29 August 2013 - 04:27 PM

My shader system worked pretty much exactly that way as well and using GLSL.  Startup times are just fine.  I reuse shader programs between materials that are already compiled.  If I have 10 materials that all use diffuse, normal, and specular, and 3 materials that are only diffuse, I only compile 2 shaders on demand.  Performance is pretty good.

 

And if you try to use an unsupported uniform or attribute for that shader, GlGetUniform will return an error, so it makes it easy to debug during development.

 

When I load up a shader I have my C++ code look at the bit mask.

 

Then I say:

 

if(shaderBitMask & NORMALS) {

   defines += "#define NORMALS\n";

}

 

Then that defines string is passed in along with the shader text to the GLSL compiler.  My shader text has

 

#ifdef NORMALS

   code that uses normal goes here

#endif

 

At the moment though I'm writing an even more flexible system but this has worked for my school project for the last year or so.  I was able to render some really huge environments with many shader permutations very nicely without any noticeable performance losses.


Edited by ill, 29 August 2013 - 04:33 PM.


#3 Juliean   GDNet+   -  Reputation: 2607

Like
0Likes
Like

Posted 29 August 2013 - 05:43 PM

Yep, this is almost as my system works. However one think that you didn't mention, which was my original question, is what happens if there is a bitmask that defines an option that is not used in the shader. Say you have a bitmask with the normal option defined, used for a shader that does not utilizes this options. If this gets compiles, you have at least one redundand version of the shader (its the same as the one with the same bitmask without the normal setting, since there is no #ifdef for the NORMAL setting). How do you detected and resolve that case?



#4 Hodgman   Moderators   -  Reputation: 30384

Like
2Likes
Like

Posted 29 August 2013 - 09:03 PM

I annotate my shader files with information about which options they support, for use by my shader compiler:
/*[FX]
Option( 'opt_integer1', {id=0, max=3} )
Option( 'opt_boolean1', {id=3} )
Option( 'opt_boolean2' )
*/
That 'FX' block also describes the cbuffers, passes, textures, etc, that are required by the shader.

When compiling a shader, you can use that information to see which #defines are valid.
The above shader will be compiled with:
#define opt_integer1 0/1/2/3
#define opt_boolean1 0/1
#define opt_boolean2 0/1

The Id/Max variables are used to describe the bitmask layout. In the bitmask, bits #0/#1 correspond to opt_integer1, opt_boolean1 corresponds to bit #3, and opt_boolean2 is auto-assigned to the first unused bit, which in this case is #2.


Also, as a final catch-all, in the D3D version, my compiler (which runs ahead of time, pre-building every possible permutation) does a byte-by-byte comparison of every compiled binary shader program, and merges any duplicates that are generated.

#5 ill   Members   -  Reputation: 320

Like
1Likes
Like

Posted 29 August 2013 - 11:00 PM

In GLSL parts of a shader get optimized out so even if you mistakenly keep an unecessary define it'll not be part of the compiled code.  I find this makes debugging REALLY hard if I randomly comment out a portion of code that uses a diffuse texture for example.  I then have to comment out the C++ code that also passes in the diffuse texture sampler or else it says unknown uniform.  

 

I found using

 

min(0, sampler2D(diffuseTexture)) for example, would force a zeroing out without optimizing the diffuseTexture uniform out of the shader, making me not have to recompile the C++ if I'm debugging.

 

Also I have a material resource where I specify things like what normal map I use, what diffuse texture I use, etc...  So these bitmasks are generated automatically for me and there's no human error.  If I provide a normal map, my system automatically says:

 

bitmask |= NORMAL_MAP

bitmask |= TEX_COORDS

bitmask |= TANGENTS

 

If I just provide a diffuse texture I'd say

 

bitmask |= DIFFUSE_MAP

bitmask |= TEX_COORDS

 

since tangents aren't necessary unless you're doing normal mapping or other tangent space calculations in the shader.  Textures all need UVs passed in so TEX_COORDS is passed in.

 

Since I have a deferred shader I always pass in normals for the deferred shading stage.

 

If I have a full bright unlit material, that doesn't need normals and is rendered in the forward shading stage.  If I want to render a lit forward shaded material I can set the Fullbright flag to false in my material, meaning this material will be affected by lights.  Then I do:

 

bitmask |= NORMALS

 

Then if I need skeletal animations for the material I also have the user say this material will be used on skinned meshes and then do:

 

bitmask |= SKELETAL_ANIMATIONS

 

It's possible to generate 2 shaders instead for the material, one with skeletal animations, and one without.  But I figured I'd most likely almost never try to texture a character with a brick wall texture, and I'd definitely never try to texture a wall with a character skin.  If I really wanted to I can just create 2 separate materials and the textures would be shared between them with my resource management system anyway.  I'd still have 2 separate shaders.

 

Then the renderer can also have some asserts and other debug build only checks that make sure valid things are passed in so i don't mistakenly have a mesh without tangents in its vbo being used with a material that needs tangents.


Edited by ill, 29 August 2013 - 11:10 PM.


#6 Juliean   GDNet+   -  Reputation: 2607

Like
0Likes
Like

Posted 30 August 2013 - 08:16 AM


I annotate my shader files with information about which options they support, for use by my shader compiler:

 

That seems practical, thanks! However, what are the different types (bool, int) used for? I always thought shader permutations are about simple binary "used/not used" scenarios?

 


Also, as a final catch-all, in the D3D version, my compiler (which runs ahead of time, pre-building every possible permutation) does a byte-by-byte comparison of every compiled binary shader program, and merges any duplicates that are generated.

 

Do you simply generate every available permutation, or do you do some content-processing to determine if a permutation does not need to be built because it can't be used? I quess otherwise the data-size could get quite out of hands...

 

Talking about it, yet another question that pops up, whats the best way for storing the permutations? I can't simply use an array / std::vector with the permutation-id as the key because of the sheer amount of possibilities. Is a std::map with the id as key more appropriate? Or an vector/list with a std::pair of key and permutation ptr?



#7 Hodgman   Moderators   -  Reputation: 30384

Like
2Likes
Like

Posted 30 August 2013 - 08:35 AM

That seems practical, thanks! However, what are the different types (bool, int) used for? I always thought shader permutations are about simple binary "used/not used" scenarios?

They're all integers in mine, but with a variable number of bits. If you don't specify, then you only get 1 bit by default, which is a boolean option wink.png
A scenario where you might want to be able to use a range of values rather than on/off is when you're passing a variable sized array of something into a shader, but want to be able to unroll the loop that operates on that array (i.e. you want the array size known at compile time).
e.g. you might have a shader that supports 1-4 spotlights.
 

Do you simply generate every available permutation, or do you do some content-processing to determine if a permutation does not need to be built because it can't be used?

At the moment I build all permutations. I don't have any shaders with 232 permutations, etc... I think the largest I've got at the moment is 25 permutations, and the largest I've had in the past was about ~240 valid permutations... so my data hasn't forced me to solved that problem yet biggrin.png
 
If I did end up in a situation where I had millions of valid permutations, I would add an extra annotation to my 'Option' field, specifying whether it is an "engine option", or a "material option".
The difference is that:
* "engine options" are set at runtime and can't be predicted -- e.g. is the object highlighted by a ray gun, so should be glowing right now. The compiler has to build all possible permutations of these options.
* "material options" are set by the artists and then don't change during gameplay. They can be known ahead of time, so the compiler can see which option values are used.
 
Then once I had that data (of knowable options and unknowable options), I'd generate a list of all the materials that use each shader, then generate a unique list of their "option" bitfields. Then I'd use that list, and the knowledge of the unknowable "engine options" to compile just the possibly-used permutations.

 

I know of one game engine where, at the moment that the artist selects a certain configuration of options, that permutation is compiled on the spot and added to a big cache on the network server (if it doesn't already exist). When it comes to shipping, this server contains all the required binary shader code (not much use for GL though...)
 

Talking about it, yet another question that pops up, whats the best way for storing the permutations? I can't simply use an array / std::vector with the permutation-id as the key because of the sheer amount of possibilities. Is a std::map with the id as key more appropriate? Or an vector/list with a std::pair of key and permutation ptr?

As above, I've not had to deal with this yet, so I just use an array of permutation bitfields... so I haven't thought about this too much.
 
Inside your rendering code, how do you intend to select a set of actual shader programs (i.e. pick the right permutation) for each object? Do you just get the material's permutation bitfield, and then find the programs that match it exactly?
Do you have any concept of an "engine option" like I mentioned above?


Edited by Hodgman, 30 August 2013 - 08:38 AM.


#8 Juliean   GDNet+   -  Reputation: 2607

Like
0Likes
Like

Posted 30 August 2013 - 09:17 AM


A scenario where you might want to be able to use a range of values rather than on/off is when you're passing a variable sized array of something into a shader, but want to be able to unroll the loop that operates on that array (i.e. you want the array size known at compile time).
e.g. you might have a shader that supports 1-4 spotlights.

 

Ah, didn't consider that before, I wouldn't even really known how I could unroll such a thing before, but with the annotations, it now makes perfect sense. In retrospective, I was also wondering how you would e.g. tie in the different names for the defines, like OPTION_NORMAL, etc.. where different types of shaders would use different annotations (it doesn't make sense to have a OPTION_SKINNING-field for a water-shader), but that seems doable with that :D

 


At the moment I build all permutations. I don't have any shaders with 232 permutations, etc... I think the largest I've got at the moment is 25 permutations, and the largest I've had in the past was about ~240 valid permutations... so my data hasn't forced me to solved that problem yet biggrin.png

 

Ah, so if you haven't used a much larger amount of permutations, I doubt I will eigther. I don't have a clue how much it will turn out to be right now, since I'm in an experimental stage and have to rewrite my shaders to support this, but I quess it won't reach that high of numbers eigther. Still, since I'm writing an engine that could probably be used by different people with different requirements, I'd like to account for that possible problem too. So I quess I'll start with taking the materials into consideration for pre-compiling only valid shaders, as to answer your last question...

 


Inside your rendering code, how do you intend to select a set of actual shader programs (i.e. pick the right permutation) for each object? Do you just get the material's permutation bitfield, and then find the programs that match it exactly?
Do you have any concept of an "engine option" like I mentioned above?

 

Right now, I don't have such a thing as engine option, and I'd just go with the materials bitfield. I'll definately plan on adding it in the future, as things like glow on highligh are features I wouldn't want to miss out at some point. Still, I do generate some materials in code rather than through tools right now, since I don't have a wrapper for e.g. fullscreen-effects like for blurring, hdr-luminance downsampling, etc... which not all can be generated from a material-definition-file (in my current design (?)). I quess that marking these shaders options as unpredictable engine options isn't the best choice here, since they are only unkown because my engine can't handle it otherwise right now. Would simply allowing the engine to compile missing effect permutations at runtime be a viable solution here?



#9 ill   Members   -  Reputation: 320

Like
0Likes
Like

Posted 30 August 2013 - 11:06 AM

I went with Deferred shading to avoid some of the crazier permutations of lights and other options, and also because it's awesome and makes a whole lot of other things suddenly much easier, like decals.

 

The most used case ends up being solid objects that are renderable with deferred shading, and then you can make a few sacrifices on things that aren't deferred shadeable.  Then again, if you're doing this on a mobile phone that may not be an option yet.  In that case, the hardware may not be powerful enough to support some options anyway.

 

I just used an unordered_map to get permutation mask to shader.






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS