Sign in to follow this  
MChiz

Shader permutation problem ( again )

Recommended Posts

MChiz    161
Hi all: I have been reading a lot about shader permutation problem and I'm getting a little bit frustated because I really don't know which solution is better: - uber-shaders: I can't use it because I don't like it because the performance hit of dynamic branching and I must target low graphic cards. - Generate custom shaders on the fly: I have tried this solution but I really don't feel good with it. I have read the article of Shawn Hargreaves and did a little implementation of something similiar but I think is not the way to go ( I like to have more control of my code ) - Use ID3DXEffectCompiler enabling constants at compile time: I like it, but it doesn't seems the way to go neither for me. The main problem with it ( it happens with 'generatin custom shaders on the fly' ) is that I can't ( as far as I know ) save all shaders at disk in one effect and read it with one 'create' call ( I'm targetting to consoles so it matters to cut down loading time ). - Precompile all the possible permutations of the shader, even if I don't use a lot of them: I think this is the best, but I prefer to ask you. I like it because I can generate it totally offline without having to parse all the game data ( even parsing it, maybe the game needs some special combination that is not contemplated on game data ). Besides I can have only one ID3DXEffect ( per shader type ) with N techniques. The only thing the material has to compute is the technique name ( and even that can be computed offline and saved as an index when possible ). Even more, I feel like this solution is the most cross platform, because I can generate all the combinations for XBox360 and PC ( HLSL ), and PS3 ( it seems like I'll have to deal with it soon... ). What do you think? I appreciate every opinion from you. Thanks a lot in advance and excuse me for my english =) Carlos

Share this post


Link to post
Share on other sites
wolf    852
my opinion in a nutshell is: it is not a technical problem but a workflow problem. I mentioned more details in some of the threads here that I also have on my blog.
In essence:
1. good naming convention for your < 20 meta-materials == *.fx files .. too many pixel shaders can be too much switch overhead
2. *.fxh include file libraries with the code that is called from the *.fx files
3. shaderlib directory that holds all the *.fxh files

Share this post


Link to post
Share on other sites
coderchris    304
Iv recently had the same kind of questions, and based on research ive done and getting professional opinions on it, an ubershader is the way to go (if your targeting recent hardware). It may seem like it would be pretty slow, but if used correctly, the compiler can actually turn your "dynamic branches" into static branches which are basically for free.

Furthermore, if the branching happens uniformly across the object and dynamic branching needs to be used, it will be fast because all of the pixels/verticies being processed will all be taking the same path, which is what the GPU is really good at.

Share this post


Link to post
Share on other sites
wolf    852
Using an ubershader is a bad idea because you will get compile times of 30+ minutes for your shaders at some point with this. You do not want to let a team of graphics programmers wait everytime they change the shader for 30 minutes.

Share this post


Link to post
Share on other sites
David Neubelt    866
Quote:
Original post by wolf
Using an ubershader is a bad idea because you will get compile times of 30+ minutes for your shaders at some point with this. You do not want to let a team of graphics programmers wait everytime they change the shader for 30 minutes.


Hey Wolf,

That seems just to me a build problem - if it is efficient for the game then use what ever is best for the game. If there are 30 minute build times then that needs to be addressed in the tools. Scattering the compiles across a build farm will solve the huge recompile. Also, while iterating on a shader the build could create a secondary shader and it only recompiles the one shader to display in game. When the graphics programmer is happy they can submit the shader to the build farm to recreate all the shader permutations.

With that said - I don't disagree with your approach though.

What we're implementing is very similiar to what you've said. We create a handful of ubershaders that represent different material types. Within each ubershader you can compile in and out different options(we don't do dynamic branching). Those options are implemented as functions in shared header files so all the different materials can reuse code.

This allows us to keep shaders at a reasonable size to make it easier to optimize and maintain.

-= Dave

Share this post


Link to post
Share on other sites
wolf    852
The problem is that you create all those shaders from one source file. This takes time. Whereas with a carefully selected shader setup you only build the shader you are currently working on.
You can't speed up the creation process of the uebershader because ... yep it is only one file that holds everything.

Share this post


Link to post
Share on other sites
David Neubelt    866
Quote:
Original post by wolf
The problem is that you create all those shaders from one source file. This takes time. Whereas with a carefully selected shader setup you only build the shader you are currently working on.
You can't speed up the creation process of the uebershader because ... yep it is only one file that holds everything.


* It is a problem that is easily parallelised onto a build farm
* As I mentioned earlier when iterating on a shader source file you can have your tool use another source file that you iterate on. This will only spawn one build with one configuration (the current one the graphic programmers select). When the programmer is happy with his shader - the tool saves the source file over the original source file and spawns the parallel build that compiles all configurations

I think you should do that and on top of that have a 'carefully selected shader setup' as well.

-= Dave

Share this post


Link to post
Share on other sites
Quote:
Original post by wolf
Using an ubershader is a bad idea because you will get compile times of 30+ minutes for your shaders at some point with this. You do not want to let a team of graphics programmers wait everytime they change the shader for 30 minutes.
and
Quote:
The problem is that you create all those shaders from one source file. This takes time. Whereas with a carefully selected shader setup you only build the shader you are currently working on.
You can't speed up the creation process of the uebershader because ... yep it is only one file that holds everything.

Wolfgang, you are stating with utter certainty things that IMO are clearly wrong. David already explained to you the way of addressing the issue you mention: by using a smarter build system.

All asset processing can be sped up by (1) parallelizing/distributing the processing, and by (2) caching results (as well as temporaries) to avoid needless reprocessing. This neatly takes care of the issue you mention, and much much more!

Obviously, creating such a system is more work than just having separate .fx files for every shader than a single ubershader.fx file from which others are created, but (1) the above build system gives you pipeline performance benefits for all asset types, and (2) in your "simpler" system you have the problems of remembering to change every relevant shader file when you want to change e.g. lighting, plus you too would rebuild (nearly) all shaders if you touched something as general as lighting falloff, except you wouldn't have the benefit of the smarter build system to help you out in that case.

Plus you are missing out on other benefits that generated shaders buys you, such as the ability to fold constants at compile-time.

But what you suggest is certainly the simplest possible good solution, and simplicity is often -- but not always -- the right thing.

Share this post


Link to post
Share on other sites
andur    781
Quote:
Original post by Christer Ericson
Obviously, creating such a system is more work than just having separate .fx files for every shader than a single ubershader.fx file from which others are created, but (1) the above build system gives you pipeline performance benefits for all asset types, and (2) in your "simpler" system you have the problems of remembering to change every relevant shader file when you want to change e.g. lighting, plus you too would rebuild (nearly) all shaders if you touched something as general as lighting falloff, except you wouldn't have the benefit of the smarter build system to help you out in that case.


You include all common elements/functions from shared seperate files, so, changing the lighting falloff only necessitates recompiling the shaders that use that particular lighting model. Easy to parse fx files for the #includes. This results in the optimal compilation time, as you are: compiling only exactly what you need to (any other way of setting up your shaders would require the same or worse), and you can easily parallelize compiling multiple fx files.

Share this post


Link to post
Share on other sites
samv    312
Quote:
Original post by andur
You include all common elements/functions from shared seperate files, so, changing the lighting falloff only necessitates recompiling the shaders that use that particular lighting model. Easy to parse fx files for the #includes.

Is this only possible in a makefile-style build system? I would like to do this in the Visual Studio dependency checking system (where .fx files are processed according to a custom build rule) but can't figure how to do it.

Share this post


Link to post
Share on other sites
andur    781
Quote:
Original post by samv
Quote:
Original post by andur
You include all common elements/functions from shared seperate files, so, changing the lighting falloff only necessitates recompiling the shaders that use that particular lighting model. Easy to parse fx files for the #includes.

Is this only possible in a makefile-style build system? I would like to do this in the Visual Studio dependency checking system (where .fx files are processed according to a custom build rule) but can't figure how to do it.


I don't think that there's any way of doing it with Visual Studio's dependency checking system, at least, not without installing/creating some 3rd party plugin.

What I did, was, I wrote a little ShaderCompiler tool that runs as a pre-build step in my project and scans through the .fx files, and seeing which need to be recompiled and what depends on those files. Then it fires up a bunch of fxc processes and compiles them in parallel. It was only about a 100 lines of C# code to write such an app (I can't share it with you, as I wrote it at work). Typically, takes only a few seconds to make the necessary changes, and virtually no time if you didn't change any .fx files.

Oh, and as a little tip, if you do this, redirect the stdout of the fxc processes that you spawn, so that they show up in Visual Studio's output window. Then you can see what its doing, and any build errors will show up in the error list and be clickable to jump to the problematic lines/files.

Share this post


Link to post
Share on other sites
samv    312
Quote:
Original post by andur
I don't think that there's any way of doing it with Visual Studio's dependency checking system, at least, not without installing/creating some 3rd party plugin.

Do you know of any particular 3rd party plugin that could help?

Quote:

What I did, was, I wrote a little ShaderCompiler tool that runs as a pre-build step [...]

I see, thanks for the advice. I just hoped I could use the "custom build rules" mechanism of VS as it is very nice to use. Unfortunately it seems it is a bit underpowered.

A different approach seems to be: run a VS macro in a pre-build step and use VCLinkerTool.AdditionalDependencies to add the dependencies dynamically. I haven't tried this though.

[Edited by - samv on November 4, 2008 3:45:38 PM]

Share this post


Link to post
Share on other sites
andur    781
Quote:
Original post by samv
Quote:
Original post by andur
I don't think that there's any way of doing it with Visual Studio's dependency checking system, at least, not without installing/creating some 3rd party plugin.

Do you know of any particular 3rd party plugin that could help?

Quote:

What I did, was, I wrote a little ShaderCompiler tool that runs as a pre-build step [...]

I see, thanks for the advice. I just hoped I could use the "custom build rules" mechanism of VS as it is very nice to use. Unfortunately it seems it is a bit underpowered.

A different approach seems to be: run a VS macro in a pre-build step and use VCLinkerTool.AdditionalDependencies to add the dependencies dynamically. I haven't tried this though.


I don't know of any available plugins that do this automatically. I know that there are a few that do hlsl syntax highlighting/intellisense, but, I've never used any of those (I just tell visual studio to use the c++ editor for .fx files, which is fine for my needs). I seem to recall seeing one of these highlighted in one of the Weekend Reading journal entries here on GameDev.

I've never tried to use a VS macro for anything, so, I'm afraid I won't be of much help there.

The seperate program has a couple of other advantages. It can be run outside of visual studio, it could be set up to be called by your game engine to compile shaders on the fly, and if you aren't tied to the visual studio build button you can compile your shaders without compiling your game/engine as well (though you could probably set up a dummy build project/configuration to only do shaders).

Share this post


Link to post
Share on other sites
wolf    852
Hey Christer,
Quote:
(2) in your "simpler" system you have the problems of remembering to change every relevant shader file when you want to change e.g. lighting
No just a function in a header file.

I hope the ubershader approach will go well for you. Looking forward to your post-mortem regarding it at some point on a PS3 conference :-)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this