shader system implementation
Author
Hi
I have read some threads about how to implement a decent shader system
especially this one
they suggest a method using dlls to keep the system flexible
they export a base class and implement it in the dll
I wonder how I could handle several shader dlls with implicit linking(without loadlibrary("xxx.dll")
From what I understand this isn t possible that way since it would always choose the dll it finds first so you probably don t get the right one
one solution I could think of, actually more a hack than a solution
ENGINE:
export baseclass
SHADERMANAGER_DLL:
import baseclass
export derived class #n
getprocedure( return instance of derived class #n)
SHADER#n_DLL:
import derived class #n
implement derived class # n
this way the SHADERMANAGER_DLL would be implicitly linked and would implicitly link to the SHADER#n_DLL
the getprocedure would return a instance of a shader implemented in a dll
any idea if that works?
a more straight forward solution would be explicit linking which allows you to
choose which dll to load at runtime
what way would you suggest/propose?
i would like to allow modders to implement custom shaders
Part of the appeal of the mentioned system comes from being able to 'patch' the game with additonal shaders after release, by simply creating a new dll. This implies you need to check at startup what dlls there are and dynamically load them (with LoadLibrary on windows).
Author
Ok that seems to be the easiest and straight forward way to handle this problem
How would you process with the implementation?
give each shader class a information struction describing its capabilities
and put the render code into the shader class?
so the mapper specifies which shader implementation to use
so the engine loads a map and finds a shader that request implementation "sm3.0" for example
now the shader manager searches for the dll that implements the sm3.0 shader
then
- sort all geometry to render by shader
- get a instance of the shader implementation
- render(geometry){shaderinstance->setstates(); shaderinstance->draw(geometry);shaderinstance->unsetstates()};
proceed until everything is rendered
or would you suggest a different approach?
How would you process with the implementation?
give each shader class a information struction describing its capabilities
and put the render code into the shader class?
so the mapper specifies which shader implementation to use
so the engine loads a map and finds a shader that request implementation "sm3.0" for example
now the shader manager searches for the dll that implements the sm3.0 shader
then
- sort all geometry to render by shader
- get a instance of the shader implementation
- render(geometry){shaderinstance->setstates(); shaderinstance->draw(geometry);shaderinstance->unsetstates()};
proceed until everything is rendered
or would you suggest a different approach?
It's probably similar to what you described yes. Although I think most people who have implemented this stuff have added an extra level of indirection. The mapper does not specify which shader he wants exactly, but the resulting effect he wants to create.
At engine startup the renderer parses all the shaders and sorts them according to the effect they implement, and the quality of it. All shaders that aren't supported by the hardware can be unloaded again.
Then at rendertime, find the effect, find the associated shader (this would normally be the best one, but according to quality settings, it might be a lower quality implementation of the effect), and then call that as you described.
You'll also want to take a good look at this thread.
At engine startup the renderer parses all the shaders and sorts them according to the effect they implement, and the quality of it. All shaders that aren't supported by the hardware can be unloaded again.
Then at rendertime, find the effect, find the associated shader (this would normally be the best one, but according to quality settings, it might be a lower quality implementation of the effect), and then call that as you described.
You'll also want to take a good look at this thread.
Maybe packing shaders into DLL-s isn't very good idea.
Why should you do that, for example?
It gives freedom to the point, one can build very custom vertex declarations, and prebuild vb/ib-s, but... this is not exactly responsibility of a shader...
Shader shades geometry, whatever it is.
Packing, or even creating geometry maybe is not quite it responsibility. It is responsibility of some renderer subsystem.
Imagine this - you released a game, and want to enhance it with grass... You build a grass-shader and release its DLL... but where to put that grass? Well...
Ok, you have the grass, but it is lame, and you want to make it cool. You write a shader, and release it... Is it pracital? I'd say - no...
This is just an argument, against a system described in threads mentioned - do not waste your precious time, implementing such monsters, especially when there is no need to.
Implement:
1. Shader compiler system (based on preprocessor), that can get a shader source, and create a family of shaders from it. For example, try to encode more parameters into single shader source, like light variation, fog variation, quality variation, etc.
2. System of managing shader parameters, and uploading shader constants to the shaders. (Setting parameters each frame vs binding pointers to shader constants)
3. Effect system, that describe a group of shaders, for each situation in which an object must be rendered - for different light scenarios, fog scenarios, shadowmap, quality, renderpath, etc...
4. A really advanced system would be, some way to let your artist control/create shaders. System can vary from relatively simple shader parametric system, where for each shader written we have some set of parameters (fog: linear, height, linear+height), (lighting: diffuse+ambient, lerp( diffuse, ambient ), diffuse-only, ...), etc. Artist can just choose options from combo-boxes, then we set proper defines to the compiler and produce a shader.
More advanced system would be composing shader from primitives/blocks, like UE3 shader editor/Offset software shader editor for example.
My 2c.
Why should you do that, for example?
It gives freedom to the point, one can build very custom vertex declarations, and prebuild vb/ib-s, but... this is not exactly responsibility of a shader...
Shader shades geometry, whatever it is.
Packing, or even creating geometry maybe is not quite it responsibility. It is responsibility of some renderer subsystem.
Imagine this - you released a game, and want to enhance it with grass... You build a grass-shader and release its DLL... but where to put that grass? Well...
Ok, you have the grass, but it is lame, and you want to make it cool. You write a shader, and release it... Is it pracital? I'd say - no...
This is just an argument, against a system described in threads mentioned - do not waste your precious time, implementing such monsters, especially when there is no need to.
Implement:
1. Shader compiler system (based on preprocessor), that can get a shader source, and create a family of shaders from it. For example, try to encode more parameters into single shader source, like light variation, fog variation, quality variation, etc.
2. System of managing shader parameters, and uploading shader constants to the shaders. (Setting parameters each frame vs binding pointers to shader constants)
3. Effect system, that describe a group of shaders, for each situation in which an object must be rendered - for different light scenarios, fog scenarios, shadowmap, quality, renderpath, etc...
4. A really advanced system would be, some way to let your artist control/create shaders. System can vary from relatively simple shader parametric system, where for each shader written we have some set of parameters (fog: linear, height, linear+height), (lighting: diffuse+ambient, lerp( diffuse, ambient ), diffuse-only, ...), etc. Artist can just choose options from combo-boxes, then we set proper defines to the compiler and produce a shader.
More advanced system would be composing shader from primitives/blocks, like UE3 shader editor/Offset software shader editor for example.
My 2c.
Hi,
Its not a wise idea to implement effects as DLLs. Effect is just another type of resource. Doing this is like trying to load textures as DLLs.
If using DX, try the D3DXFramework. It can compile the effects at runtime and if you create a effect manager, then you can load and unload your effects at will and setup them with some simple params.
Luck!
Guimo
Its not a wise idea to implement effects as DLLs. Effect is just another type of resource. Doing this is like trying to load textures as DLLs.
If using DX, try the D3DXFramework. It can compile the effects at runtime and if you create a effect manager, then you can load and unload your effects at will and setup them with some simple params.
Luck!
Guimo
I think what you're looking for is an effect, not shader, system. Basically, something that will define:
1)How many rendering passes will there be
2)The initial states(matrices,lights,textures,blending modes...) for each pass
3)The shaders that the pass will use
IMO, if you're after easy modding, it's very risky to require modders to compile a dll that will be linked with the main .exe. They have to be very careful to get it right. Different compilers or settings(aligment,padding,etc...) may result to big problems.
I think the best solution by far is to use some sort of scripting. Then the game will be modded by just editing a text file. If you're using D3D, then the D3DXFramework is just fine. Otherwise, you can use Lua or SmallTalk or something like that(even boost::spirit?), or even implement a scripting system yourself. It doesn't have to be anything fancy, the most complex thing you need to do is set the states. You can probably pull it off with some sscanf() commands.
1)How many rendering passes will there be
2)The initial states(matrices,lights,textures,blending modes...) for each pass
3)The shaders that the pass will use
IMO, if you're after easy modding, it's very risky to require modders to compile a dll that will be linked with the main .exe. They have to be very careful to get it right. Different compilers or settings(aligment,padding,etc...) may result to big problems.
I think the best solution by far is to use some sort of scripting. Then the game will be modded by just editing a text file. If you're using D3D, then the D3DXFramework is just fine. Otherwise, you can use Lua or SmallTalk or something like that(even boost::spirit?), or even implement a scripting system yourself. It doesn't have to be anything fancy, the most complex thing you need to do is set the states. You can probably pull it off with some sscanf() commands.
Author
I think you missunderstood the concept behind this implementation
the shader dll doesn t implement the effects you want to implement it implements the functionality you offer for shaders
e.g.:
a grass shader with (normal grass, boring grass, animated grass)
lets say you work with an older graphics card and implement these shaders mentioned above
they all provide some properties used to render them
now you switched to a modern graphic card and you want to update the graphics quality you just release a new dll with a higher priority to provide the functionality used by the shaders, just in a modern manner maybe with enhanced features
the shader dll itself runs a test on you system to check if everything is available otherwise the system chooses the old dll
this allows you to specialize a generalized representation of a effect for certain graphic cards without rewriting and updating a ton of code
the shader dll itself just does the state processing
the effect properties are still seperated in a effects/shader file
and i don t use D3D and i would like to stay independent from APIs so the effects framework of D3D is no option in my eyes
the shader dll doesn t implement the effects you want to implement it implements the functionality you offer for shaders
e.g.:
a grass shader with (normal grass, boring grass, animated grass)
lets say you work with an older graphics card and implement these shaders mentioned above
they all provide some properties used to render them
now you switched to a modern graphic card and you want to update the graphics quality you just release a new dll with a higher priority to provide the functionality used by the shaders, just in a modern manner maybe with enhanced features
the shader dll itself runs a test on you system to check if everything is available otherwise the system chooses the old dll
this allows you to specialize a generalized representation of a effect for certain graphic cards without rewriting and updating a ton of code
the shader dll itself just does the state processing
the effect properties are still seperated in a effects/shader file
and i don t use D3D and i would like to stay independent from APIs so the effects framework of D3D is no option in my eyes
Quote:Original post by mikeman
I think what you're looking for is an effect, not shader, system. Basically, something that will define:
1)How many rendering passes will there be
2)The initial states(matrices,lights,textures,blending modes...) for each pass
3)The shaders that the pass will use
IMO, if you're after easy modding, it's very risky to require modders to compile a dll that will be linked with the main .exe. They have to be very careful to get it right. Different compilers or settings(aligment,padding,etc...) may result to big problems.
I think the best solution by far is to use some sort of scripting. Then the game will be modded by just editing a text file. If you're using D3D, then the D3DXFramework is just fine. Otherwise, you can use Lua or SmallTalk or something like that(even boost::spirit?), or even implement a scripting system yourself. It doesn't have to be anything fancy, the most complex thing you need to do is set the states. You can probably pull it off with some sscanf() commands.
I disagree with your definition of an effect. Personally, to me an effect has absolutely nothing to do with how the effect is implemented (so, render passes and so on). All the effect does is provide resources to implement the effect - so for instance, it says "You have these streams and uniforms available to create this effect", some form of shader (I'm not talking about GLSL or co) would then implement the effect however it likes.
I personally went with the scripting approach, whereby I have an effect file - which says what effects there are (with a little description of what that effect should do), and what resources are available to implement that effect. I then have a series of shader files that have a number of profiles ( to support different types of hardware ), with different compliances with how well it supports the said effect. Each profile can then implement the effect in as many passes or using whatever tech it needs. At load time the effects are resolved to a profile in a shader based on what the hardware supports such that the profile with the highest supported compliance is used. Multiple shaders can support the same effect - but the one with the highest compliance will always be used, so patching using a new shader is easy.
There are disadvantages to using a script based system - one big one being you can't really do arbitary processing.
Although the DLL shader approach works really well in practice, and is very robust and extensible, I was always thinking of ways to get rid of it someday. It's complex, platform dependent (you need to distribute your shaders specifically compiled for each platform), and doesn't really allow for rapid prototyping and visual shader development as easily as I would like to.
Unfortunately, no alternative (especially primitive script or shader program based ones, like the DX effect framework) offer the extreme flexibility and scalability of the DLL based effect-shader system. So, currently we're still using it extensively. Our current system uses over 60 shader DLLs, with more than 300 shaders. And still, it is lightning fast.
But the future lies in a different approach: JIT compiled shader meta languages. Right now, the resources needed to render a specific effect through a shader are almost always explicitely defined. This has to change: the shader compiler should extract, optimize, and request all needed resources on his own. Things like the number of passes shouldn't even appear in the code anymore: a render pass, a shader program or an attribute are just resources, much like a CPU register or opcode. An appropriate compiler must automatically recognize and optimize the use of such resources, and request them from the render manager on demand.
Approaches like the Sh meta language go into the right direction, but are still not flexible enough.
Unfortunately, no alternative (especially primitive script or shader program based ones, like the DX effect framework) offer the extreme flexibility and scalability of the DLL based effect-shader system. So, currently we're still using it extensively. Our current system uses over 60 shader DLLs, with more than 300 shaders. And still, it is lightning fast.
But the future lies in a different approach: JIT compiled shader meta languages. Right now, the resources needed to render a specific effect through a shader are almost always explicitely defined. This has to change: the shader compiler should extract, optimize, and request all needed resources on his own. Things like the number of passes shouldn't even appear in the code anymore: a render pass, a shader program or an attribute are just resources, much like a CPU register or opcode. An appropriate compiler must automatically recognize and optimize the use of such resources, and request them from the render manager on demand.
Approaches like the Sh meta language go into the right direction, but are still not flexible enough.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
