Sign in to follow this  
Anfaenger

Designing efficient Material/Shader system

Recommended Posts

I'm having trouble designing a new material system for my engine.

My primary goals are:

0) Minimal CPU overhead and memory footprint

1) I'd like to avoid setting shader parameters (uniforms) one by one
(as in 'shader scripts' approach, 'DirectX effects - style': query location by variable name, need to store metadata about uniforms, etc.).
but that means creating custom structs with the matching layout and 'hardcoding' shaders which is bad (?).

2) Improve efficiency via multithreading (issue graphics API calls in a separate 'render' thread, render commands buffering).
I intend on using two command buffers, there'll be a special UpdateConstantBuffer command,
and i'll need to copy entire constant/uniform buffer contents (each shader parameter) into the command buffer (?).
Seems like a lot of data to memcpy() around...

What are your ideas for designing a powerful, flexible and efficient graphics material system ?

Currently, i have a C++ wrapper class for each shader (generated with a tool, map CB, cast to struct, set params, unmap -> no metadata needed),
and several material classes (Phong, Plain, SSS) which i find cumbersome to use.
i'd rather have one material class, but then i don't know how to set proper shaders,
because material will be generic and shaders are hardcoded.

Share this post


Link to post
Share on other sites
Thank you, L. Spiro, for your reply, this is an excellent solution!


Though it surely takes a very long time to implement properly.



1. Do you also perform sematic checking (otherwise, when you translate into HLSL -> line numbers differ -> error messages useless) in your translator?


2. What parser generator did you use (or rolled your own hand-written recursive decent) ?

and i'm sure, there's a plethora of optimizations that can be done for generating the most efficient C++ wrapper code.


(i decided to use Direct3D 11 deferred contexts instead of writing a command buffer class,
so my low-level wrappers will work both with the immediate context and deferred contexts on other threads.)



3. Is it fine to create a separate constant buffer for each shader class if needed?
e.g. i have a few global, shared CBs, and i create new CBs for each shader (if it declares its own CB(s)).

4. Is it better to use a universal, generic material class instead of separate CWaterMaterial, CTerrainMaterial, CSkinMaterial ?
And how should material system talk to the low-level shader system? Through hardcoded shader variable semantics (e.g. WORLD_MTX, VIEW_POS, DIFFUSE_COLOR) ?

Share this post


Link to post
Share on other sites
[quote name='Anfaenger' timestamp='1343985785' post='4965793']
1. Do you also perform sematic checking (otherwise, when you translate into HLSL -> line numbers differ -> error messages useless) in your translator?
[/quote]
Somewhat. I let the HLSL or GLSL compilers catch things such as undeclared identifiers etc. I only add my own checks when they are related to my own shader-language rules.
You don’t need to worry about mismatching lines in error print-outs. Use #line before every statement, if/else/for/while/do/etc.
You can see actual translated code in the first post of [url="http://www.gamedev.net/topic/628661-directx-11-sudden-saturday-shadow-sadness-syndrome/"]this[/url] topic. Click the Spoiler button.
My language also eliminates unreferenced functions and constants, or else my constant buffers would be huge. Eliminated functions is the reason for so many redundant #line directives.

You should be aware that you also must perform post-processing manually and so must also write your own preprocessor.
Think about this case:
[CODE]
#define DEF_CONST( NUMBER ) Texture2D g_sTex ## NUMBER:register(t ## NUMBER)
DEF_CONST( 0 );
DEF_CONST( 7 );
[/CODE]

Your language needs to know that g_sTex0 and g_sTex7 exist, and the only way to do that is to perform preprocessing. I use a second parser for that. Don’t even think of trying to do this during the real parsing stage.


[quote name='Anfaenger' timestamp='1343985785' post='4965793']
2. What parser generator did you use (or rolled your own hand-written recursive decent) ?
[/quote]
Flex and Bison.


[quote name='Anfaenger' timestamp='1343985785' post='4965793']
3. Is it fine to create a separate constant buffer for each shader class if needed?
e.g. i have a few global, shared CBs, and i create new CBs for each shader (if it declares its own CB(s)).
[/quote]
If by “class” you mean “instance” then it is one way to go. It will not kill your performance but it uses more memory and is not the absolute most efficient system there is. Obviously that would be for them to share buffers.


[quote name='Anfaenger' timestamp='1343985785' post='4965793']
4. Is it better to use a universal, generic material class instead of separate CWaterMaterial, CTerrainMaterial, CSkinMaterial ?
And how should material system talk to the low-level shader system? Through hardcoded shader variable semantics (e.g. WORLD_MTX, VIEW_POS, DIFFUSE_COLOR) ?
[/quote]
Yes. The shader/material systems should not know what those things are. A water object should just use the generic shader system to make shaders for water.
The terrain should use the generic shader system to make terrain shaders.

My engine communicates with shaders using a set of predefined semantics. They allow the engine to update uniforms/constants automatically.
For user-defined uniforms/constants, users can get an engine handle to anything by name and then use that handle to update them. Internally, this is how the engine updates the semantic values. The parser keeps track of what globals are created with what semantics and later uses the names of those constants to get handles to all the known semantic types. The fact that the engine updates them automatically is just an extension of this.



If you are thinking about making a language, a shader language is not a terrible way to start because they are fairly simple.
But be aware that making even simple languages requires a great deal of experience in order to have a nice class structure and to be stable/leak-free.
As one example, when you parse the syntax tree in order to generate HLSL output, you can’t use recursion or you will risk stack overflow in large complicated shaders. So you have to have experience working with explicit stacks.
Making an entire shader system revolve around this is a large and risky undertaking. If one thing doesn’t work, the whole system does not work.
Just be aware.


L. Spiro

Share this post


Link to post
Share on other sites
Great, this clears up a lot.

For fast iteration, a system for reloading shaders on-the-fly is a 'must-have'.
But only affected files should be recompiled.

1. Do you track which files get #included into source files and how?
Do you keep some sort of a project file with compile timestamps ?

2. What if some shader doesn't have a particular semantic?
i'd like to avoid checks for invalid handles before making every UpdateShaderConstantByHandle() call.

3. Where are material parameters (e.g. colors, texture layers, etc.) stored?
i assume there're no such inefficient solutions as 'map<string,handle>'.

Share this post


Link to post
Share on other sites
[quote name='Anfaenger' timestamp='1343997106' post='4965823']
1. Do you track which files get #included into source files and how?
Do you keep some sort of a project file with compile timestamps ?[/quote]My game doesn't compile shaders, an external program compiles all the data files for the game. Yep, this external file maintains a 'project' file of all data that will end up shipping with the game, and maintains timestamps as part of the build process ([i]and dependency graphs, e.g. for #includes[/i]). It also listens to OS file-system events in the export directories, so when a file is modified the OS tells the build-system, which highlights the 'rebuild' button in the UI. When the user clicks 'rebuild', it re-runs the build process ([i]which checks the timestamps/dependencies and and rebuilds the modified files[/i]), and if the game is running, it sends a message to the game with the names of the modified output files so the game can re-load them.
[quote name='Anfaenger' timestamp='1343997106' post='4965823']3. Where are material parameters (e.g. colors, texture layers, etc.) stored?[/quote]In an array of bytes, which correspond to a particular shader cbuffer/UBO layout.[quote name='Anfaenger' timestamp='1343907326' post='4965488']
1) I'd like to avoid setting shader parameters (uniforms) one by one[/quote]Make sure that when you're writing you shaders, you group uniforms together into cbuffers/UBOs based on who is supposed to set them. E.g. one buffer for camera parameters, one for transform data, one for data set by the artists that will never change at runtime, etc... Edited by Hodgman

Share this post


Link to post
Share on other sites
1. How do you update shader constants?
- at runtime, keep shader metadata (info about used variables: name, type, offset, dimension);
- get parameter handle by string name or hardcoded semantic;
- if the handle is valid:
- - - SetParamByHandle() (copy float, matrix, int, bool into the parameter buffer);
- - - see if UniformBuffer::isDirty flag is 'true' and update the corresponding constant buffer;
?

sounds a bit messy and suboptimal, imho.

2. How do you implement support for multiple passes?

3. Could you please sketch your material class in pseudocode?

i guess, materials should only contain data declarations, no actual rendering code.

Share this post


Link to post
Share on other sites
Thanks, Hodgman, this info is super useful!

1. Where do you keep 'shader resources' (such as pointers to material textures) and how do you bind them to the pipeline?

2. Do your shaders carry additional information about allowed render passes, vertex formats, etc. ?
(e.g. shader for filling g-buffer should only be used during SCENE_PASS_FILL_GBUFFER.)

casting to a native C struct is a better solution than hardcoded semantics, imho.
i still can't imagine having no material class (dunno how i would reference materials in assets and set material properties).

Share this post


Link to post
Share on other sites
[quote name='Anfaenger' timestamp='1344084647' post='4966088']
i still can't imagine having no material class (dunno how i would reference materials in assets and set material properties).
[/quote]

I think the idea is that you don't set material properties from within the game per se. Is sounds like the renderer has no idea what artist-defined parameters it's setting, it just loads a pre-processed cbuffer blob and plugs it into the shader. Likely I'm simplifying :)

Interesting stuff, Hodgman.

Share this post


Link to post
Share on other sites
Thanks for the answers, hopefully, my next material system will be much better than a nasty combination of inheritance abuse and multitude of .fx files.

i've decided to bite the bullet and write my own shader language parser/translator.
it will also be used for parsing resource declarations (render targets, sampler,depth-stencil,rasterizer and blend states, state blocks and vertex formats)
and generating most of boilerplate code (especially D3D11-style verbose initialization stuff, description structs should be filled in automatically).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this