Jump to content
  • Advertisement
Sign in to follow this  
Anfaenger

Designing efficient Material/Shader system

This topic is 2173 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm having trouble designing a new material system for my engine.

My primary goals are:

0) Minimal CPU overhead and memory footprint

1) I'd like to avoid setting shader parameters (uniforms) one by one
(as in 'shader scripts' approach, 'DirectX effects - style': query location by variable name, need to store metadata about uniforms, etc.).
but that means creating custom structs with the matching layout and 'hardcoding' shaders which is bad (?).

2) Improve efficiency via multithreading (issue graphics API calls in a separate 'render' thread, render commands buffering).
I intend on using two command buffers, there'll be a special UpdateConstantBuffer command,
and i'll need to copy entire constant/uniform buffer contents (each shader parameter) into the command buffer (?).
Seems like a lot of data to memcpy() around...

What are your ideas for designing a powerful, flexible and efficient graphics material system ?

Currently, i have a C++ wrapper class for each shader (generated with a tool, map CB, cast to struct, set params, unmap -> no metadata needed),
and several material classes (Phong, Plain, SSS) which i find cumbersome to use.
i'd rather have one material class, but then i don't know how to set proper shaders,
because material will be generic and shaders are hardcoded.

Share this post


Link to post
Share on other sites
Advertisement
Thank you, L. Spiro, for your reply, this is an excellent solution!


Though it surely takes a very long time to implement properly.



1. Do you also perform sematic checking (otherwise, when you translate into HLSL -> line numbers differ -> error messages useless) in your translator?


2. What parser generator did you use (or rolled your own hand-written recursive decent) ?

and i'm sure, there's a plethora of optimizations that can be done for generating the most efficient C++ wrapper code.


(i decided to use Direct3D 11 deferred contexts instead of writing a command buffer class,
so my low-level wrappers will work both with the immediate context and deferred contexts on other threads.)



3. Is it fine to create a separate constant buffer for each shader class if needed?
e.g. i have a few global, shared CBs, and i create new CBs for each shader (if it declares its own CB(s)).

4. Is it better to use a universal, generic material class instead of separate CWaterMaterial, CTerrainMaterial, CSkinMaterial ?
And how should material system talk to the low-level shader system? Through hardcoded shader variable semantics (e.g. WORLD_MTX, VIEW_POS, DIFFUSE_COLOR) ?

Share this post


Link to post
Share on other sites

1. Do you also perform sematic checking (otherwise, when you translate into HLSL -> line numbers differ -> error messages useless) in your translator?

Somewhat. I let the HLSL or GLSL compilers catch things such as undeclared identifiers etc. I only add my own checks when they are related to my own shader-language rules.
You don’t need to worry about mismatching lines in error print-outs. Use #line before every statement, if/else/for/while/do/etc.
You can see actual translated code in the first post of this topic. Click the Spoiler button.
My language also eliminates unreferenced functions and constants, or else my constant buffers would be huge. Eliminated functions is the reason for so many redundant #line directives.

You should be aware that you also must perform post-processing manually and so must also write your own preprocessor.
Think about this case:

#define DEF_CONST( NUMBER ) Texture2D g_sTex ## NUMBER:register(t ## NUMBER)
DEF_CONST( 0 );
DEF_CONST( 7 );


Your language needs to know that g_sTex0 and g_sTex7 exist, and the only way to do that is to perform preprocessing. I use a second parser for that. Don’t even think of trying to do this during the real parsing stage.



2. What parser generator did you use (or rolled your own hand-written recursive decent) ?

Flex and Bison.



3. Is it fine to create a separate constant buffer for each shader class if needed?
e.g. i have a few global, shared CBs, and i create new CBs for each shader (if it declares its own CB(s)).

If by “class” you mean “instance” then it is one way to go. It will not kill your performance but it uses more memory and is not the absolute most efficient system there is. Obviously that would be for them to share buffers.



4. Is it better to use a universal, generic material class instead of separate CWaterMaterial, CTerrainMaterial, CSkinMaterial ?
And how should material system talk to the low-level shader system? Through hardcoded shader variable semantics (e.g. WORLD_MTX, VIEW_POS, DIFFUSE_COLOR) ?

Yes. The shader/material systems should not know what those things are. A water object should just use the generic shader system to make shaders for water.
The terrain should use the generic shader system to make terrain shaders.

My engine communicates with shaders using a set of predefined semantics. They allow the engine to update uniforms/constants automatically.
For user-defined uniforms/constants, users can get an engine handle to anything by name and then use that handle to update them. Internally, this is how the engine updates the semantic values. The parser keeps track of what globals are created with what semantics and later uses the names of those constants to get handles to all the known semantic types. The fact that the engine updates them automatically is just an extension of this.



If you are thinking about making a language, a shader language is not a terrible way to start because they are fairly simple.
But be aware that making even simple languages requires a great deal of experience in order to have a nice class structure and to be stable/leak-free.
As one example, when you parse the syntax tree in order to generate HLSL output, you can’t use recursion or you will risk stack overflow in large complicated shaders. So you have to have experience working with explicit stacks.
Making an entire shader system revolve around this is a large and risky undertaking. If one thing doesn’t work, the whole system does not work.
Just be aware.


L. Spiro

Share this post


Link to post
Share on other sites
Great, this clears up a lot.

For fast iteration, a system for reloading shaders on-the-fly is a 'must-have'.
But only affected files should be recompiled.

1. Do you track which files get #included into source files and how?
Do you keep some sort of a project file with compile timestamps ?

2. What if some shader doesn't have a particular semantic?
i'd like to avoid checks for invalid handles before making every UpdateShaderConstantByHandle() call.

3. Where are material parameters (e.g. colors, texture layers, etc.) stored?
i assume there're no such inefficient solutions as 'map<string,handle>'.

Share this post


Link to post
Share on other sites

1. Do you track which files get #included into source files and how?
Do you keep some sort of a project file with compile timestamps ?
My game doesn't compile shaders, an external program compiles all the data files for the game. Yep, this external file maintains a 'project' file of all data that will end up shipping with the game, and maintains timestamps as part of the build process (and dependency graphs, e.g. for #includes). It also listens to OS file-system events in the export directories, so when a file is modified the OS tells the build-system, which highlights the 'rebuild' button in the UI. When the user clicks 'rebuild', it re-runs the build process (which checks the timestamps/dependencies and and rebuilds the modified files), and if the game is running, it sends a message to the game with the names of the modified output files so the game can re-load them.
3. Where are material parameters (e.g. colors, texture layers, etc.) stored?
In an array of bytes, which correspond to a particular shader cbuffer/UBO layout.

1) I'd like to avoid setting shader parameters (uniforms) one by one
Make sure that when you're writing you shaders, you group uniforms together into cbuffers/UBOs based on who is supposed to set them. E.g. one buffer for camera parameters, one for transform data, one for data set by the artists that will never change at runtime, etc... Edited by Hodgman

Share this post


Link to post
Share on other sites
1. How do you update shader constants?
- at runtime, keep shader metadata (info about used variables: name, type, offset, dimension);
- get parameter handle by string name or hardcoded semantic;
- if the handle is valid:
- - - SetParamByHandle() (copy float, matrix, int, bool into the parameter buffer);
- - - see if UniformBuffer::isDirty flag is 'true' and update the corresponding constant buffer;
?

sounds a bit messy and suboptimal, imho.

2. How do you implement support for multiple passes?

3. Could you please sketch your material class in pseudocode?

i guess, materials should only contain data declarations, no actual rendering code.

Share this post


Link to post
Share on other sites
Thanks, Hodgman, this info is super useful!

1. Where do you keep 'shader resources' (such as pointers to material textures) and how do you bind them to the pipeline?

2. Do your shaders carry additional information about allowed render passes, vertex formats, etc. ?
(e.g. shader for filling g-buffer should only be used during SCENE_PASS_FILL_GBUFFER.)

casting to a native C struct is a better solution than hardcoded semantics, imho.
i still can't imagine having no material class (dunno how i would reference materials in assets and set material properties).

Share this post


Link to post
Share on other sites

i still can't imagine having no material class (dunno how i would reference materials in assets and set material properties).


I think the idea is that you don't set material properties from within the game per se. Is sounds like the renderer has no idea what artist-defined parameters it's setting, it just loads a pre-processed cbuffer blob and plugs it into the shader. Likely I'm simplifying :)

Interesting stuff, Hodgman.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!