Jump to content

  • Log In with Google      Sign In   
  • Create Account

Yet another shader generation approach


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
3 replies to this topic

#1 AvengerDr   Members   -  Reputation: 726

Like
3Likes
Like

Posted 25 October 2013 - 08:27 AM

In the past few months I found myself juggling between different projects  aimed at several different platforms (Windows 7,8, RT and Phone). Some of those have different capabilities so some shaders needed to be modified in order to work correctly. I know that premature optimization is just as bad but in this specific situation I thought that addressing the problem sooner than later would be the right choice.
 
To address this problem I created a little tool that allows me to dynamically generate a set of shaders through a graph like structure. Which is nothing new, as it is usually the basis for this kind of application. I did probably reinvent a lot of wheels but since I couldn't use MS's shader designer (it only works with C++ I think) nor Unity's equivalent (as I have my own puny little engine) I decided to roll my own. I am writing here to get some feedback on my architecture and if there is something I overlooked.
 
Basically I have defined classes for most of the HLSL language. Then there are nodes such as constants, math operations and special function nodes. The latter are the most important ones as they correspond to high-level functions such as Phong Lighting, shadow algorithms and so on. Each of these function nodes expose several switches that enable me to enable/disable specific features. For example if I set a Phong node's "Shadows" to true then it will generate a different signature for the function than if it had it set to false. Once the structure is complete the graph is traversed and the actual shader code is generated line by line. From my understanding I think that dynamic shader linking works similarly but I've not been able to find a lot of information on the subject.
 
Right now shaders can only be defined in code, in the future I could build a graphical engine. A classic phong lighting pixel shader looks like this and this is the generated output. It is also possible to configure the amount of "verbosity". The interesting thing is that once the shader is compiled it gets serialized to a custom format that contains other information. Variables and nodes that are part of the shader are decorated with engine references. If I add a reference to the camera position for example, that variable tells the engine that it has to look for that value when initialising the shader. Analogously for the values needed to assemble constant buffers (like world/view/projection matrices). 

Once the shader is serialised, all this metadata helps the engine to automatically assign each shader variable or cbuffer with the right values. Before in my engine, each shader class had these huge parts of code that fetched needed values from somewhere else in the engine. Now all that has been deleted and is taken care automatically as long as the shaders are loaded in this format.
 
ShaderGenV01.png

Another neat feature is that within the tool I have built I can define different techniques; i.e: a regular Phong shader, one using a Diffuse Map, one using a Shadow Map. Each technique maps a different combination of vertex and pixel shaders. The decoration that I mentioned earlier helps the tool generate a "TechniqueKey" for each shader that is then used by the engine to fetch the right shader from the file on disk. For example the PhongDiffusePS shader is decorated with attributes defining its use of a DiffuseMap (among other things). When in the actual application I enable the DiffuseMap feature, the shader system checks whether that feature is supported by the current set of shaders assigned the material. If a suitable technique is found, then the systeme enables the relevant parameters. In this way it is also possible to check for different feature levels and so on.

Probably something like this is overkill for a lot of small projects and I reckon it is not as easy to fix something in the generated code of this tool as it is when making changes in the actual source code it self. But once it does finally work, the the automatic configuration of shader variables is something that I like (at least if compared to my previous implementation, I don't how everyone else handles that). What I am asking is how extensible or expandable this approach is (maybe it is too late to ask this kind of questions biggrin.png)? Right now I have a handful of shaders defined in the system. If you had a look at the code, what kind of problems am I likely to run into when adding nodes to deal with Geometry shaders and other advanced features?

Finally, if anyone could be interested in having a look at the tool I'm happy to share it on GitHub.

Edited by AvengerDr, 25 October 2013 - 08:30 AM.

--Avengers UTD Chronicles - My game development blog

Sponsor:

#2 N.I.B.   Members   -  Reputation: 1194

Like
0Likes
Like

Posted 25 October 2013 - 08:47 AM

This looks cool.

Can you share it?



#3 AvengerDr   Members   -  Reputation: 726

Like
0Likes
Like

Posted 25 October 2013 - 01:45 PM

Sure, it is available here. You only need the librariers named "Odyssey.Common" and "Odyssey.Content". The rest is part of another project, a 2D UI library. The tool itself is the project named ShaderGenerator.

 

In order to build it you will probably have to change the SharpDX SDK line in the Build/ ... targets file to the location you have installed it. The two above projects should be compiled using the Win8Debug configuration. It should work on W7 too. The shaders themselves are defined in the ShaderGenerator/Shaders/Techniques subfolder. The Data/ResourceManager.cs class defines which shaders are loaded at startup. Though saving and loading shader graphs should work. You will need the ColorCode and Galasoft.MvvmLight nuget packages installed.

 

Let me know if you are able to run it. In order to compile shaders every item listed must be assigned to a technique. At the moment these are hardcoded but I am in the process of adding a new window that will allow defining new techniques.

 

Each technique can be decorated in code by Vertex/PixelShaderAttribute(s). These attributes are used when validating nodes. All attributes defined will be used to validate nodes that have the same attribute attached. For example the PhongPS shader does not have the PixelShaderAttribute for DiffuseMap. Thus when validating the PhongLightingNode, the DiffuseMapSampler node won't be checked and so on.

 

Clearly in order to use the automatic configuration of shader variables you would need to parse the metadata in the shader collection file from your own engine. I can post some examples if needed.


Edited by AvengerDr, 25 October 2013 - 04:59 PM.

--Avengers UTD Chronicles - My game development blog

#4 AvengerDr   Members   -  Reputation: 726

Like
0Likes
Like

Posted 11 November 2013 - 07:00 PM

Hi all, I've added support to regular HLSL source code files. Now it can act as a graphical frontend for offline shader compilation. Just press the "import" button and select .hlsl source code files. You can find a link to an installer from here. In the post you'll find also more detailed information regarding the shader graph structure, which now supports Yaml (an example is included in the installer).


--Avengers UTD Chronicles - My game development blog




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS