Sign in to follow this  
AvengerDr

Unity Yet another shader generation approach

Recommended Posts

In the past few months I found myself juggling between different projects  aimed at several different platforms (Windows 7,8, RT and Phone). Some of those have different capabilities so some shaders needed to be modified in order to work correctly. I know that premature optimization is just as bad but in this specific situation I thought that addressing the problem sooner than later would be the right choice.
 
To address this problem I created a little tool that allows me to dynamically generate a set of shaders through a graph like structure. Which is nothing new, as it is usually the basis for this kind of application. I did probably reinvent a lot of wheels but since I couldn't use MS's shader designer (it only works with C++ I think) nor Unity's equivalent (as I have my own puny little engine) I decided to roll my own. I am writing here to get some feedback on my architecture and if there is something I overlooked.
 
Basically I have defined classes for most of the HLSL language. Then there are nodes such as constants, math operations and special function nodes. The latter are the most important ones as they correspond to high-level functions such as Phong Lighting, shadow algorithms and so on. Each of these function nodes expose several switches that enable me to enable/disable specific features. For example if I set a Phong node's "Shadows" to true then it will generate a different signature for the function than if it had it set to false. Once the structure is complete the graph is traversed and the actual shader code is generated line by line. From my understanding I think that dynamic shader linking works similarly but I've not been able to find a lot of information on the subject.
 
Right now shaders can only be defined in code, in the future I could build a graphical engine. A classic phong lighting pixel shader looks like this and this is the generated output. It is also possible to configure the amount of "verbosity". The interesting thing is that once the shader is compiled it gets serialized to a custom format that contains other information. Variables and nodes that are part of the shader are decorated with engine references. If I add a reference to the camera position for example, that variable tells the engine that it has to look for that value when initialising the shader. Analogously for the values needed to assemble constant buffers (like world/view/projection matrices). 

Once the shader is serialised, all this metadata helps the engine to automatically assign each shader variable or cbuffer with the right values. Before in my engine, each shader class had these huge parts of code that fetched needed values from somewhere else in the engine. Now all that has been deleted and is taken care automatically as long as the shaders are loaded in this format.
 
[attachment=18541:ShaderGenV01.png]

Another neat feature is that within the tool I have built I can define different techniques; i.e: a regular Phong shader, one using a Diffuse Map, one using a Shadow Map. Each technique maps a different combination of vertex and pixel shaders. The decoration that I mentioned earlier helps the tool generate a "TechniqueKey" for each shader that is then used by the engine to fetch the right shader from the file on disk. For example the PhongDiffusePS shader is decorated with attributes defining its use of a DiffuseMap (among other things). When in the actual application I enable the DiffuseMap feature, the shader system checks whether that feature is supported by the current set of shaders assigned the material. If a suitable technique is found, then the systeme enables the relevant parameters. In this way it is also possible to check for different feature levels and so on.

Probably something like this is overkill for a lot of small projects and I reckon it is not as easy to fix something in the generated code of this tool as it is when making changes in the actual source code it self. But once it does finally work, the the automatic configuration of shader variables is something that I like (at least if compared to my previous implementation, I don't how everyone else handles that). What I am asking is how extensible or expandable this approach is (maybe it is too late to ask this kind of questions biggrin.png)? Right now I have a handful of shaders defined in the system. If you had a look at the code, what kind of problems am I likely to run into when adding nodes to deal with Geometry shaders and other advanced features?

Finally, if anyone could be interested in having a look at the tool I'm happy to share it on GitHub. Edited by AvengerDr

Share this post


Link to post
Share on other sites

Sure, it is available here. You only need the librariers named "Odyssey.Common" and "Odyssey.Content". The rest is part of another project, a 2D UI library. The tool itself is the project named ShaderGenerator.

 

In order to build it you will probably have to change the SharpDX SDK line in the Build/ ... targets file to the location you have installed it. The two above projects should be compiled using the Win8Debug configuration. It should work on W7 too. The shaders themselves are defined in the ShaderGenerator/Shaders/Techniques subfolder. The Data/ResourceManager.cs class defines which shaders are loaded at startup. Though saving and loading shader graphs should work. You will need the ColorCode and Galasoft.MvvmLight nuget packages installed.

 

Let me know if you are able to run it. In order to compile shaders every item listed must be assigned to a technique. At the moment these are hardcoded but I am in the process of adding a new window that will allow defining new techniques.

 

Each technique can be decorated in code by Vertex/PixelShaderAttribute(s). These attributes are used when validating nodes. All attributes defined will be used to validate nodes that have the same attribute attached. For example the PhongPS shader does not have the PixelShaderAttribute for DiffuseMap. Thus when validating the PhongLightingNode, the DiffuseMapSampler node won't be checked and so on.

 

Clearly in order to use the automatic configuration of shader variables you would need to parse the metadata in the shader collection file from your own engine. I can post some examples if needed.

Edited by AvengerDr

Share this post


Link to post
Share on other sites

Hi all, I've added support to regular HLSL source code files. Now it can act as a graphical frontend for offline shader compilation. Just press the "import" button and select .hlsl source code files. You can find a link to an installer from here. In the post you'll find also more detailed information regarding the shader graph structure, which now supports Yaml (an example is included in the installer).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Partner Spotlight

  • Forum Statistics

    • Total Topics
      627662
    • Total Posts
      2978516
  • Similar Content

    • By Jcyshadow97
      Hi,guys.I m working on a Fantasy RPG.Currently i m work alone on the project.I really need someone can make UI stuff.If someone can handle it please feel free to contact me on email: 270514974@libero.it.
      Thank you guys and sorry for my english.
       
       



    • By STRATUM the Game
      Hey, everyone! This is my first post here.
      I would like to know what you think about my project called STRATUM. It's a 2D platformer that is heavily based on storytelling and boss fighting while trekking through the world.

      Everything in STRATUM takes place in the first century AD, in a world that wraps our own universe, called  The Stratum. A parallel Universe that is the home of the Christian deities . In this game you will play as a Dacian warrior, unfamiliar with everything in this world, you’ll get to know and understand The Stratum together with him.
      The main thing that I want with STRATUM is to reinvent the known lore and history of the Christian deities and realms. 
      The story is unconventional, it plays down a lot of the mysticism of Hell or Heaven and it gives it a more rational view while keeping the fantastic in it. What do I mean by that? Well, think about Hell. What do you know about it? It's a bad place where bad people go, right? Well, that's not the case in STRATUM. I don't want to describe such a world. In STRATUM, there is a reason for everything, especially for the way Hell is what it is in the game. "Hell" is called The Black Stratum in the game.
      This world is not very different from Earth, but it is governed by different natural laws.
      The story will also involve the reason why this world entered in touch with ours.

       
      What do you think about all that I said? Would you be interested in such a game? I have to say that everything is just a work of fiction made with my imagination. I do not want to offend anyone's beliefs.
      I want this to be a one man game. I have been working alone on it (this was my decision from the beginning) from art to effects to programming to music to sound effects to everything.
      I also have a youtube video HERE if you want to see the way the game moves and the way my music sounds.
      Please, any kind of feedback will be highly appreciated. If you have something bad to say, do it, don't keep it for yourself only. I want to hear anything that you don't like about my project.
       
    • By LimeJuice
      Hi, it's my first post on this forum and I would like to share the game I am working on at the moment.
      Graphics have been made with Blender3D using Cycle as a renderer and I am using Unity3D. It's a 2D game, one touch side-scrolling game for iOS and Android.
      Here some pictures, and you can have a look to the gameplay on this video :
      Feedbacks ?
      And if you want to try it, send me your email and I will add you to the beta tester list!
       
       








    • By Kirill Kot
      An adventure indie game with quests in a beautiful, bright world. Characters with unique traits, goals, and benefits. Active gameplay will appeal to players found of interactivity, especially lovers of quests and investigations.
      Available on:
      Gameroom (just open the web page and relax)
      AppStore
      GooglePlay
      WindowsPhone

    • By Kirill Kot
      Big Quest: Bequest. An adventure indie game with quests in a beautiful, bright world. Characters with unique traits, goals, and benefits.
      Mobile game, now available on Gameroom. Just open the web page and relax.
  • Popular Now