• entries
    5
  • comments
    6
  • views
    7415

Common Shading Language

Sign in to follow this  
ongamex92

3708 views

The saga started here : https://www.gamedev.net/topic/668484-hlsl-and-glsl-wraping-again/#entry5234746

Basically I want a clean way to be able to use one shader program with multiple rendering APIs. As described in the linked topic above, my current solution to this problem is to match and modify specific tokens in my intermediate "language" in order to generate the final HLSL/GLSL code. This may sound good ?n paper, but as you can see (in the topic above) the shader code is bloated, ugly and highly unextendable (this is why I'm trying to solve this issue with a different approach).

After asking the public, I've decided to give the language writing a try (thanks to L. Spiro and a colleague of mine - G. Totev).

The idea is to write a language that could be translated to HLSL/GLSL/(something else). I was really scared of that task because... well, it is about writing a compiler and certainly I don't know (and still don't) how to write one.

Thankfully, there is a pair of tools - Flex and Bison (their ancestors are lex and yacc). These tools are used to describe the language.
Flex is used for finding (matching) keywords, strings, literals, identifiers, and Bison is used for matching those tokens in expressions (in other words describing the grammar of the language by using context free grammars). The Output of these tools is a set of C/C++ function(s) that knows how to parse your language. But more on C/C++ integration with Flex/Bison later (or maybe in another Journal).

After grinding a lot I found out that this isn't that hard of a task for my humble needs. The big problem was finding good resources on the topic. After hours of researching I found these :
http://people.seas.harvard.edu/~bwaggoner/writeups/jumpstart/flexbison/jumpstart_flexbison.pdf
http://epaperpress.com/lexandyacc/download/LexAndYaccTutorial.pdf
(a quick hint: add yacc to your search query in order to increase the chance for correct results).

I'm currently in the process of developing that language. This is still not a complete solution, but the hard work is almost done.

The language can :
- Deduce the type of an expression ( for example the languages knows that matrix * vector in HLSL should be mul(matrix, vector) )
- Variable shadowing.
- Is able to produce some HLSL/GLSL (the shading stage specifics aren't yet supported, but it's just a matter of time. Currently there are input/output varyings, global uniforms, vertex attributes).

- Error checking is done only if the code generation depends on it. I want the code simple, why I want that is described below.

The code can be found here : https://github.com/ongamex/playground/tree/master/lang KEEP IN MIND THAT IT IS A PLAYGROUND REPOSITORY, but you can look at it if you're curious.

So what is the final goal?

The ultimate goal is to provide the base functionality (not a complete solution), that is easily extendable by others.
There are similar projects that either dead or their code is too complicated which basically means that insane amount of time is needed to create a modification. Currently the project doesn't meet this requirement (but the code is still very simple). In order to achieve that I must find the best architecture for the code and write (or find) a good tutorial about Bison.

And finally a quick demo of the language :// KEEP IN MIND THAT THIS IS NOT A REAL SHADER :)// GLSL like declarations with nice flavor of HLSL's semantics.attribute vec3f a_pos : POSITION;uniform Texture2D diffuseTex;out vec2f uv; //output varyingin mat4f world; // input varyingvoid main() { mat4f m; vec4f v = vec4f(5); int x = lerp(0.3, 0.0, 1.0); uv = 2; // the language still doesn't care about types, but... // HERE it does // the compiler knows that we are multiplying a matrix by a ?????. // And outputs the correct result m * lerp(0.3, 5.0, 10.0) * vec4f(0.0); m * mix(0.3, vec4f(0), vec4f(1.0)) // if-else branching works fine... if(x>0) v = -(world * -v.xxxy * +(getm() * v)); // member access and swizzles. else if(x<0) x = 10; vertex_output = v; }mat4f getm() { return; } // the language doesn't care what you're are actually returning, but you can get that error from the hlsl/glsl compiler.
And so far the output HLSL(I will skip the GLSL) looks like this(with a bit of shuffling) :uniform Texture2D diffuseTex;uniform sampler diffuseTex_sgeSS;float4x4 getm() { return;}struct SGE_SHADER_RESULT { float2 uv : uv; float3 vertex_output : SV_Position;};SGE_SHADER_RESULT main(float3 a_pos : POSITION, float4x4 world : world) { SGE_SHADER_RESULT sge_shader_result; float4x4 m; float4 v=float4(5); int x=lerp(0.30, 0.0, 1.0); sge_shader_result.uv = 2; if(x > 0)v = -(mul(world, -v.xxxy) * (mul(getm(), v))); else if(x < 0)x = 10; sge_shader_result.vertex_output = v; mul(m * lerp(0.30, 5.0, 10.0), float4(0.0)); mul(m, lerp(0.30, float4(0), float4(1.0))); return sge_shader_result;}
I will post more updates about the language here or In a new journal. Feel free to ask me anything, report an error, and give any type of feedback!
Sign in to follow this  


6 Comments


Recommended Comments

Glad to see that you are enjoying the journey through creating a new shader language. Many times people just assume that these kinds of huge tasks are completely tangential and avoid them, but as you have found they can also end up being a fun and rewarding new journey in themselves, especially since once you’ve gotten the learning process out of the way you can apply your experience to many more things in the future.


L. Spiro

Share this comment


Link to comment
Enjoy. This stuff is fun. I'm engaged in a similar project myself (scripting language and VM) although I've stayed away from the tools you mention in favour of manual token scanning and building a node tree (AST) myself. It's not nearly as hard as one might think and code generation from an AST is a pleasure to implement.

Share this comment


Link to comment

Cool. Yeah, I think this could be simpler with a basic top-down parser. Lex+yacc are kinda crazy. Also, if you used a scripting language, your AST could basically be a nested array of strings; no need to convert to enums and back. (Problem is, too many langs to choose from.)

 

If people keep going along these lines we could have a shader language that compiles to LLVM bytecode for Vulkan... or direct to GPU machine code in a no-OS scenario (potentially better than implementing OpenGL+GLSL warts-and-all from scratch). But let's not get too crazy just yet... :D

Share this comment


Link to comment

Just a suggestion, but instead of making a new language, why not use GLSL as "your" language --or HLSL. Then, write code to export the GLSL to HSLS/someotherlanguage?  In this way, you do not need to write an extra language, but work with what already exists -- it might be easier this way as well?

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now