• 04/27/15 02:52 PM
    Sign in to follow this  

    Shader Cross Compilation and Savvy - The Smart Shader Cross Compiler

    Engines and Middleware

    Apostol Dadachev

    Introduction

    Many computer scientists suggest that the modern use of shaders originated from the RenderMan Interface Specification, Version 3.0, originally published in May, 1988. In the present, shaders have proven their flexibility and usefulness when writing graphics intensive applications such as games and simulations. As the capabilities of Graphics Processing Units (GPUs) keep increasing, so do the number of shader types and shader languages. While there are many shader languages available to the modern programmer, the most prominent and used ones are the High Level Shading Language for the DirectX graphics API and the OpenGL Shading Language for the OpenGL API.

    The Problem

    Every modern rendering or game engine supports both APIs in order to provide their users with maximum portability and flexibility. What does this mean to us programmers? It means we either write every single shader program twice or find a way to convert from one language to the other. This is easier said than done. The functionality between HLSL and GLSL doesn't differ that much, however the actual implementation of even the simplest diffuse program is completely different in both languages. Every developer who has ever done cross platform development has ran into the same problem. Of course the problem has already been partially solved in many different ways, however all of the available tools seem to lack two-way conversion, are completely outdated or completely proprietary.

    A Solution

    Because of the lack of a general-purpose solution, I thought it would be great to create a flexible free tool, which deals with at least Vertex and Fragment/Pixel shader conversion between modern GLSL 4.5 and HLSL 5.0. That's how the idea for Savvy - The Smart Shader Cross Compiler came to be. The initial idea was to create the tool with support for just the above mentioned languages, however the final implementation can easily be extended to support conversion from and to any language. What this enables is the fact that Savvy is entirely written in C++ and utilizes template classes as well as the latest C++11 advancements. The solution chosen is far from being the best for the presented problem, however it is a solution, which is worth serious consideration.

    Approach

    The approach I decided to use is pure text-based parsing and conversion. The way the system works is really simple, but very powerful. The input shader is first ran through a lexical scanner (generated by the great Flex tool), which matches predefined sequences of characters and returns a specific token. Each returned token is then processed through a grammar parser (a simple flat state machine in this case), which determines whether the text is legitimate and should be saved. The saving is performed inside a database structure, which holds all processed information. That information is later used by a shader constructor class, which constructs the output shader.

    External Dependencies

    The goal of the whole project was to keep the external dependencies as low as possible. The only external software used was flex - the fast lexical analyzer, created by Vern Paxson. It is fast, reliable and great at matching extremely complex character combinations using regular expressions. I absolutely recommend it to anyone looking to do advanced file parsing and token matching. Initially I also wanted to use a third party grammar parser, however after a lot of thought on the subject I decided that syntax checking is not going to be part of the initial release, as the scope would just become overwhelming. This made me use a simple flat state machine which would handle all the grammar. So far so good, now let's see how all of it actually fits together. I'll try to keep things as abstract as possible, without delving too much in implementation details.

    Architecture

    The image below shows a very basic look of the architecture. The Shader Converter class is the main contact point between the user and the internal components of the tool. All conversion calls are made through it. It owns a list of Lexical Scanners, Grammar Parsers and Shader Constructors. All of them are pure virtual base classes, which are implemented once for each supported language. Each Grammar Parser owns a Database and each Shader Constructor owns a Shader Function Converter. The Shader Function Converter takes care of converting all intrinsic functions, which do not have direct equivalents in the output language. The Database also stores the output language equivalents of all built-in data types and intrinsic functions. This type of architecture makes sure that the tool is easily extendable if support for a new language is added.
    YfATfEV.png

    Interface and Functionality

    The Shader Converter has functions for converting a shader from file to file, file to memory, memory to file and memory to memory. All the conversion functions follow the same pattern. Inside the function, all the input is first validated and then an input stream is opened from the path specified. After that, the Lexical Scanner for the appropriate input language is called until an End of File instruction is reached. Each call of the function [tt]GetNextToken[/tt] returns the next token in the stream. The token corresponds to a predefined set of characters in a sequence. For example, the token [tt]SAVVY_DATA_TYPE[/tt] is returned for every data type use. The returned token and its string are then used as an input to the Parser class' [tt]ParseToken[/tt] function, which determines the state and saves the word to the database if needed. If the input and output shader languages specified are the same, the shader is simply copied over to the specified path without any alterations. Any included files are also parsed in the same fashion, by calling the parsing function recursively. After the file has been parsed, the input steam is closed and an output stream is opened. Then the Constructor is called and everything saved in the database is output to the stream. The order of construction is: 1. Construct Defines 2. Construct Uniform(Constant) Buffers 3. Construct Samplers(Textures) 4. Construct Input Variables 5. Construct Output Variables 6. Construct Custom User Types(structs) 7. Construct Global Variables 8. Construct Functions

    The Function Converter

    I feel like I should spend some time explaining what the Function Converter class actually does. Its job is to make sure each intrinsic function of the input language is translated to the appropriate equivalent of the output language. Unfortunately, there are some functions which are absolutely impossible to translate as they refer to very specific graphics API calls. To give an example, consider the HLSL function [tt]D3DCOLORtoUBYTE4[/tt]. The problem here becomes apparent, as there is no [tt]UBYTE4[/tt] data type in GLSL. Upon reaching a function which cannot be converted to the specified output language, an exception will be thrown (or an error code will be returned if the preprocessor directive [tt]SAVVY_NO_EXCEPTIONS[/tt] is defined) by the tool and conversion will stop. There are, however, some functions which can be translated, despite the fact that they do not have direct alternatives in other languages. One such function is the arc hyperbolic cosine function in GLSL - [tt]acosh[/tt] (well, technically all hyperbolic functions apply here, as none of them are supported in HLSL). The function itself, given an input value [tt]x[/tt] can easily be defined as the following equation: [code] log(x + sqrt(x * x - 1)); [/code] When functions of this type are encountered, they are substituted by their inline version. The final type of function conversions which the Function Converter handles are those which do have alternatives, but for some reason the output language implementation takes a different amount of arguments or the argument order is swapped. An example of a function which has the exact same functionality, but is implemented differently in both languages is the arc tangent function - [tt]atan[/tt]. In GLSL, the function has two possible blueprints. One takes one argument (the [tt]y[/tt] over [tt]x[/tt] value) and the other takes the two inputs, [tt]x[/tt] and [tt]y[/tt], separately. This is a problem, as the HLSL equivalent does not have an overloaded blueprint for two arguments. Instead, it uses a separate function - [tt]atan2[/tt]. To account for this difference the function converter determines the number of arguments a function call has and according to that, outputs the correct type of function call. If the input shader language has a function which takes one argument less than its output language equivalent, a dummy value will be declared on the line above the call and the last argument will be filled by it, in order to preserve the functionality.

    An Odd Case

    To add one more function example to the last type - the [tt]fmod[/tt] function in HLSL and its "supposed" GLSL equivalent - [tt]mod[/tt]. At first glance everything looks great and both versions of the shader should produce the same results, right? Wrong! The internal equations used by those functions are not the same. The GLSL one, according to the official documentation is: [code] x - y * floor(x/y) [/code] While the HLSL one is: [code] x = i * y + f [/code] Both implementations produce the same results if dealing with positive numbers as inputs, however, the moment the input becomes negative, the HLSL version fails to produce the expected results. It also seems like other cross compilers prefer the former direct approach of converting [tt]mod[/tt] to [tt]fmod[/tt] and vice versa, as it is faster when executing the shader. I decided to choose the mathematically correct equation and whenever these functions are encountered in the input shader, the proper inline equation will be constructed in the output shader.

    Conversion from File to File

    Here is what the declaration of the file to file conversion function looks like: [code] /* Converts a shader file from a file to another file. */ ResultCode ConvertShaderFromFileToFile(FileConvertOptions& a_Options); [/code] As you can see, the function takes a structure of type [tt]FileConvertOptions[/tt], which contains all the needed data for the conversion. For example - shader input path, shader output path, entry points and shader type. Here is a sample usage of the file to file conversion: [code] Savvy::ShaderConverter* converter = new Savvy::ShaderConverter(); Savvy::ResultCode res; Savvy::FileConvertOptions options; options.InputPath = L"PathToMyInputFragShader.glsl"; options.OutputPath = L"PathToMyOutputFragShader.hlsl"; options.InputLang = Savvy::GLSL_4_5; options.OutputLang = Savvy::HLSL_5_0; options.ShaderType = Savvy::FRAGMENT_SHADER; res = converter->ConvertShaderFromFileToFile(options); [/code]

    Conversion from Memory

    Another great feature which wasn't initially planned but was implemented at a later stage is conversion of shaders from memory to memory, memory to file and file to memory. In order to make things easier for the user, the [tt]Blob[/tt] class was created, which is very similar to the DirectX 11 one and is just a container for raw data. Its interface is very simple, but effective for the user as it serves for sending raw character strings to the converter and also retrieving the converted ones after the conversion has been done. The internal conversion is done by constructing a string stream from the [tt]Blob[/tt] and having the scanners and parsers operate on that. A simple example of how one can use the [tt]Blob[/tt] to [tt]Blob[/tt] conversion is the following: [code] Savvy::ShaderConverter* converter = new Savvy::ShaderConverter(); Savvy::ResultCode res; // Load file in memory std::ifstream is("SomeFile.something"); if (!is.is_open()) { std::cout << "Error reading file" << std::endl; } std::string fileStr(static_cast(std::stringstream() << is.rdbuf()).str()); is.close(); // Create a blob with the loaded file in memory Savvy::Blob inputBlob(&fileStr[0], fileStr.size()); Savvy::Blob outputBlob; Savvy::BlobConvertOptions options; options.InputBlob = &inputBlob; options.OutputBlob = &outputBlob; options.InputType = Savvy::HLSL_5_0; options.OutputType = Savvy::GLSL_4_5; res = converter->ConvertShaderFromBlobToBlob(options); // Output the converted blob to file to verify its integrity std::ofstream str("BlobToBlobTest.txt"); std::string mystring(options.OutputBlob->GetRawDataPtr(), options.OutputBlob->GetDataSize()); str << mystring; str.close(); [/code]

    Extending the Tool

    While the current implementation only supports conversion of Vertex and Fragment programs between modern GLSL and HLSL shading languages, it is very easy to extend the tool and add support for custom languages and shader types. The supported languages are separated into supported output languages and supported input languages. If the user wishes to extend the tool to support an extra output language, they need to create their own Constructor class by inheriting from the base class. After they have done that, the only thing needed is to call the following function supplying an ID and a default extension for the new output language. [code] /* Registers a new custom shader language for output purposes only. If the ID of this shader type is used as an input, the conversion will fail. */ template ResultCode RegisterCustomOutputShaderLang(uint32 a_ID, const wchar* a_DefaultExtension); [/code] In order to add support for a new input language, the user needs to supply custom Lexical Scanner and Parser classes and also register them using the following function: [code] /* Registers a new custom shader language for input purposes only. If the ID of this shader type is used as an output, the conversion will fail. */ template ResultCode RegisterCustomInputShaderLang(uint32 a_ID); [/code] Note that a default extension here is not needed as a shader from this language will never be constructed. Likewise, support for a new shader type (geometry, compute) can also be added.

    Current Limitations

    Of course no software is without its flaws. Savvy is far from perfect. In this section I'll go in a bit more detail about the current shortcomings of the tool and the chosen approach in general.

    Texture Samplers

    The thing with GLSL is that the Texture Sampler and Texture objects are joined together in one, while in HLSL you have to explicitly declare a Sampler Object together with the Texture Object. When manually writing a shader you can use one Sampler to sample multiple Textures, however when programatically converting a GLSL shader to an HLSL one, there is no way of knowing how many samplers we might need, as OpenGL keeps Sampler Objects on the C++ side. A lot of people don't even use Samplers in OpenGL yet, as they are quite a new extension there. In order to make sure every texture is going to be properly sampled in HLSL, every declaration inside a GLSL shader produces both a Texture Object and a Sampler Object, which is named: NameofTextureObject + Sampler. For example: DiffuseTextureSampler. This is less than ideal, as we are consuming more Sampler Registers than we need, but unfortunately there is no other way at this point in time, using purely text-based conversion.

    The Preprocessor

    Another thing which is not 100% foolproof is the preprocessor conversion. Unfortunately there is no way of knowing which variables on global scope will be inside a specific preprocessor block. Because the tool stores everything in a database, there is no way for it to check if a specified variable is active or inactive in the current global scope. There is also no way of guaranteeing how many variables are inside a preprocessor block and of what type they are (inputs, outputs or uniforms/constant buffers). In order to avoid the issues which come with many branching preprocessor directives, I have decided not to support any conditional compilation outside of functions (Global Scope). Defines are still supported and will always be the first thing constructed. Conditional compilation is also supported inside functions without any constraints. This is possible due to the fact that every function is translated line by line and is guaranteed to keep its consistency across multiple languages.

    Syntax Checking

    Another big constraint of the tool is that it does not support any syntax checking, so if the user feeds it a shader which does not compile as an input, the output shader will not compile either. There are some internal checks in place, to make sure the tool isn't processing complete garbage, but they are far from being a reliable syntax verifier. In truth, syntax checking is possible to implement and might even be considered in the future, but for now it stays in the unsupported category.

    Future Development

    The current officially supported languages are GLSL 4.5 and HLSL 5.0. I have done some testing with GLSL shaders as far back as version 3.3 and they seem to work nicely, however take my words with a pinch of salt. The officially supported shader types are Fragment/Pixel and Vertex. Savvy is currently in Beta stage and after some proper testing and code documentation will be released, together with the source in the public domain. That way anyone who is interested in the project can contribute or use it as a base to make something else. There are many features planned for the tool, the biggest one of which is to add support for the upcoming intermediate shader language SPIR-V, which was announced at the Game Developer Conference 2015. Adding support for extra shader types like geometry and compute is also something planned for the future. Implementing a legacy GLSL profile is not out of the question either, if there is interest for it.

    Conclusion

    I must say, developing this tool was a very interesting journey. A lot of hours were put in it and a lot of valuable lessons were learned. I decided to write this article as there aren't many articles dedicated to this particular problem out there. I hope it was useful to anyone interested in shader cross-compilation and cross-platform graphics development in general. Its aim was to explain the solution I chose to overcome the problem of shader cross-compilation and also highlight some positive/negative points about the approach. If I missed anything or you have extra questions, ideas or feedback, please do share them. I love hearing other people's opinions and I'm always open to constructive criticism.

    Savvy 1.1 (Beta) Released

    I am proud to announce that the source of Savvy version 1.1 (Beta) is now available at this location. Feel free to give it a go and post any issues, requests, etc.

    Useful Links

    Article Update Log

    [b]19 April 2015[/b]: Initial version of the article released. [b]15 May 2015[/b]: Savvy Version 1.1 (Beta) released on Github. Supplied Link.


      Report Article
    Sign in to follow this  


    User Feedback

    Create an account or sign in to leave a review

    You need to be a member in order to leave a review

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now


    khawk

    Report ·

      

    Share this review


    Link to review
    Ken Hu

    Report ·

      

    Share this review


    Link to review
    xexuxjy

    Report ·

      

    Share this review


    Link to review
    EMascheG

    Report ·

      

    Share this review


    Link to review
    behc

    Report ·

      

    Share this review


    Link to review
    Laval B

    Report ·

      

    Share this review


    Link to review