Sign in to follow this  
Followers 0
Hodgman

OpenGL
Emulating CBuffers

11 posts in this topic

We just to do constant buffer emulation back in the dark ages when we used D3D9. It was really simple...we just split up the constants by update frequency and kept track of what register each group started at. Then we'd author a corresponding C++ struct with matching parameters (and matching alignment), along with a templated class that would just set the constant data by passing a pointer to the struct to SetVertexShaderConstantF/SetPixelShaderConstantF at the proper offset. For per-draw constants we had multiple structs with the same register offset, and you'd just only use one in particular shader.

Thank god we switched to D3D11 and I don't have to deal with that crap anymore. [img]http://public.gamedev.net//public/style_emoticons/default/smile.png[/img]
2

Share this post


Link to post
Share on other sites
Thanks for taking the time to discuss this topic :)

I hadn't thought of using Lua to describe the FX section but I'll look into it. At the moment I was going on the path of having my own HLSL-like syntax which I would parse. This FX section would basically define all the constant buffers and techniques/contexts for the shader - very similar to Horde3D except with cbuffer support I suppose.

The main problem I'm having is the part where you talk about a mask so the engine knows what buffers to set before drawing. With the design above, it forces the cbuffers defined in the FX section to have unique names so that a user could look up a cbuffer for a particular shader type. Also, how does the mask fit in with your state group idea where you have cbuffer bind commands, are these commands necessary if you already have a mask to know what has to set?

This is an example of the structure/usage I was going for:

Effect:
[CODE]

[FX]
cbuffer cbPerObjectVS : register( b0 )
{
matrix g_mWorldViewProjection;
matrix g_mWorld;
};

cbuffer cbPerObjectPS : register( b0 )
{
float4 g_vObjectColor;
};


context SIMPLE
{
VertexShader = compile VS_SIMPLE;
PixelShader = compile FS_SIMPLE;
}


[VS_SIMPLE]

cbuffer cbPerObjectVS : register( b0 )
{
matrix g_mWorldViewProjection : packoffset( c0 );
matrix g_mWorld : packoffset( c4 );
};

...


[PS_SIMPLE]
cbuffer cbPerObjectPS : register( b0 )
{
float4 g_vObjectColor : packoffset( c0 );
};

...
[/CODE]

Usage:
[CODE]

perObjectVSIndex = model->Effect()->FindCBuffer("cbPerObjectVS");
perObjectVSData = model->Effect()->CloneBuffer(perObjectVSIndex);
[/CODE]

This would mean I'd need to have a lookup table to know what shader type (GS/VS/PS/etc) defines the CBuffer the user is asking for. In the case above it would map to the vertex shader in the effect. The CBuffer defined in the FX section is mainly just for creating the lookup table, the actual CBuffer layout is created in each shader type via reflection.

I'm not really sure this is the best way to go about it so I'm open to suggestions as I'm still in design/thinking stages :)
1

Share this post


Link to post
Share on other sites
By the way, seeing as you use C# for your content pipeline, does that mean you only target windows for your engine or do you have bindings for other languages too?
0

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1326068445' post='4900792']With the design above, it forces the cbuffers defined in the FX section to have unique names so that a user could look up a cbuffer for a particular shader type.[/quote]Yeah at the moment, I force shader authors to give every cbuffer both a name and an ID.

[quote]Also, how does the mask fit in with your state group idea where you have cbuffer bind commands, are these commands necessary if you already have a mask to know what has to set?[/quote]The mask is separate from the binding commands.
After the user has issued a bunch of items to draw, it ends up as a list of state-changes (e.g. bind commands) and draw-calls. In the D3D9 renderer, a bind command simply writes some data into a cache and sets a dirty bit. When a draw-call is issued, this cbuffer cache is used to perform the actual [font=courier new,courier,monospace]Set*ShaderConstantF[/font] calls just prior to the [font=courier new,courier,monospace]Draw*Primitive[/font] call.

Before setting shader constants (before a draw-call), the renderer first has to select a shader permutation, which has one of these cbuffer masks. Only the cbuffers in the cache which are specified in this mask will be flushed through to D3D. Note that this is just a slight optimisation though, not a required feature.

There's a few reasons that the user may have issued a bind-cbuffer command that isn't actually needed --- perhaps the cbuffer is used by most permutations, but there's one permutation where it's not used. e.g. maybe your scene has a "[i]flash of lightning[/i]" effect, where you simply disabling lighting calculations for that frame. The user might bind a light cbuffer regardless, but then some other layered render-state causes a "[i]no lighting[/i]" permutation to be selected. In this situation, this allows the author of the "[i]lighting[/i]" effect to quickly implement their idea without changing any of the "[i]calculate/bind lighting cbuffers[/i]" code.

Another possibility is that you've got some global cbuffers, which you simply always bind out of convenience. e.g. maybe you define cbuffer #12 as holding fog/atmosphere settings, and bind this data to this slot by default ([i]unless a particular renderable overrides that state with it's own binding[/i]). In this case, every draw-call would have [i]something[/i] bound to cbuffer slot #12, but there might be some shaders/permutations that ignore fog, and hence don't need that buffer to be bound.

[quote]The CBuffer defined in the FX section is mainly just for creating the lookup table, the actual CBuffer layout is created in each shader type via reflection.
I'm not really sure this is the best way to go about it so I'm open to suggestions as I'm still in design/thinking stages [/quote]My only objection to that format is that you're repeating yourself (declaring cbuffers twice), which adds an extra place where mistakes can be made.

Would it be possibe to automatically generate one of these sets of information from the other? ([i]e.g. generate the HLSL variables from the FX, or generate the FX by parsing/reflecting the HLSL?[/i]).

[quote]perObjectVSIndex = model->Effect()->FindCBuffer("cbPerObjectVS");
perObjectVSData = model->Effect()->CloneBuffer(perObjectVSIndex);[/quote]I've got an API like this, but for many cases I don't have to use it. For most engine-provided data, I can instead do:
[font=courier new,courier,monospace]perObjectVSData = CBuffer::Create( sizeof(StructThatIPromiseMatchesMyHLSL) );[/font]

And for debugging:
[font=courier new,courier,monospace]CBufferInfo* cb = model->Effect()->FindCBuffer("cbPerObjectVS");[/font]
[font=courier new,courier,monospace]ASSERT( cb->SizeInBytes() == sizeof(StructThatIPromiseMatchesMyHLSL) );[/font]
[font=courier new,courier,monospace]ASSERT( OFFSETOF(StructThatIPromiseMatchesMyHLSL::foo) == cb->OffsetOf("foo") );[/font]

[quote]By the way, seeing as you use C# for your content pipeline, does that mean you only target windows for your engine or do you have bindings for other languages too?[/quote]The engine is multi-platform, but the content tools are only designed to work on a Windows PC, because that's what we use for development ;)

There's a few options for integrating your windows-only content tools with your cross-platform engine that I've personally used:
1) Have some kind of "editor" build of the engine, which does more stuff than any of the single-platform builds (e.g. contains multiple different platform-specific data structures - so you can serialize your data for each different platform).
2) Link your content tools against your Windows build of your engine, and use formats that serialize identically for all platforms.
3) Don't directly link your content tools to your engine at all. Instead, define a specification for the input/output data formats, and implement a that spec once in the tools (as a producer) and once in the engine (as a consumer).

I currently use method #3 -- The C# tools uses easy-to-use-but-bloated data structures internally, and have hand-written binary serialisation code for outputting data to the engine. The C++ engine loads these data files into byte-arrays and casts them to hand-written structs that match the expected data layouts.
0

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1326415883' post='4902171']
've got an API like this, but for many cases I don't have to use it. For most engine-provided data, I can instead do:
perObjectVSData = CBuffer::Create( sizeof(StructThatIPromiseMatchesMyHLSL) );

And for debugging:
CBufferInfo* cb = model->Effect()->FindCBuffer("cbPerObjectVS");
ASSERT( cb->SizeInBytes() == sizeof(StructThatIPromiseMatchesMyHLSL) );
ASSERT( OFFSETOF(StructThatIPromiseMatchesMyHLSL::foo) == cb->OffsetOf("foo") );
[/quote]

In my engine setup, all C structures that are required to match against HLSL cbuffers are compile-time forced provide a decl function for matching the alignments.
I have a templated subclass of the buffer object that turns CBuffer::Create( sizeof(StructThatIPromiseMatchesMyHLSL) ); into TCBuffer<structthatipromisematchesmyhlsl>, which when compiled under paranoia flags performs validation.

FWIW, I've taken the same approach to declaring structures in this way for vertex stream formats and pixel shader outputs.
The pixel shader output being defined in said way, allows my shader compiler to split a pixel shader with say, 7 float4 outputs into 2 seperate shaders for platforms that only support 4 targets under MRT.
The vertex stream format decl was helpful in getting hassle-free instancing up and running. The Transform CBuffer struct can be easily treated as a secondary vertex stream for DrawInstanced calls.</structthatipromisematchesmyhlsl>
1

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1326415883' post='4902171']
My only objection to that format is that you're repeating yourself (declaring cbuffers twice), which adds an extra place where mistakes can be made.

Would it be possibe to automatically generate one of these sets of information from the other? (e.g. generate the HLSL variables from the FX, or generate the FX by parsing/reflecting the HLSL?).
[/quote]

I can understand your point of view but in both cases I see some potential problems, but maybe you have a way around those. If I specify the CBuffers in the FX, how do you know what shader type and permutation actually require that particular buffer? Or, if you specify them only in the shader code, how can you ensure there will always be the ability to do reflection? I'm only aware of DX having a reflection API for HLSL, is there an equivalent for GLSL?

Having a custom FX section allows me to have non platform specific meta data for uniforms, samplers, cbuffers, etc which would be useful in content generation tools (I don't think GLSL has semantic/annotation data?). In saying this, I'd probably have to go with the [i]generating HLSL from FX section[/i] path as I can leave the generation of the shader code up to the low-level rendering backend, I simply pass it a cbuffer object created from the FX section and it'll return me the shader language representation - but as I mentioned above I'm not sure how to specify which shader type and in what permutation actually requires a particular CBuffer so it can be inserted into the shader code and compiled. I'd probably need a more complex FX section which specifies that link/mask which is what I gather your Lua FX section does?

[quote]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]The engine is multi-platform, but the content tools are only designed to work on a Windows PC, because that's what we use for development ;)[/left][/size][/font][/color]
[/quote]

I'm attempting to go on the path of the engine API itself being used for any of the content tools. That way I only have to write things once and it can be used on any platform. Obviously creating the models and art would be done on whatever machine artists use, but there would be exporters for the main tools like 3dsmax and Blender.

[quote]
[color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left]When a draw-call is issued, this cbuffer cache is used to perform the actual [/left][/size][/font][/color][color=#282828][size=3][left]Set*ShaderConstantF[/left][/size][/color][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left] calls just prior to the [/left][/size][/font][/color][color=#282828][size=3][left]Draw*Primitive[/left][/size][/color][color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif][size=3][left] call.[/left][/size][/font][/color]
[/quote]

This sort of stems from the same issue I described earlier, but in your system how do you determine what shader type a CBuffer is linked to so you can know what [color=#282828]Set*ShaderConstantF [/color]function to call?

Thanks appreciate your help!
0

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1326678355' post='4903105']I can understand your point of view but in both cases I see some potential problems, but maybe you have a way around those.
(1) If I specify the CBuffers in the FX, how do you know what shader type and permutation actually require that particular buffer?
(2) Or, if you specify them only in the shader code, how can you ensure there will always be the ability to do reflection?[/quote]Yeah, being explicit (repeating yourself) can sometimes be a good thing. For example, it's common to have some variables that only show up when in an editor like Maya, but aren't "officially" part of the shader, so you could omit them from the [FX] part.
For (1), seeing you're writing your own syntax here, you could add a property to your FX description that explicitly states this, e.g.
[font=courier new,courier,monospace]cbuffer cbPerObjectVS : register( [u][b]vs_b0[/b][/u] ){ ... };[/font]

For (2), you can write a full-blown HLSL parser yourself, and convert your HLSL source into an abstract-syntax-tree, which you can reflect over yourself. That's of course quite a bit more complex than the alternative of producing HLSL from your 'reflection' format though ;)
This is what we do at work, and the high investment cost to write your own parser/translater pays off by allowing you to use any kind of custom syntax that you like, to translate your code into multiple output languages, and to do things like converting [font=courier new,courier,monospace]if[/font]/[font=courier new,courier,monospace]for[/font] statements into permutations, etc...

[quote]I'm not sure how to specify which shader type and in what permutation actually requires a particular CBuffer so it can be inserted into the shader code and compiled. I'd probably need a more complex FX section which specifies that link/mask which is what I gather your Lua FX section does?[/quote]At the moment, my shader format at home is still fairly primitive, so I output [u]all[/u] [fx] cbuffers into both the VS and PS. This has the effect of artificially limiting the number of constant registers available to me -- e.g. if a vertex shader cbuffer is bound to c0-c200, then those registers are unusable for ps variables for no good reason.
I'll probably remedy this by adding an explicit description like (1) above, where the [fx] section can say which shader-types should include the cbuffer ([i]but still output the cbuffer for [u]every[/u] permutation, for simplicity's sake[/i])
[quote]This sort of stems from the same issue I described earlier, but in your system how do you determine what shader type a CBuffer is linked to so you can know what [color=#282828]Set*ShaderConstantF [/color]function to call?[/quote]There's three different situations here:
1) The engine/game code is binding a known cbuffer structure -- the person writing that code can hard-code either a SetPsCBuffer/SetVsCBuffer/etc command, because they know which shader they want to set the data to.
2) The content tools are compiling an artist's material -- all shader uniforms ([i]which have been annotated with a "should appear in DCC GUI"-type tag[/i]) are available to be set by the artists when they author a model/material. When their models are imported, the content tools will take these uniform values, and search through every cbuffer (for every shader type) for a variable who's name matches.
When a match is found, an instance of that cbuffer is instantiated (or the existing instance grabbed) and the default value is overwritten with the artist's value.
If that name only shows up in a PS cbuffer, then only a PS cbuffer will be instantiated, and only a PS binding command generated.
3) The engine/game code is binding some cbuffer values, but the structure of the cbuffer isn't hard-coded. In this case, they can use the reflection API to iterate every cbuffer for each shader type, and find which cbuffers (and shader types) contain the variable they're trying to set, and then create the appropriate buffer instances and binding commands. If this is something that was going to happen every frame, you'd instantiate the relevant cbuffers/binding commands once, and store the relevant offsets into those cbuffers where your dynamic data will be written. N.B. in most cases, use-case #1 can be used instead of this #3 use-case.
1

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1326702538' post='4903165']
For (1), seeing you're writing your own syntax here, you could add a property to your FX description that explicitly states this, e.g.
cbuffer cbPerObjectVS : register( vs_b0 ){ ... };
[/quote]

I think I'll definitely go with that approach. It seems pretty elegant and flexible and like you mentioned you can easily hide non "official" shader parameters from content tools simply by not putting it in the FX section. I also think I was over thinking some of this and outputting the cbuffers to every permutation would probably suffice :)

[quote]
1) The engine/game code is binding a known cbuffer structure -- the person writing that code can hard-code either a SetPsCBuffer/SetVsCBuffer/etc command, because they know which shader they want to set the data to.
[/quote]

I hadn't actually thought of having separate bind commands for each shader type but it makes sense to do that. Do you think there is a need to actually flag what shader type a cbuffer is meant for, or would a generic cbuffer object suffice? Off the top of my head I can't think of a need why you might want to check if it's a "VS cbuffer" or a "PS cbuffer" etc.
0

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1326752260' post='4903398']Do you think there is a need to actually flag what shader type a cbuffer is meant for, or would a generic cbuffer object suffice? Off the top of my head I can't think of a need why you might want to check if it's a "VS cbuffer" or a "PS cbuffer" etc.[/quote]I don't think there's a need -- my C++-side cbuffer instance structures are very minimal -- they're basically just an array of bytes. They don't even know what their "name" is, what their layout is ([i]i.e. no link to a reflection structure[/i]), or which shader/technique they were originally created for.
0

Share this post


Link to post
Share on other sites
I have couple more questions that sprung to mind while I was thinking of how to link materials with the effect/cbuffer system.

1) If the CBuffer structures are so minimal how do you determine what buffer register to bind them to when creating the bind commands for the state groups in your materials, etc? (assuming you only store the buffer register index in the CBufferInfo class and not in your CBuffer classes)

One option I thought was that the position in the vector of cbuffers would be the register index but then realised that wouldn't work because the vector contains buffers from all shaders types and so you could have vector items [i]0,1 = {vs_b0, vs_b1}[/i] and items [i]2,3 = {ps_b0, gs_b0}[/i].

2) When dealing with DX11, at what point do you create the actual buffer resources and how do you store/link them with the emulated cbuffer structure?

As for designing a material format, I was thinking each material would get a copy of all the default cbuffers from the effect/shader it holds and then use the material specification to determine what cbuffer constants to lookup and change. Is this essentially what you do in your system?
0

Share this post


Link to post
Share on other sites
[quote name='Shael' timestamp='1327287081' post='4905306']1) If the CBuffer structures are so minimal how do you determine what buffer register to bind them to when creating the bind commands for the state groups in your materials, etc? (assuming you only store the buffer register index in the CBufferInfo class and not in your CBuffer classes)[/quote]I'd create the buffers and their binding commands at the same time, where you've got the shader reflection information available. The command has a link to the buffer instance, but neither retains a link to the reflection info from which they were created.[quote]2) When dealing with DX11, at what point do you create the actual buffer resources and how do you store/link them with the emulated cbuffer structure?[/quote]When loading a cbuffer from disk, or when creating one programmatically, I'd make the DX11 resource immediately. There may not even be a need to keep around an 'emulated copy' of the cbuffer's contents, especially if it's a cbuffer that will not ever be modified.[quote]As for designing a material format, I was thinking each material would get a copy of all the default cbuffers from the effect/shader it holds and then use the material specification to determine what cbuffer constants to lookup and change. Is this essentially what you do in your system?[/quote]Yeah, but I don't instantiate all of the default cbuffers -- only the ones that have at least one value set by the material are instantiated ([i]and have binding commands inserted into the material's state-group[/i]).
N.B. not every cbuffer is a material cbuffer -- e.g. you don't want your materials binding your camera cbuffer, especially if your material state-group has a higher priority than your camera state-group!
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • By recp
      Hi,
      I'm working on new asset importer (https://github.com/recp/assetkit) based on COLLADA specs, the question is not about COLLADA directly
      also I'm working on a new renderer to render (https://github.com/recp/libgk) imported document.
      In the future I'll spend more time on this renderer of course, currently rendering imported (implemented parts) is enough for me
      assetkit imports COLLADA document (it will support glTF too),
      importing scene, geometries, effects/materials, 2d textures and rendering them seems working
      My actual confusion is about shaders. COLLADA has COMMON profile and GLSL... profiles,
      GLSL profile provides shaders for effects so I don't need to wory about them just compile, link, group them before render

      The problem occours in COMMON profile because I need to write shaders,
      Actually I wrote them for basic matrials and another version for 2d texture
      I would like to create multiple program but I am not sure how to split this this shader into smaller ones,

      Basic material version (only colors):
      https://github.com/recp/libgk/blob/master/src/default/shader/gk_default.frag
      Texture version:
      https://gist.github.com/recp/b0368c74c35d9d6912f524624bfbf5a3
      I used subroutines to bind materials, actually I liked it,
      In scene graph every node can have different program, and it switches between them if parentNode->program != node->program
      (I'll do scene graph optimizations e.g.  view frustum culling, grouping shaders... later)

      I'm going to implement transparency but I'm considering to create separate shaders,
      because default shader is going to be branching hell
      I can't generate shader for every node because I don't know how many node can be exist, there is no limit.
      I don't know how to write a good uber-shader for different cases:

      Here material struct:
      struct Material { ColorOrTexture emission; ColorOrTexture ambient; ColorOrTexture specular; ColorOrTexture reflective; ColorOrTexture transparent; ColorOrTexture diffuse; float shininess; float reflectivEyety; float transparency; float indexOfRefraction; }; ColorOrTexture could be color or 2d texture, if there would be single colorOrTex then I could split into two programs,
      Also I'm going to implement transparency, I am not sure how many program that I needed

      I'm considering to maintain a few default shaders for COMMON profile,
      1-no-texture, 2-one of colorOrTexture contains texture, 3-........

      Any advices in general or about how to optimize/split (if I need) these shaders which I provied as link?
      What do you think the shaders I wrote, I would like to write them without branching if posible,
      I hope I don't need to write 50+ or 100+ shaders, and 100+ default programs

      PS: These default shaders should render any document, they are not specific, they are general purpose...
             I'm compiling and linking default shaders when app launched

      Thanks
    • By CircleOfLight97
      Hi guys,
      I would like to contribute to a game project as a developer (open source possibly). I have some experiences in C/C++ in game development (perso projects). I don't know either unreal or unity but I have some knowledges in opengl, glsl and shading theory as I had some courses at university regarding to that. I have some knowledges in maths and basic in physics. I know a little how to use blender to do modelling, texturing and simple game assets (no characters, no animation no skinning/rigging). I have no game preferences but I like aventure game, dungeon crawler, platformers, randomly generated things. I know these kind of projects involve a lot of time and I'd be really to work on it but if there are no cleary defined specific design goals/stories/gameplay mechanics I would like to not be part of it x) and I would rather prefer a smaller but well defined project to work on that a huge and not 'finishable' one.
      CircleOfLight97
    • By gamesthatcouldbeworse
      Hi, I finally released KILL COMMANDO on gamejolt for free. It is a retro-funsplatter-shooter with C64 style. Give it a try.
    • By phil67rpg

      void TimerFunction(int value) {  glutPostRedisplay();  glutTimerFunc(1000, TimerFunction, 1); } void drawScene() {  glClear(GL_COLOR_BUFFER_BIT);      drawScene_bug();  TimerFunction(1);  eraseScene_bug(); // drawScene_bug_two(); // eraseScene_bug_two(); drawScene_ship(); drawScene_bullet();  glutSwapBuffers(); }
  • Popular Now