Some of you probably participated in my recent thread about shader reflection and already know what's going on, and I thank those of you for your time and help. But anyways, here's where I'm at now... So I realized I had to abandon the Effects Framework which I've been using through SlimDX/C# for a long time. Now I'm having to essentially design my own Effects Framework to create an API/platform-agnostic solution for my engine. And right now I'm stressing over how to design the variable and cbuffer system for my shading API...

Essentially, what I have in mind is to implement an abstract "ShaderVariable" base class, from which other types will inherit. And I'm wanting to do all of this is a nice, clean manner... taking advantage of C#'s generics capabilities and such. Thus I have in mind some other types like ShaderVariable<T> where T: struct for variables which hold scalars and data (e.g., Vector2/3/4, Color, Matrix, etc), a "ShaderStructure" class which holds a ShaderVariable[] array of members which can be individually addressed, etc...

Right now I'm trying to prototype the system and design, simply using a DirectX10 mini-app to figure out what I'm doing, test some ideas and come up with a good design for the engine. Taking the above idea I started a new code file to begin defining these types:

[source lang="csharp"]
public abstract class ShaderVariable : IData
{
protected ShaderVariable() {
this.Name = string.Empty;
this.Semantic = string.Empty;
}

protected ShaderVariable(string name) : this(name, "") { }

protected ShaderVariable(string name, string semantic) {
this.Name = name;
this.Semantic = semantic;
}

/// <summary>
/// Gets or sets the name of the variable
/// </summary>
public string Name { get; set; }

/// <summary>
/// Gets or sets the semantic of the variable
/// </summary>
public string Semantic { get; set; }

/// <summary>
/// Gets the size, in bytes, of the variable
/// </summary>
public abstract int SizeInBytes { get; }

/// <summary>
/// Converts the variable to bytes
/// </summary>
/// <returns>byte[]</returns>
public abstract byte[] GetBytes();

/// <summary>
/// Returns the name of this variable
/// </summary>
/// <returns>variable name</returns>
public override string ToString() {
return this.Name;
}
};

public class ShaderStructure : ShaderVariable
{
/// <summary>
/// Gets or sets the variables members
/// </summary>
public ShaderVariable[] Members { get; set; }

/// <summary>
/// Gets the size, in bytes, of the structure
/// </summary>
public override int SizeInBytes {
get
{
#if DEBUG || PERFORM_CHECKS
if (Members == null)
throw new NullReferenceException("Member array of ShaderStructure is null reference.");
#endif

int size = Members.Sum( (v) => { return v.SizeInBytes; } );
return size;
}
}

/// <summary>
/// Converts the variable to bytes
/// </summary>
/// <returns>byte[]</returns>
public override byte[] GetBytes() {
return this.Members.ToBuffer();
}
};


public class ShaderVariable<T>
: ShaderVariable where T: struct
{
/// <summary>
/// Gets or sets the value of the variable
/// </summary>
public T Value { get; set; }

/// <summary>
/// Gets the size, in bytes, of the variable
/// </summary>
public override int SizeInBytes {
get { return Marshal.SizeOf( typeof(T) ); }
}

/// <summary>
/// Converts the variable to bytes
/// </summary>
/// <returns>byte[]</returns>
public override byte[] GetBytes() {
return this.Value.ToBytes();
}
};
[/source]

These are by no means complete or final. They're just definitions I started and I wanted you to see what I'm talking about. This is just what I've written thus far for the variable system... Does this seem like a good path to go or am I going about it all wrong?

I'm intending to create a "CBuffer" class which holds a list of variables that can be set/read, and converted into a DirectX10 Buffer to the graphics device. I need to have a strong interface where user code can set variables with ease and not have to be concerned with the technical details of constant buffers, addresses/offsets, etc. However that functionality should be available as well...

The next level up I suppose there will be the "Shader" base class which holds one or more CBuffers, and has an interface for getting/setting the CBuffers, getting info/data about the shader and binding it to the device. There would be derrived types of this for VertexShader, PixelShader, etc...

Next comes the "Effect" class, which I'm unsure how to design. It seems to me it will of course have its own "VertexShader", "PixelShader", etc properties... but beyond that I'm still unsure how I should put it together...

This is my first go with this sort of shading/effect system, so any tips and pointers you can give me would be extremely helpful...

Thanks,

--ATC--

It seems no one knows the answer? :-(

I've made some progress on this front but I'm still in need of some help/advice if anyone can be of any assistance to me. There are also people following this thread who I assume also have a similar design challenge to overcome and would be happy to hear anything helpful.

Thanks,

--ATC--

Hello,

After much thought i approached the problem like this: (Note that this is largely based on the Hieroglyph 3 engine)

I have a ShaderParameter abstract class that every variable type (vector, matrix, texture, etc..) inherits from. The derived classes (one for each type) contain the data. You can call it to get/set data, get type.. This class also has an Update ID (more on this in a second).

Next i have a class that the user interacts with to set shader parameters. The variables are created when a new shader is compiled and stored in an std::map (or hashmap if you want).

Now I divide my parameters into 3 'types' just like the D3D API: Constant buffers, textures and sampler states. A shader class has 3 vectors that contain wrappers around the D3D parameters. The Constant Buffer wrapper has a pointer to the actual buffer and a list of variable descriptions that belong in that buffer. The description struct looks like this:

struct VariableDesc
{
ShaderParameter* pParameter; // pointer to variable
unsigned int UpdateID;
unsigned int Offset; // offset into the buffer
};

Now the UpdateID variable is used to check if the buffer needs to be updated. Whenever the user sets a new value for the variable, its UpdateID that i talked about before is incremented. When you're binding a constant buffer, you loop through its variables and if one of the VariableDesc's UpdateID doesn't match the ShaderParameter's UpdateID, you remap the buffer's content. This is a simple optimization.

Next up you will have a RenderEffect. This is a simple POD struct that contains shaders and render states. It is passed to the Pipeline class which is responsible for binding everything. The RenderEffect doesn't really need to know about the details and so it has no functions.

Finally you have a ShaderStage. This one is an intermediary between the Shader class and the Pipeline class. It simply exists to take care of redundant API calls. Since the Shader class has a list of wrappers in contiguous memory and not the actual D3D pointers, the ShaderStage collects the D3D data and binds all shader resources in 4 calls. One for the shader, one for the cbuffers, one for the textures and one for the samplers.

I can't post code now but if you want i will when i get home. (Be warned though, there's a LOT of it since we're talking about the core of a rendering engine)

You are getting to the point where you may want to consider just making your own shading language and a Flex/Bison parser for it, especially if you plan to support GLSL in the future.
That is what I did and while it is very effective and the “ultimate” solution, it may be too much work.


If you are not willing to go through that much effort, I outline a simpler method in my upcoming book in which you do a very basic pre-process on the shaders yourself, looking for lines that begin with @.
Instead of declaring constants directly, you declare them using your own syntax such as:

@cbuffer buf0
@constant mat4x4 mWorldViewProj
@constant mat4x4 mWorldView
@endbuf
@cbuffer buf1
@constant float fFogFar
@endbuf


You run the shader through your own custom preprocessor and scan for these lines.
When found, you extract the data and store whatever you want from it, including which constant buffers contain which constants and what types those constants are.
Then for each of these lines, your preprocessor replaces them with valid HLSL code, making the final result this:
cbuffer buf0:register(b0) {
float4x4 mWorldViewProj:packoffset(c0);
float4x4 mWorldView:packoffset(c4);
};
cbuffer buf1:register(b1) {
float fFogFar:packoffset(c0);
};

On your application’s end, it would create 2 cbuffer copies, which are just byte arrays.
It knows how big they are because it knows what constants are in each, and it also has a dictionary of the offsets of each constant.


The rest works like any shader system.
The user can get a handle to a constant by name, but the handle is again a custom handle, not a shader handle.
The custom handle you return to the user is really an index into an array.
When the user passes that handle back to you (to set data in a constant), you plug that handle into an array and each entry in that array has all the information you need to upload data into the constant quickly.
The structure would look like this:
struct ATC_SHADER_CONSTANT {
unsigned int uiBufferIndex; // In which cbuffer is it?
unsigned int uiOffset; // Offset into that cbuffer.
};


That is all the information you need to copy the data passed to you into the constant in whatever buffer holds it.
You also need to redundancy-check the data with a simple ::memcmp() (or whatever C# has). Only copy the data over if it is different from what it was before.
And in that case, set the dirty flag on your local cbuffer copy.

Finally, when the shader is activated, check the dirty flags on your local copies of your cbuffers and only update their Direct3D buffers if they have changed since the last time the shader was activated.



Simple as that.


L. Spiro

@Waaayoff:

Sure, I'd be interested to see it if you're willing to post. My project is in C# but I know C++, so we're good. :-)

@L. Spiro:

The cbuffer system you're describing is essentially what I've been working on during the time this thread sat cold and untouched (I guess great minds think alike haha). Though it provides a lot more data/information.

What I'm wondering now is what the best way to further abstract that, create a "ShaderVariable" type (and derrived types like ShaderVariable<T> where T: struct or ShaderResource) and build it up to a new "Effect" class, totally replacing the DirectX Effects Framework.

What is the best way to figure out what resources a shader needs and design a system to deal with them?