Sign in to follow this  
schupf

"Dynamic" effect files

Recommended Posts

Hello, I am trying to write a mesh class to store the geometry, the indices and a material/shader. I am using DX10 and I want to use effect files. Unfortunately I discovered severel problems. I havent written any code, but my class might look like this:
class Mesh {
private:
	ID3D10Effect*			mEffect;	
	ID3D10EffectTechnique*		mTechnique;

	// How to store shader variables?

	ID3D10InputLayout*		mVertexLayout;
	ID3D10Buffer*			mVB;
	ID3D10Buffer*			mIB;    
public:
    Mesh(string meshName, string effectFile, string techniqueName);

My first problem is: How should I store the global effect file variables (like worldViewProj)? After loading the effect file I can access each shader variable with:
Quote:
ID3D10EffectVariable* e = effect->GetVariableByName("camPosWorld");
and thus I could store the variables as something like std::vector<ID3D10EffectVariable*>. But then I still dont know how to SET the variables when it comes to rendering. If the variable is a matrix (float4x4) I actually have to use this code:
Quote:
worldViewProjSC->SetMatrix( (float*)worldViewProj);
thus I have to cast the ID3D10EffectVariable to a ID3D10EffectMatrixVariable and then call SetMatrix(). But I dont want to use big switch constructs for every ID3D10EffectVariable. How could I automatically load, store and set my effect file variables?

Share this post


Link to post
Share on other sites
This is my idea of handling materials, shaders and meshes:

In my simple engine I've separated shaders (materials) from meshes. Because I assume I want to have one material per many meshes. So I have one class MaterialManager which is responsible for holding effect files and wraps all functions to set shader variables. Also it uploads once per frame global variables in constant buffer (time, view and projection matrices etc.). Then I have material class which contains hlsl effect technique name and all variables. And when I call method Material::Update() all variables which belong to this material are uploaded. Then finally my mesh class has pointer to material and before rendering Material::Update is called.

Maybe it is not the best but it is fairly simple and I hope you can develop smth from this for yourself :)

I know I'm not answering your question directly but handling shaders and materials are not so simple and you must consider it well for a greater good :)

Share this post


Link to post
Share on other sites
Thanks for your answer.

I really like your system and I think I am going to develop something very similar. Just 2 questions about your system:

1. You say your MaterialManager is responsible for setting the variables of an effect file. But some sentences later you say your material class has the variables. What do you mean with this?
The problem is: An effect file contains global variables and these are used in techniques. The global variables can be used by many techniques, for example it is possible to do something like this:
// pseudocode:
lightDirConstant->SetFloatVector( &lightDir );
technique1->GetPassByIndex(0)->Apply(0);
renderTechnique1();
technique2->GetPassByIndex(0)->Apply(0);
renderTechnique2();


So technique1 and technique2 use the "global" shader variable "lightDir", but you just have to set them once. How do you handle this?

2. How does the system get the values for the automatic update? When you call Material::Update() the system has to set all automatic shader constants, but how do they get the values? Do you have to register member variables like this?
material->registerAutomaticConstant("lightDir", &m_lightDir);

Share this post


Link to post
Share on other sites
1: Look at "Effect Pools" for sharing variables between effects. They are very easy to use, but you may need to refactor your effect creation code slightly if you haven't used them before.

2: The effect interface provides the GetVariable* methods for convenience, so you don't have to manually determine the constant buffer where the variable actually lives within the shaders, or manually calculate the offset of the variable within its containing buffer.

The effect system doesn't actually care where you update the variables, but you do need to use the GetVariable* methods as well as the classes derived from D3D10EffectVariable - or update the effect's constant buffers manually - if you want the variables to be uploaded to the device when you call Apply.

I assume that your Material::Update method takes its encapsulated effect and uses GetVariableByName for each effect variable that it handles + sets them to the effect? This is as automatic as you get.

Remember that you can also get effect variables by semantics, or you can annotate them in free-form fashion in the effect files. This way, you can implement custom logic for finding variables, but somewhere in your code you need to update them manually. If your general architecture is robust enough, you only need to write the code once, though.

For better performance, you can store the variable handles that you get by finding the effect variables. This way, you will avoid potentially expensive string lookups during rendering.

Setting effect variables is by using stored handles is very lightweight, since the effect interface lives on user mode and effectively only calls core API methods (Set*Shader etc.) when you call Apply.

Share this post


Link to post
Share on other sites
@Nik02: Thanks for your answer, but they did not fit 100% to my question;)

I did not ask about variables shared across different effect files. I asked about how to handle variables that are shared across different TECHNIQUES in one effect file (see my code).

But my main problem stays: Every technique is different. One needs a float3 lightDir, the other technique a float exposure and so on. I wonder how I can treat all the different techniques. If I have one Technique class for all techniques, I cant give the class member variables that reflect the shader constants of that technique. For example I cant just write this:
class Technique {
public:
ID3D10EffectScalarVariable* exposure;
};

because most of my techniques dont need the exposure. Currently I can just imagine two solutions: A) Having ONE Technique class, which holds NO shader variables as member. To set a constant I need to do something like this:
class Technique {
public:
ID3D10Effect* effect;
void setScalarConstant(string name, float val) {
effect->GetVariableByName(name)->AsScalar()->SetFloat(val);
}
// Similar functions setMatrixConstant() and so on...
};

The problem with this approach: On every update I call 3 functions (GetVariableByName, AsXX, SetYY) for all constants. I don't know how expensive these calls are, but I don't have a good feeling with this approach. I would strongly prefer just calling scalarVar->setFloat(val); But again this isnt possible if I have ONE Technique class that handles every technique.

B) My second idea is to have a base class Technique and for EVERY technique in an effect file I create a subclass, that has the appropriate shader variables for this technique:
class TechniqueDiffuse : public Technique {
public:
ID3D10EffectScalarVariable* exposure;
ID3D10EffctMatrixVariable* world;
};

Now I can just call techniqueDiff->exposure->setFloat(val);

I neither like the first nor the second idea. I dont like the first idea because I don't want to call 3 functions to update a shader constant. And I don't like the second idea because this would lead to a lot of classes...

I hope you understand my problem. What do you think of these approaches? How do you handle it?

Share this post


Link to post
Share on other sites
I really like that you like my system :)

Asnwering your question:

1. As I said in MaterialManager I have wrapers for setting dx variables. And what I meant is that in material class contains values for this variables and in Update function it updates all its variables.

But material instance contains only data relevant to one mesh instance ( Diffuse, Normal, specular map etc.). Global variables like lights values are set once and in the beginning of the frame. But if some material want to change some global variable ( I can't imagine why :) it has to do it on its own and then restore original value.

Ad 2. I'm using constant buffers for variables shared between materials like :

cbuffer {
Texture diffuse, normal, specular;
}

and all other variables ( for different materials) are stored like typical variables :) ie:

//Reflectance material
Texture refl;
float refl_strength;

And when Material::Update is called it uploads ALL variables specific to this material via MaterialManager wrapers. Just as Nik02 said.

In my engine I've got only few materials because I'm using deferred shading :)

Share this post


Link to post
Share on other sites
To be honest I still have no idea how I efficiently set the shader constants of an effect file.
Lets say I have an effect file foo.fx with 2 techniques: techniqueA and techniqueB. BOTH techniques use the shader constant "float3 light" and additionally techniqueA uses "float fooA" and techniqueB uses "float2 fooB".

If I wanted to write a own Technique class for each technique everything would be easy. Since I know which shader constant each technique is using, I can add the right types in the class:
struct TechniqueA {
TechniqueA(ID3D10Effect* effect_) : effect(effect_) {
fooA = effect->GetVariableByName("fooA")->AsScalar();
}
ID3D10Effect* effect;
ID3D10EffectScalarVariable* fooA; // Scalar
};

struct TechniqueB {
TechniqueB(ID3D10Effect* effect_) : effect(effect_) {
fooA = effect->GetVariableByName("fooA")->AsVector();
}
ID3D10Effect* effect;
ID3D10EffectScalarVariable* fooB; // Vector
};



But I DON'T want to write a class for each technique. But if I just have ONE generic Technique class I don't know which types (ID3D10EffectScalarVariable etc.) I will need (since every technique can use different constants of different types) and thus I cant store the shader constants as members. Thus I am forced to provide methods for each type:

void setScalarConstant(string name, float val) {
effect->GetVariableByName(name)->AsScalar()->SetFloat(val);
}



But this sucks, because actually everything is static but this code fetches the shader constants on every update again and again with GetVariableByName(). I think this is unnecessary slow. So how should me Technique class look like? Some (pseudo)code would be helpful.

Thanks

Share this post


Link to post
Share on other sites
The GetVariable* methods don't actually fetch the data stored in the variables; instead, they effectively calculate the constant buffer and the offset in which the variable resides, and return an encapsulation of that information to you in form of effect variable interfaces. Likewise, the Set* methods of the variable interfaces do not immediately upload the variable data to the hardware.

The effect compiler (and shader compiler) is smart enough to optimize out uploading of unused variables. Thus, if you set an effect variable but the constant buffer containing it isn't actually used in the shaders of the passes of the current technique, the system doesn't upload that constant buffer at all.

I recommend storing specialized variable pointers with your objects, and updating them in a strongly typed manner. This way, you will gain maximum performance out of the system. As I see it, you're over-abstracting your design by storing generic ID3D10EffectVariable:s, because in practice you know in advance what type of variables your objects need to update in the effects, as well as the actual variable semantics (meanings).

Ask yourself, how many different technique archetypes do you actually need? In a medium sized production, the figure isn't very high. Thus, it shouldn't be a problem to implement specialized classes for each completely different technique. Note that you can modify the techniques themselves very flexibly by using effect variables.

[Edited by - Nik02 on August 19, 2009 7:18:04 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Nik02 As I see it, you're over-abstracting your design by storing generic ID3D10EffectVariable:s,
Uhm, no. There is no generic ID3D10EffectVariable in my code. Only concrete types like ID3D10EffectScalarVariable*.


So if I understand you correctly you suggest to write technique "archetypes", which can handle more than one technique, where all these handled techniques are similar.
For example lets say I have 2 techniques TechniqueNormalMapping and TechniqueParallaxMapping in an effect file. Both techniques are very similar. Lets say TechniqueNormalMapping needs a "float3 light" as input and TechniqueParallaxMapping a "float3 light" and a "float height". So in my host application I write one class and it will look like this:

TechniqueBumpMapping {
ID3D10EffectScalarVariable* height
ID3D10EffectVectorVariable* light;

public:
TechniqueBumpMapping(string technique, string effectfile);
void setHeight(float height);
void setLight(Vec3 light);
};

// Usage:
TechniqueBumpMapping* tbm = new TechniqueBumpMapping("TechniqueParallaxMapping", "foo.fx");
tbm->setLight(light);
...



Of course method setHeight() doesnt make sense for TechniqueNormalMapping, but thats no problem: I will just never call setHeight() for this technique object.

Is this what you meant?

Share this post


Link to post
Share on other sites
Yes.

You can fully modify the execution of a technique by changing the effect variables. If many of your objects are rendered with the same basic technique, but their positions and textures differ, you can just set the effect variables for the positions and textures but keep using the same effect.

The hardware doesn't care about extraneous data that the effect might set. It only uses what the shaders use.

That said, you should group your constant buffers by the frequency of use, as the SDK recommends. This way, the effect system doesn't have to upload all the constant buffers all the time if you keep modifying only small amount of variables in high frequency (as in for each renderable object).

Share this post


Link to post
Share on other sites
As a side note, I still think you're over-abstracting a bit.

Do you have a genuine need to define a separate class for rendering techniques?

In all the engines I've worked with (including my own), the best approach has turned out to be that both the effect handle (ID3D10Effect*) and the technique handle (ID3D10EffectTechnique*) are stored directly as a member of renderable object interface (or base class), and the concrete object classes store the effect variable handles that they need. The objects then update only those effect variables that they use, no more, no less.

The point being, D3D already seems to introduce a sufficient level of abstraction for most applications in this regard.

Share this post


Link to post
Share on other sites
The problem i am seeing with archtypes for your techniques/effects/shaders is that you lose the ability to add them without changing the rendering code. So for each new techniques/effects/shaders you need to add a new archtype if it's not known to the engine yet.

Why not describe the parameters you use in another file and write a generic way of setting variables into the device. Yes this would require a switch for each shader variable type, but an interger compare is fast. And its code you only write once.

This gives you the abillity to add techniques/effects/shaders on the fly without changing the code base.

Or am I misunderstading the issue you are having?

Share this post


Link to post
Share on other sites
Quote:
Original post by NightCreature83

This gives you the abillity to add techniques/effects/shaders on the fly without changing the code base.


But the question is, is that ability so important that you spend a considerable time to implement it, and lose runtime performance as a result anyway? In most projects I've seen - small, medium and large - the answer is unequivocally "no".

It can be done by inspecting the variables and their types at runtime, but this will result in switch statements and lookups by strings, which (I assume) the OP actually wanted to avoid.

Share this post


Link to post
Share on other sites
Quote:
Original post by Nik02 but this will result in switch statements and lookups by strings, which (I assume) the OP actually wanted to avoid.

Exactly. All of you guys are right. The drawback of writing one Technique-Archetype for every technique "class" is obvious. For every new Technique (or if an existing technique needs a new shader constant) I have to change my host code and in the end it spams my source folder.
I also don't want to use "dynamic" structures like a map or switch statements for something that is actually pure static. At runtime I know how the shader looks and the objects know which shader constants there have to set.

Thus I will do it like Nik02 proposed. I will just use the DirectX effect and technique class and directly add the effect variables (ID3D10EffectXYZVariable) to my objects. The reason why I initially wanted to introduce an additional layer of abstraction: I looked at some engines (ogre, irrlicht) and they all had some kind of Material/Shader classes. So I thought its a good idea to provide such classes. But since the effect framework of Direct already delivers a nice form of abstraction I will just use them without something on top of that:)

Share this post


Link to post
Share on other sites
The reason why Ogre and similar engines use further abstraction is that they can potentially render with other systems than D3D.

However, if your abstraction classes are tied to D3D anyway, the additional layer is somewhat unnecessary.

Even when you only use D3D, it still probably is a good idea to encapsulate material-specific variables to their own class or structure, but not for abstraction or for direct interaction with Effects - rather, for storage convenience.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this