"Dynamic" effect files

Graphics and GPU Programming Programming

Started by schupf August 18, 2009 06:14 PM

13 comments, last by Nik02 14 years, 8 months ago

221

Author

August 18, 2009 06:14 PM

Hello, I am trying to write a mesh class to store the geometry, the indices and a material/shader. I am using DX10 and I want to use effect files. Unfortunately I discovered severel problems. I havent written any code, but my class might look like this:

class Mesh {
private:
	ID3D10Effect*			mEffect;	
	ID3D10EffectTechnique*		mTechnique;

	// How to store shader variables?

	ID3D10InputLayout*		mVertexLayout;
	ID3D10Buffer*			mVB;
	ID3D10Buffer*			mIB;    
public:
    Mesh(string meshName, string effectFile, string techniqueName);

My first problem is: How should I store the global effect file variables (like worldViewProj)? After loading the effect file I can access each shader variable with:

Quote:ID3D10EffectVariable* e = effect->GetVariableByName("camPosWorld");

and thus I could store the variables as something like std::vector<ID3D10EffectVariable*>. But then I still dont know how to SET the variables when it comes to rendering. If the variable is a matrix (float4x4) I actually have to use this code:

Quote:worldViewProjSC->SetMatrix( (float*)worldViewProj);

thus I have to cast the ID3D10EffectVariable to a ID3D10EffectMatrixVariable and then call SetMatrix(). But I dont want to use big switch constructs for every ID3D10EffectVariable. How could I automatically load, store and set my effect file variables?

dys129

116

August 18, 2009 07:45 PM

This is my idea of handling materials, shaders and meshes:

In my simple engine I've separated shaders (materials) from meshes. Because I assume I want to have one material per many meshes. So I have one class MaterialManager which is responsible for holding effect files and wraps all functions to set shader variables. Also it uploads once per frame global variables in constant buffer (time, view and projection matrices etc.). Then I have material class which contains hlsl effect technique name and all variables. And when I call method Material::Update() all variables which belong to this material are uploaded. Then finally my mesh class has pointer to material and before rendering Material::Update is called.

Maybe it is not the best but it is fairly simple and I hope you can develop smth from this for yourself :)

I know I'm not answering your question directly but handling shaders and materials are not so simple and you must consider it well for a greater good :)

schupf

221

Author

August 18, 2009 09:18 PM

Thanks for your answer.

I really like your system and I think I am going to develop something very similar. Just 2 questions about your system:

1. You say your MaterialManager is responsible for setting the variables of an effect file. But some sentences later you say your material class has the variables. What do you mean with this?
The problem is: An effect file contains global variables and these are used in techniques. The global variables can be used by many techniques, for example it is possible to do something like this:

// pseudocode:lightDirConstant->SetFloatVector( &lightDir );technique1->GetPassByIndex(0)->Apply(0);renderTechnique1();technique2->GetPassByIndex(0)->Apply(0);renderTechnique2();

So technique1 and technique2 use the "global" shader variable "lightDir", but you just have to set them once. How do you handle this?

2. How does the system get the values for the automatic update? When you call Material::Update() the system has to set all automatic shader constants, but how do they get the values? Do you have to register member variables like this?
material->registerAutomaticConstant("lightDir", &m_lightDir);

Nik02

4,359

August 18, 2009 11:06 PM

1: Look at "Effect Pools" for sharing variables between effects. They are very easy to use, but you may need to refactor your effect creation code slightly if you haven't used them before.

2: The effect interface provides the GetVariable* methods for convenience, so you don't have to manually determine the constant buffer where the variable actually lives within the shaders, or manually calculate the offset of the variable within its containing buffer.

The effect system doesn't actually care where you update the variables, but you do need to use the GetVariable* methods as well as the classes derived from D3D10EffectVariable - or update the effect's constant buffers manually - if you want the variables to be uploaded to the device when you call Apply.

I assume that your Material::Update method takes its encapsulated effect and uses GetVariableByName for each effect variable that it handles + sets them to the effect? This is as automatic as you get.

Remember that you can also get effect variables by semantics, or you can annotate them in free-form fashion in the effect files. This way, you can implement custom logic for finding variables, but somewhere in your code you need to update them manually. If your general architecture is robust enough, you only need to write the code once, though.

For better performance, you can store the variable handles that you get by finding the effect variables. This way, you will avoid potentially expensive string lookups during rendering.

Setting effect variables is by using stored handles is very lightweight, since the effect interface lives on user mode and effectively only calls core API methods (Set*Shader etc.) when you call Apply.

Niko Suni

schupf

221

Author

August 19, 2009 02:05 AM

@Nik02: Thanks for your answer, but they did not fit 100% to my question;)

I did not ask about variables shared across different effect files. I asked about how to handle variables that are shared across different TECHNIQUES in one effect file (see my code).

But my main problem stays: Every technique is different. One needs a float3 lightDir, the other technique a float exposure and so on. I wonder how I can treat all the different techniques. If I have one Technique class for all techniques, I cant give the class member variables that reflect the shader constants of that technique. For example I cant just write this:

class Technique {public:    ID3D10EffectScalarVariable* exposure;};

because most of my techniques dont need the exposure. Currently I can just imagine two solutions: A) Having ONE Technique class, which holds NO shader variables as member. To set a constant I need to do something like this:

class Technique {public:    ID3D10Effect* effect;    void setScalarConstant(string name, float val) {        effect->GetVariableByName(name)->AsScalar()->SetFloat(val);    }    // Similar functions setMatrixConstant() and so on...};

The problem with this approach: On every update I call 3 functions (GetVariableByName, AsXX, SetYY) for all constants. I don't know how expensive these calls are, but I don't have a good feeling with this approach. I would strongly prefer just calling scalarVar->setFloat(val); But again this isnt possible if I have ONE Technique class that handles every technique.

B) My second idea is to have a base class Technique and for EVERY technique in an effect file I create a subclass, that has the appropriate shader variables for this technique:

class TechniqueDiffuse : public Technique {public:    ID3D10EffectScalarVariable* exposure;    ID3D10EffctMatrixVariable* world;};

Now I can just call techniqueDiff->exposure->setFloat(val);

I neither like the first nor the second idea. I dont like the first idea because I don't want to call 3 functions to update a shader constant. And I don't like the second idea because this would lead to a lot of classes...

I hope you understand my problem. What do you think of these approaches? How do you handle it?

dys129

116

August 19, 2009 05:34 AM

I really like that you like my system :)

Asnwering your question:

1. As I said in MaterialManager I have wrapers for setting dx variables. And what I meant is that in material class contains values for this variables and in Update function it updates all its variables.

But material instance contains only data relevant to one mesh instance ( Diffuse, Normal, specular map etc.). Global variables like lights values are set once and in the beginning of the frame. But if some material want to change some global variable ( I can't imagine why :) it has to do it on its own and then restore original value.

Ad 2. I'm using constant buffers for variables shared between materials like :

cbuffer {
Texture diffuse, normal, specular;
}

and all other variables ( for different materials) are stored like typical variables :) ie:

//Reflectance material
Texture refl;
float refl_strength;

And when Material::Update is called it uploads ALL variables specific to this material via MaterialManager wrapers. Just as Nik02 said.

In my engine I've got only few materials because I'm using deferred shading :)

schupf

221

Author

August 19, 2009 06:45 AM

To be honest I still have no idea how I efficiently set the shader constants of an effect file.
Lets say I have an effect file foo.fx with 2 techniques: techniqueA and techniqueB. BOTH techniques use the shader constant "float3 light" and additionally techniqueA uses "float fooA" and techniqueB uses "float2 fooB".

If I wanted to write a own Technique class for each technique everything would be easy. Since I know which shader constant each technique is using, I can add the right types in the class:

struct TechniqueA {TechniqueA(ID3D10Effect* effect_) : effect(effect_) {   fooA = effect->GetVariableByName("fooA")->AsScalar();}ID3D10Effect* effect;ID3D10EffectScalarVariable* fooA;  // Scalar};struct TechniqueB {TechniqueB(ID3D10Effect* effect_) : effect(effect_) {   fooA = effect->GetVariableByName("fooA")->AsVector();}ID3D10Effect* effect;ID3D10EffectScalarVariable* fooB;  // Vector};

But I DON'T want to write a class for each technique. But if I just have ONE generic Technique class I don't know which types (ID3D10EffectScalarVariable etc.) I will need (since every technique can use different constants of different types) and thus I cant store the shader constants as members. Thus I am forced to provide methods for each type:

void setScalarConstant(string name, float val) {        effect->GetVariableByName(name)->AsScalar()->SetFloat(val);    }

But this sucks, because actually everything is static but this code fetches the shader constants on every update again and again with GetVariableByName(). I think this is unnecessary slow. So how should me Technique class look like? Some (pseudo)code would be helpful.

Thanks

Nik02

4,359

August 19, 2009 07:18 AM

The GetVariable* methods don't actually fetch the data stored in the variables; instead, they effectively calculate the constant buffer and the offset in which the variable resides, and return an encapsulation of that information to you in form of effect variable interfaces. Likewise, the Set* methods of the variable interfaces do not immediately upload the variable data to the hardware.

The effect compiler (and shader compiler) is smart enough to optimize out uploading of unused variables. Thus, if you set an effect variable but the constant buffer containing it isn't actually used in the shaders of the passes of the current technique, the system doesn't upload that constant buffer at all.

I recommend storing specialized variable pointers with your objects, and updating them in a strongly typed manner. This way, you will gain maximum performance out of the system. As I see it, you're over-abstracting your design by storing generic ID3D10EffectVariable:s, because in practice you know in advance what type of variables your objects need to update in the effects, as well as the actual variable semantics (meanings).

Ask yourself, how many different technique archetypes do you actually need? In a medium sized production, the figure isn't very high. Thus, it shouldn't be a problem to implement specialized classes for each completely different technique. Note that you can modify the techniques themselves very flexibly by using effect variables.

[Edited by - Nik02 on August 19, 2009 7:18:04 AM]

Niko Suni

schupf

221

Author

August 19, 2009 08:15 PM

Quote:Original post by Nik02 As I see it, you're over-abstracting your design by storing generic ID3D10EffectVariable:s,

Uhm, no. There is no generic ID3D10EffectVariable in my code. Only concrete types like ID3D10EffectScalarVariable*.

So if I understand you correctly you suggest to write technique "archetypes", which can handle more than one technique, where all these handled techniques are similar.
For example lets say I have 2 techniques TechniqueNormalMapping and TechniqueParallaxMapping in an effect file. Both techniques are very similar. Lets say TechniqueNormalMapping needs a "float3 light" as input and TechniqueParallaxMapping a "float3 light" and a "float height". So in my host application I write one class and it will look like this:

TechniqueBumpMapping {   ID3D10EffectScalarVariable* height   ID3D10EffectVectorVariable* light;public:   TechniqueBumpMapping(string technique, string effectfile);   void setHeight(float height);   void setLight(Vec3 light);};// Usage:TechniqueBumpMapping* tbm = new TechniqueBumpMapping("TechniqueParallaxMapping", "foo.fx");tbm->setLight(light);...

Of course method setHeight() doesnt make sense for TechniqueNormalMapping, but thats no problem: I will just never call setHeight() for this technique object.

Is this what you meant?

Nik02

4,359

August 20, 2009 12:26 AM

Yes.

You can fully modify the execution of a technique by changing the effect variables. If many of your objects are rendered with the same basic technique, but their positions and textures differ, you can just set the effect variables for the positions and textures but keep using the same effect.

The hardware doesn't care about extraneous data that the effect might set. It only uses what the shaders use.

That said, you should group your constant buffers by the frequency of use, as the SDK recommends. This way, the effect system doesn't have to upload all the constant buffers all the time if you keep modifying only small amount of variables in high frequency (as in for each renderable object).

Niko Suni

"Dynamic" effect files

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

"Dynamic" effect files

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines