Hi,
I've just checked out the new UE4 (Unreal Eninge 4) and its material editor and I'm thinking how it works.
Is a new shader compiled for each material? If yes, isn't the lots of shader switches affecting the performance? How can it be optimized?
Edit:
According to this paper, the material is compiled into shader code.
This node graph specifies inputs (textures, constants), operations, and outputs, which are compiled into shader code
You definitely don't need to compile to a new shader if just some data of the material differs (e.g. two materials share everything except for one diffuse albedo texture), but I don't know the UE4 source so I can't say what they do. But it's not at all atypical to compile new shaders when certain things in a material or related rendering parameters change and end up with systems that have thousands, tens of thousands (or much more) shader combinations. In this blog post, aras_p of Unity mentions 1 million variations for a single shader, which seems kind of ludicrous to me but apparently it happens.
The reason this is doable at runtime is because you only use very few of those resulting variations at the same time in a scene. Undoubtedly if you assembled a scene that needed to go through 1M shader switches every frame, your CPU wouldn't react kindly.
Wow, that sounds crazy! 
The reason this is doable at runtime is because you only use very few of those resulting variations at the same time in a scene. Undoubtedly if you assembled a scene that needed to go through 1M shader switches every frame, your CPU wouldn't react kindly.
That's that makes me disturbed. I thought switching to a new shader is expensive; I have to set up new attributes, uniforms and everything (I'm targeting OGL 2) which is related to the new shader even if only 1 object is drawn by that. And it could happen: in this tutorial series a single wood material is created and maybe it's used by only 1 object in the whole scene.
The Unreal engine does optimize the shaders for you when you apply them.
Each material consist of small shaders doing something simple, these small shaders combined to form a large shader for a material, this is why the unreal will build multiple shaders when you add a new kind of material.
This way you can have two materials that share a single part of a shader, lets say Wood and Metal, thy share a vertex shader but each has a unique pixel shader. Now instead of using four shaders you use three.
Now if you have a red chair and you copy it's material to make a green chair, thy will share most of there shaders. However this doesn't change the fact that you added more materials.
If you wan't to make say a large swarm of a hundred aliens who very in color, you will need hundreds of materials and this will cause a large performance hit.
Luckily Unreal has a simple and effective solution, material instancing: https://docs.unrealengine.com/latest/INT/Engine/Rendering/Materials/MaterialInstances/index.html
If you are familiar with animation or mesh instancing this works the same.
A over simplified explanation is that you render the same thing with slight changes.
Oh yea, I didn't write the original post accurate enough. I meant new shader for each material which differs not just the values but the "structure" as well. So the same as you wrote here. :) BTW, this is a pretty flexible system, I love it. Has anyone tried to implement similar system?
Thanks for your replies! :)
If by "structure" you mean the shader as a text file, like:
void main() {
gl_TexCoord[0] = gl_MultiTexCoord0;
// Set the position of the current vertex
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
}
and
void main() {
// Set the position of the current vertex
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
gl_TexCoord[0] = gl_MultiTexCoord0;
}
are two files, yet thy are the same shader.(This is GLSL not for Unreal)
Unreal's material has no problem understanding that it's the same shader.
Unreal understands shaders as a group of instructions so both these are the same vertex shader with two instructions.
When unreal turn materials into shaders it doesn't add instructions that exists, instead it refers to the ones that exist.
However if you have some thing like:
UV = (vec3(0,0,0)) it's a instruction.
and !=
UV = (vec3(0,0,1)) so both instructions will be saved.
All shaders using UV = (vec3(0,0,0)) will refer to the first instruction, so no unnecessary data is stored.
In the end if you have a two vertex shader that use UV = (vec3(0,0,0)) and thy each uses other instructions that differ then you will still have two shaders.
For two shaders to be considered the same thy must have the exact same instructions.
Has anyone tried to implement similar system?
I have. It's a very intuitive way to work with shaders. The beauty of this is that you can design your nodes and shader structure such that the artists/users can customize exactly what you want them to customize, while maintaining control over the underlying structure of the shader, as opposed to generating the entire shader code only from nodes and requiring users to hook certain nodes in certain ways.
I am planning to work more on the system and release it as middleware for game developers so they can integrate it in their editors/pipeline/engines. Unfortunately I'm busy with another project currently. Here is a screenshot:
Some shader parameters can be dynamic and some can be "static". Changing a static parameter will require recompiling the shader, thus resulting in a new permutation of the shader. Another beauty of this system is makes it easier to create a platform-independent shader solution.
Has anyone tried to implement similar system?
I built one a while back.
It basically starts chunks of pre-written shader code, fills in a bunch of variables in the text, and sticks them all together in a single file - the only tricky part is mapping inputs to outputs, and you can create a GUI interface to let the user manage those.
I have. It's a very intuitive way to work with shaders
Agree. Oh, and your tool looks pretty cool. 
I built one a while back.
Do you have any reference? My main problem is just how I can identify that two materials should use the same shader with different uniform values. Or let it be controlled by the user? Like in UE4, they have material instances which uses the same "base material" aka shaders.
A shader is any unique (legal) combination of nodes. A material is that shader plus all the instance data required to render it (uniforms, textures, etc). i.e. you need strong separation between the functionality of a given block, and the instance data it requires at runtime.My main problem is just how I can identify that two materials should use the same shader with different uniform values. Or let it be controlled by the user? Like in UE4, they have material instances which uses the same "base material" aka shaders.
