So I'm about to start writing a scene manager and renderer for a game and I'm going to be using deferred shading. My scenes are going to be populated by lots of objects so I'll be using instancing to cut down on draw calls. I also want to try and defer material and texture sampling by having my objects output a material ID into a single-channel FB. Some kind of mega-shader would then convert the material ID into a diffuse and normal sample (probably by having a texture store all material attributes and using the material ID as a v coordinate, and by using texture arrays to get access to different textures).

This raises an issue with material blending. My scene will also feature a terrain at all times (other than when the player is looking up) and I want the textures on my terrain to blend from one to another across a tile. Of course, since the mapping between a pixel and material ID is 1-1, there's no way to represent blending between multiple materials using the setup described above. The way I see it, I have 2 options:

1. Instead of using a single material ID per pixel, use multiple material IDs and have a second output which would be the weights of the materials to blend with in the mega-shader.
2. Draw the terrain seperately straight into the GBuffer after the mega-shader has been run on the scene, using the depth-buffer from the material ID FB. Texture sampling would occur normally here. Lights would have to be rendered in a seperate pass from the mega-shader.

Both options have pros and cons. With option 1, the advantages of drawing into a relatively small FB are lost (now we have 2 textures to render into with weights) and you can only fit up to 4 materials to blend at once (though this might be enough for terrain blending?). Additionally, the uber-shader now has to make up to 4 samples into the material texture and corresponding diffuse/normal textures and blend everything together, potentially taking 4x as long to draw. I'm not sure if it's possible to optimise this somehow such that only 1 sample is made if only 1 material is used with full weight.

With option 2, the uber-shader remains relatively light-weight but there's an extra pass involved and there's probably some caveat of drawing everything in a strange order. Also benefits of the deferred rendering in this case are lost for the terrain (which will have a very large surface area and is also likely to be occluded by many polygons (trees, grass etc) and so would probably benefit from deferred rendering the most).

My question is are there any other options I haven't considered? There is surpsingly a lack of documentation and literature on this matter floating around. If there are no other better options, which option do you think is best for me to choose? Alternatively, is it worth me using deferred rendering at all and should I spend my time making a more intelligent batching system/render queue and try and squeeze more frames out of there?

If it makes any difference I'll be using OpenGL core profile 3.3. My host language is C# / OpenTK.

I'm curious - what's the motivation for doing the material buffer and the second pass expansion of shading attributes? Deferred shading is already somewhat limited in what kind of material variation you can create - but having interesting materials that control what gets painted into the gbuffer can mitigate that. Your design is going to severely limit what you can do - and you're just hitting the first example of that. Are you worried about performance, or is there some other goal here?

Writing out material IDs in a first pass and then creating a G-buffer from that in a second pass then doing lighting in a third pass sounds like an overly complicated solution to a problem you don't even know you have (the problem of doing too many texture samples). How do you know writing out an entire frame buffer of material IDs is going to be faster than doing a small number of unnecessary texture samples?

I would ditch that idea and just use a regular deferred renderer. Write out the G-buffer in the first pass, and do lighting in the second pass. Using whatever materials your objects have during the G-buffer pass. This will let you blend the diffuse color of your terrain in the G-buffer pass just by using the correct terrain shader. No hassle. If you're concerned about wasting time drawing terrain pixels that eventually get occluded, then simply draw the terrain last. The G-buffer pass should render everything front to back, anyway, to mitigate overdraw.

Also, typically if one were to write out a material ID at any point in the deferred pipeline, it would be written directly into the G-buffer, and it would be used to select a set of shading parameters (specular power, surface roughness, etc etc) for the lighting pass.

Performance is my largest concern. Rendering an object with 5 materials would otherwise cost 5 draw calls (1 for each material). Using material IDs, that's only 1 draw call (a material ID could be assigned per-vertex).

I suppose I'm probably over-thinking things and inventing unnecessary solutions to problems which haven't surfaced yet, and may never surface. The biggest thing that will help me is instancing (since I'm likely to have many trees on-screen at any given point), so in theory each material is only going to cost 1 draw call anyway if my render queue batches things intelligently enough. I don't think I can control the order in which objects are drawn via instancing though.

Performance is my largest concern. Rendering an object with 5 materials would otherwise cost 5 draw calls (1 for each material). Using material IDs, that's only 1 draw call (a material ID could be assigned per-vertex).

You may disable writing to z-buffer for the other passes than the first Also, you may find a way to combine the 5 materials into one.

Otherwise you were talking about a terrain rendering, in BF3 they used a terrain tile cache. Instead of blending 5 materials in the g-buffer, you may render the terrain materials into a tile texture and "flatten" the materials, then in the G-buffer pass you can draw pretty complex terrains with very simple shader since the materials have been blended already. So there is no need to overcomplicate your deferred renderer.

Of course, there are limitations, such as tile texture detail, but this can be fixed to certain degree with detail texturing and volume decals etc. Also, the amount of tiles will eat up some GPU memory.

Cheers!