Yesterday was spent doing some preparation for the decal system.
If you look at the screenshots from yesterday, and look at the stairs, you'll note that in the level editor, the stairs look like they are made from a smooth material, wheras in the game, it looks cracked. This is because the editor wasn't taking normal maps into account.
But because decals may be effectively normal map only ( like for hieroglyphics, carving, scratches or symbols ), I needed to display at least the decals with some sense of the normal map. So, I changed the rendering for the base materials as well to use their normal map in the rendering. This fixes the stairs so that they look more similar to the in-game version.
Of course, I'm only doing vertex lighting in the level editor, so how to display the bump maps? The blue channel of a tangent space bump map is a good indicator of how locally smooth a surface is, so I just grab the blue channel of the normal map, and do some math do emphasize contrast :
smoothness = ( ( blue * blue ) - 0.75f ) * 4.0f;
Then I modulate this by diffuse tex * diffuse lighting to bring out the bumpy sections.
Here is an example of what it should look like. Here are two different materials, both use the same diffuse map, and one has a lizard bump map.
I actually used to use a similar trick in my ambient pass during in the game engine. One problem with bump maps is that if they are totally in shadow from all lights ( only receiving ambient ), they are invisible. One good approach is probably hemisphere lighting, but I recently changed from just the blue channel trick to adding a dim virtual light coming from above ( sort of like a sky light ).
Decals in the Editor
This morning I put decal mode into the editor. So, you can place, move, scale, rotate, create and delete decals.
The way the decal system is supposed to work is to be able to apply a diffuse map and normal map to a set of nearby triangles. It can't be a property of the triangles themselves, however, because the level editor lets you create and delete triangles willy nilly, and you wouldn't want to have to re-create your decals constantly if you made a wall taller or shorter, or just subdivided some triangles and added some vertex noise.
So, I settled on representing the decals as oriented bounding boxes. This is also how the triggers work, so most of the code was already in place. The idea is that if one of the decal bounding boxes intersects a set of triangles, one side of the decal box will define the UV mapping of the triangles the box intersects. This way the level author can make the decal bigger or smaller by scaling the box along the decal's UV axes, and make the decal apply to more or less triangles by varying the decal box's depth.
Here is a shot of the editor with a grey decal box intersecting part of a wall. The decal would be applied to the wall oriented and scaled to match the box.
The challenge then comes in to make the decal adjust when its underlying triangles change. I am a strong believer of first doing the simplest thing that could possibly work. It usually doesn't, but implementing something potentially complex in a simple way often leads to more insight into how to do it better, and it may be that you get a simpler design in the end then if you had tried to handle every corner case from the get-go.
In this spirit, the simplest possible plan would be to recreate the decals as needed per-frame, based on the geometry intersecting each box.
The next most complicated method would re-use the system I already have in place for the editor's render chunks. Render chunks are rectangular boxes of triangles that are culled and drawn together. The engine knows when triangles have changed, and invalidates their renderchunk, so a new one will be created at the next frame. This system requires only triangles that have been changed ( and their neighbors in the same chunk ) to be reassigned to a new chunk, speeding up editing considerably.
I could create a renderchunk for all decals, or instead just track the bounding box of each render chunk that has changed, and only re-create those.
Actually, the best optimization idea is to simply track the bounding box of all triangles that have changed since last frame, and do a simple bbox check to avoid re-projecting decals that haven't changed.
The next topic is how to blend the decals in properly. My game engine using compositing to blend them perfectly with lighting & shadows, but my editor's renderer is much simpler and more abstract. I really don't want to add compositing to the editor, and it already lacks shadows, so making the decals look perfect in the editor is a non-goal. You already have to run the level in the game to see what it really looks like.
So, I suppose I will simply draw the initial triangles normally, with lighting, and do a blend with the decal triangles afterwards, lighting the decals as well.
Another approach would involve re-rendering the decals with their original diffuse & normal textures & the decal's diffuse & normal textures, in one pass, but that won't work for more than one decal blended over the same triangles.
I'll post some shots once I have it working.