Hey guys,

Are lightmaps still the best way to handle static diffuse irradiance, or is SH used for both diffuse and specular irradiance now?

Also, do any modern games use direct light in lightmaps, or are all direct lighting handled by shadow maps now?

Finally, how is SH usually baked?

Thanks!

4 hours ago, KarimIO said:

Are lightmaps still the best way to handle static diffuse irradiance, or is SH used for both diffuse and specular irradiance now?

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

For specular you are typically out of memory to store high detailed enviroment at every lightmap texel, that's why a sparse set of high resolution enviroment probes is common for this. (A 3 band SH has less detail than a cubemap with 6 * 2*2 texels, a 2 band SH is similar to 6*1 texel cubemap - not enough for sharp reflections)

In contrast, sampling diffuse indirect lighting from a sparse set of enviroment probes would be much worse than using lightmaps. Lightmaps have high resolution in the surface so they can store accurate local details, like shadows and darkening in corners. High resolution directional information is not necessary.

But there are many games that use either sparse lobes or dense lightmap approach for everything, others use sparse 3D texture bricks instead 2D lightmaps, many do not use precomputed lighting at all.

4 hours ago, KarimIO said:

Finally, how is SH usually baked?

Typically you generate n rays evenly distributed to any direction, for each ray calculate the incoming light (e.g. by path tracing), and sum up. Finally you normalize SH and done. 

So you could do something like contribute each texel of a cubemap to a SH as well, just calculate direction to each texel and a weight (texels at corners are more dense than at a face center, the weight should compensate this effect)

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

6 hours ago, JoeJ said:

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

But then this isn't a lightmap, this is just SH projected onto surfaces

6 hours ago, JoeJ said:

For specular you are typically out of memory to store high detailed enviroment at every lightmap texel, that's why a sparse set of high resolution enviroment probes is common for this. (A 3 band SH has less detail than a cubemap with 6 * 2*2 texels, a 2 band SH is similar to 6*1 texel cubemap - not enough for sharp reflections)

Yeah I was specifically talking about irradiance, or indirect light.

6 hours ago, JoeJ said:

In contrast, sampling diffuse indirect lighting from a sparse set of enviroment probes would be much worse than using lightmaps. Lightmaps have high resolution in the surface so they can store accurate local details, like shadows and darkening in corners. High resolution directional information is not necessary.

Okay so this kinda does answer my major part of the question.

My current plan is to use:

CSM + Regular Shadow Maps for direct light shadows, normal-mapped lightmap (Valve's technique, but with only two targets: ambient color and direction) for indirect diffuse for static geometry, SH for indirect specular on all geometry + indirect diffuse on dynamic geometry, "The Last Of Us"'s occluders for indirect dynamic shadows, and parallax cubemaps, planar reflections, and SSAO for direct specular (planar reflections being mostly used for water).

I just want to verify that these techniques are reasonable and won't cause issues.

6 hours ago, JoeJ said:

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

I saw this, but from what I understand, spherical gaussians are just broken down SH (where directions are not intrinsically calculated, rather they're provided). They don't seem to be much of a solution.

1 hour ago, KarimIOAH said:

7 hours ago, JoeJ said:

You can have lightmaps where each texel is SH instead just a single color. Advantage would be directional information, so you can apply indirect light to normal mapped surface properly.

But then this isn't a lightmap, this is just SH projected onto surfaces

The SH data already is generated from the surface, so no need to projection. Your original question was lightmap or SH, but you can store SHs in the lightmap. (which would look better than just ambient with primary light direction, but if that's worth the memory depends on the game.)

1 hour ago, KarimIOAH said:

My current plan is to use:

CSM + Regular Shadow Maps for direct light shadows, normal-mapped lightmap (Valve's technique, but with only two targets: ambient color and direction) for indirect diffuse for static geometry, SH for indirect specular on all geometry + indirect diffuse on dynamic geometry, "The Last Of Us"'s occluders for indirect dynamic shadows, and parallax cubemaps, planar reflections, and SSAO for direct specular (planar reflections being mostly used for water).

I just want to verify that these techniques are reasonable and won't cause issues.

That's a large collection of hacks... of course it will cause issues i guess, 

But it looks pretty state of the art to me so see how it goes... (I do not understand the distinction between SH indirect specular and cubemaps for reflections - you might want to explain in more detail.)

2 hours ago, KarimIOAH said:

 

8 hours ago, JoeJ said:

There are other options than SH, e.g. Spherical Gaussians (https://mynameismjp.wordpress.com/2016/10/09/new-blog-series-lightmap-baking-and-spherical-gaussians/), low res cube maps (Valves Ambient Cube) or sphere maps.

I saw this, but from what I understand, spherical gaussians are just broken down SH (where directions are not intrinsically calculated, rather they're provided). They don't seem to be much of a solution.

I agree it does not matter that much but i see advantages of SG vs. SH. What matters is the idea to store the entire enviroment at dense locations on the surface to achieve highest quality, and to be able to calculate full BRDF from that data without the need to read from some cubemaps, do some screenspace raytracing, do some other traces in volume data, and whatever other hacks we come up with. 

1 hour ago, JoeJ said:

The SH data already is generated from the surface, so no need to projection. Your original question was lightmap or SH, but you can store SHs in the lightmap. (which would look better than just ambient with primary light direction, but if that's worth the memory depends on the game.)

Makes sense. The only annoying bit is that I think you'll store half the information aiming into the texel rather than away from it, which seems like a waste.

1 hour ago, JoeJ said:

That's a large collection of hacks... of course it will cause issues i guess, 

Well all of realtime CGI is hacks But what do you mean more specifically? Do you see any problems with it?

1 hour ago, JoeJ said:

(I do not understand the distinction between SH indirect specular and cubemaps for reflections - you might want to explain in more detail.)

Well, you've got both diffuse and specular, each based off direct and indirect light. For specular, I want to handle direct with SSAO + Cubemaps, and ambient with SH. Because low-res cubemaps aren't very good for ambient.

20 minutes ago, KarimIOAH said:

Makes sense. The only annoying bit is that I think you'll store half the information aiming into the texel rather than away from it, which seems like a waste.

Yes, that's one argument for SG (you could align all its directions to the hemisphere) or tiny spheremaps (which is what i do but in realtime: 4x4 texel enviroment per lightmap texel. Even 2x2 would be better than primary light direction if the scene is colorful.)

It's also possible to modify the SH idea so it only covers the hemisphere, but i guess the win is not worth the effort to fully understand the math

31 minutes ago, KarimIOAH said:

For specular, I want to handle direct with SSAO + Cubemaps, and ambient with SH.

I'm still confused. I would define 'direct specular' as the reflection of light sources, which can not be stored in SH practically.

I would define 'ambient specular' as the reflection of the lit scene excluding light sources.

So how and why would you blend between cubemap and SH representation of the same thing? Why SH at all and not just cube mip? (Or 4 samples in higher cube mip, to fight the cubes axis-alignment.)

Maybe this indicates a flaw in your design, but maybe i just miss something (i know more of generating the data than using it.)

53 minutes ago, JoeJ said:

Yes, that's one argument for SG (you could align all its directions to the hemisphere) or tiny spheremaps (which is what i do but in realtime: 4x4 texel enviroment per lightmap texel. Even 2x2 would be better than primary light direction if the scene is colorful.)

It's also possible to modify the SH idea so it only covers the hemisphere, but i guess the win is not worth the effort to fully understand the math

I don't think you'll be saving much with SG, considering you still need to include the direction.

Anyhoo, you were right, I just got confused I'm actually working on something else so I didn't get my facts straight, but yeah, SSAO/Cubemaps for indirect specular, SH for indirect diffuse, maybe with an option for lightmaps instead. Maybe in addition, some options for dynamic indirect diffuse to be calculated, whether through voxel-GI or otherwise.

Thanks, JoeJ!

11 hours ago, KarimIOAH said:

SSAO/Cubemaps for indirect specular, SH for indirect diffuse, maybe with an option for lightmaps instead. Maybe in addition, some options for dynamic indirect diffuse to be calculated, whether through voxel-GI or otherwise.

Sounds good. I liked CryEngines dynamic voxel GI approach - IIRC it's about reflective shadowmaps using voxels for occlusion, so still expensive but voxels require only one bit. Details on their manual pages.

On the precalculated side this one is interesting (static geometry but dynamic lighting): https://users.aalto.fi/~silvena4/Projects/RTGI/index.html

I was considering only using those kinds of approaches on smaller areas, if at all (RSM, VoxelGI, and the like would be supported in volumes) because they're so expensive. But I'll check out your link, thanks