Sign in to follow this  
RedDrake

Real time global illumination on GPU - per pixel ...

Recommended Posts

Now that title sounds like I really got ahead of myself but if my idea works out it will be "that huge" :D ... So all of you know what a deferred render is, and now what if you apply deferred rendering process to the shadow map render ? 1.) You essentially store all the gbuffer data from light POV in a g-buffer 2.) Next you do standard forward render or deferred light pass and you include first order reflections from the environment by sampling that light g-buffer for material properties 3.) This is a lot of work even for small res gbuffer but you could use montecarlo technique to sample, but there is an event better approach and that is mip-mapping - basically you can sample hierarchically the light gbuffer to find out average contribution and then go down the hierarchy and stop when average contribution is low, you would probably want to skip first and second level of hierarchy (the more you skip the more accurate your image would be but more time it would consume) Now there is one problem with this approach, and that is shadowing the reflected light, and you can do that by doing exactly what you would with normal shadowing - you render from camera that's near plane is equal to the far plane of the light camera and you render in direction of light, but you only draw shapes behind old z buffer - i don't know how you could implement this in hardware tough. You could get second order reflections by rendering the new gbuffer as you render the reflected shadow map and then reflect again and render the new shadow map up to infinity steps, and i doubt that low resolution would cause any problems since IMO that does not matter much when doing GI ... I've thought about this for some time now and I've been wanting to implement this as a tech demo for my portfolio but i think i'm better off checking if it can be done here first before spending a bunch of time doing a demo of the idea that won't work :) I think you can even use NV-CUDA if standard DX rendering won't work for the reflected shadowing, i just wish I have a NV dx card and not that ati overheating crap :( Anyway the larger your light gbuffer and the deeper you force your sampling in to the mipmap chain the higher res you get - it's a way to use rasterization instead of ray tracing for ray marching, it should provide benefits even for SW renderer or hybrids that can use software/cuda + gl/dx for high quality GI off line rendering ? So any thoughts why this approach would/would not work ? EDIT: Note that my response time to this thread might be long since i don't have a constant net connection, il be online for 1-2 h from now and then back in a week or so (don't get frustrated if i don't respond :D ) [Edited by - RedDrake on May 9, 2008 5:54:16 AM]

Share this post


Link to post
Share on other sites
I'm interested, I'd love to see a relative simple solution since I'm hurting myself with trying realtime GI (I tried by sampling with nodes per vertex, and now trying SH like the ATI demo does).

If you want it pixel-perfect, you could measure the incoming light for each pixel POV, basically by rendering the surrounding environment. This is how lightMap generators with radiosity can work. But of course, doing that per pixel is way too slow for now. Even when doing it in larger steps (pixel chunks/patches or something) its either very low-res, and/or still slow.

As for your approach, I don't quite understand how you "lit" pixels indirectly. From a light POV, you only the say the pixels that would normally be litten directly as well, just like you can make shadowMaps. If I understand you right, the G-Buffer data you use only contains data of the pixels that the light actually sees. How can you use that data to determine which light(s) / how much / from which direction, the unlit pixels should use? Probably I don't understand step 2 :)

Greetings,
Rick

Share this post


Link to post
Share on other sites
Quote:
Original post by spek
As for your approach, I don't quite understand how you "lit" pixels indirectly. From a light POV, you only the say the pixels that would normally be litten directly as well, just like you can make shadowMaps. If I understand you right, the G-Buffer data you use only contains data of the pixels that the light actually sees. How can you use that data to determine which light(s) / how much / from which direction, the unlit pixels should use? Probably I don't understand step 2 :)

You use standard GBuffer (camera view) and for each pixel you determine the reflective contribution by sampling all the pixels in the light visible range this will give you secondary reflections ? Basicaly for a pixel on the screen you march trough all the pixels light primary rays hit and calculate the reflection to that (screen) pixel (a reflection) using gbuffer data (in order to avoid going trough all pixels you use the upper mipmap trick)
The you should be able to figure out the parameters from gbuffer and calculate GI solution, i don't exactly know the math but you should be able to calculate the pixel world size (transform it in to a patch) by using its depth and its projection, this way the light gbuffer actually becomes a grid of patches with normals/diffuse/specular and all other gbuffer data and you can do the light transfer by using some GI light transport algorithm

[Edited by - RedDrake on May 9, 2008 7:02:53 AM]

Share this post


Link to post
Share on other sites
So if I understand you right, in the final pass you calculate the indirect contribution by checking out all the neighbour pixels. From previous passes, each pixel already has its diffuse/specular/depth(or xyz)/normal data stored in buffers, just like you would do with deferred lighting.

All the surrounding pixels act as lightsources, since they have a reflection and a normal that can be used to determine the light direction. You downsample the buffers in order to gain speed(far less neighbour pixels to check).

Correct? Probably not, but ifso, I've been thinking about something similiar as well, but there a couple of problems. First of all, you can't tell if something is blocking the light between neighbour pixel X and the receiver pixel Y. Imagine a screenshot with a chamber. The pixels from the left wall affect the pixels on the right wall, which is good. But what if there is a large pillar in the center? Unless you do some collision detection on the screenshot for each pixel, there is no way to tell something is blocking the light.

Maybe that is not so bad, since GI doesn't have to be 100% perfect (for now). And with enough bounces, the light from the left wall will eventually reach the wall on the right anyway. But there is another bigger problem. What about all the stuff behind the camera? The wall behind you reflects light as well, but its never rendered in the final pass, so you can't check its pixels as well.


Unless you do it a little bit different. Probably this is what you mean. You mentioned earlier that the G-buffer data is from a lightsource point of view, so basically you would have textures with diffuse/normals/specular/.../ for each light (which can consume quite a lot of memory if you have a lot of lights, but lets forget about that for now). You can check the buffers for each light, and project them in the final pass. For example, one light shines on the wall behind you. You won't render those pixels in the final pass, but its data is stored in the buffers from that light. When doing the final pass, you loop through the pixels in that buffer, check the direction between the receiver pixel and those from the light-buffer. This would add light, depending on the pixel normal, its reflected color, and distance between the 2 pixels.

Collision detection is still difficult though. Imagine all the lighting is done behind the camera. Somewhere behind you, a light is shining on the wall behind you. Everything in front of you will catch the reflected light from the stuff behind the camera, which is good again. But what if there is a another wall between you, the light, and the wall that reflects:

north
------------x------------------
| |
| ^ camera, looking north|
| |
| ------------wall------ |
| L light, pointing south|
| |
------- wall, reflects L ------
south

When rendering pixel X, it will use the buffer from light L. In this case you have to, since the wall between L and the camera is not entirely closed. The problem is that the wall in between is never rendered, so it won't be blocking any light.

Interesting method nevertheless... Little bit difficult to imagine how it exactly performs, I would say just try it :)

Greetings,
Rick

Share this post


Link to post
Share on other sites
Quote:
Original post by spek
Correct? Probably not, but ifso, I've been thinking about something similiar as well, but there a couple of problems. First of all, you can't tell if something is blocking the light between neighbour pixel X and the receiver pixel Y. Imagine a screenshot with a chamber. The pixels from the left wall affect the pixels on the right wall, which is good. But what if there is a large pillar in the center? Unless you do some collision detection on the screenshot for each pixel, there is no way to tell something is blocking the light.

I noticed this too - i was thinking of modifying the shadow map algorithm in some way to achieve the shadowing tough - basically you render the shadow map as i said below and then you find where your reflected ray hits the map and is this depth same (near) the screen pixel depth

Quote:
Original post by RedDrake
Now there is one problem with this approach, and that is shadowing the reflected light, and you can do that by doing exactly what you would with normal shadowing - you render from camera that's near plane is equal to the far plane of the light camera and you render in direction of light, but you only draw shapes behind old z buffer - i don't know how you could implement this in hardware tough.

this shadowmap quality would be even worse than the light gbuffer but unless its resolution is increased but i think it's safe to ignore artifacts as you said gi does not have to be 100% :)

Share this post


Link to post
Share on other sites
Quote:
Original post by spek
The wall behind you reflects light as well, but its never rendered in the final pass, so you can't check its pixels as well.

actually no - if light points towards the wall it will be included in light gbuffer like you said latter

Quote:
Original post by spek
Unless you do it a little bit different. Probably this is what you mean. You mentioned earlier that the G-buffer data is from a lightsource point of view, so basically you would have textures with diffuse/normals/specular/.../ for each light (which can consume quite a lot of memory if you have a lot of lights, but lets forget about that for now).

there is no need to keep all gbuffers, iteratively render the lights with deferred shading so you can just allocate one gbuffer and use it for all lights

Share this post


Link to post
Share on other sites
I've been thinking along the same thoughts for some time now.
There are some limitations though:
* Only the first light bounce is modelled.
* The light map needed must cover the whole scene, something that isn't practical.
Normally you'd just render what's in the frustum into the map and even then you need some fancy non-linear projection to get good texel to pixel ratio.
The problem is that objects that are outside the frustum ain't going to reflect any light until they are visible, this will lead to popping.
Imagine a red ball on a white floor, when the ball is outside the frustum (not drawn on the map) the floor won't get any red light but as soon as a few pixels of the ball is visible the floor will turn reddish.

One approximation would be to limit the radius of the reflected light and attenuate it (something that you need to do anyway) similar to what you do for fixed range point lights.
If you apply a limit of let's say 10 meters (so that samples 10m away has zero intensity) then you'd just need to "grow" your frustum 10 m and render the shadow map so that the expanded frustum is visible.
Another benefit of doing it this way is that you decrease your sample radius.

My 2c.

Share this post


Link to post
Share on other sites
@eq the detail of the lightmap is less important since second reflections are a lot less visible than the first ones (direct lights) - you can even render the second reflections alone in to a separate RT and blur them before blending to screen in order to avoid aliasing

Quote:

One approximation would be to limit the radius of the reflected light and attenuate it (something that you need to do anyway) similar to what you do for fixed range point lights.
If you apply a limit of let's say 10 meters (so that samples 10m away has zero intensity) then you'd just need to "grow" your frustum 10 m and render the shadow map so that the expanded frustum is visible.
Another benefit of doing it this way is that you decrease your sample radius.

This is what i had in mind originally , i don't see how it's a problem :) You would only need to handle big lights like this, and in indoor scenes it's not a big issue since most of the light don't exceed the view frustum size by much IMO
In outdoor cases you would probably scale the light frustum like you said (10 m is rather high for second reflections imo ?)

Anyway i would definitely not use the same result for shadow mapping, as you said thats just a waist of space and resolution would be crappy, but i really think it would not affect visual quality that much

as for only one bounce - i think it can be handled :
you render standard "light map" from light POV
then you calculate the "inverse" POV - construct a camera that points towards the origin of original POV and the camera "near" plane must be equal to the original light POV "far" plane, and you render your scene - you would also use the "lightmap" depth buffer as the initial depth buffer and only render objects behind it, you would have to do some wired fish eye projection because reflected rays can go 180deg
this would give you 2nd reflections but it can't be handled by standard rendering it would have to be done in software

Still i think the approach is worth it, it would probably generate better results than screen space ambient occlusion and the cost will not be that far off IMO

Share this post


Link to post
Share on other sites
Quote:

actually no - if light points towards the wall it will be included in light gbuffer like you said latter


In the ASCII picture, the light is pointing south, to the wall at the bottom. Well, try to imagine, I couldn't make a better picture :). Anyway, the camera and light L are pointing in the opposite directions. The wall in between will never be rendered since its in none of the view frustums. Not in the final view, not in a shadowMap and/or G-buffer from light L. Therefore I guess its impossible to involve the wall for the collision detection. It will alot of light in this case, but you can't tell it from the light or final screen.

I guess the approach will work in quite alot situations. Not 100% physically correct though, but that is for most of the GI techniques. Its an approximation. However, there are cases where the light leaks through objects if the objects are outside the camera and light frustum. Maybe not that much of a problem, as long as the blocking objects aren't very big. But imagine the ASCII situation, but then with a couple of skyscrapers. Maybe the users won't even notice it, its hard to tell if the indirect lighting is correct or not (you first notice the direct lighting, ambient is an extra to make realism). But you might need to be aware of light popping in when the camera or light frustum changes, because that is an artifact that users will definitely notice.


Quote:

there is no need to keep all gbuffers, iteratively render the lights with deferred shading so you can just allocate one gbuffer and use it for all lights


True. But in case you have quite alot lights, it can slow down the rendering pretty much because you need to switch between FBO's and depth buffers with different resolutions all the time. Its either speed or memory that has to suffer. Also if you want multiple bounces to increase realism, you might need to store the data somehow.

But... as long as you don't have a huge collection of lights and/or keep low resolutions, it won't be that much memory I think. By the way, have you considered storing all light data in a texture array or 3D texture? Might be usefull, since you don't have to swap textures for each light in that case.

When it comes down to performance, I think your technique can run pretty fast, as long as there are not too much lights. The nice thing about other GI technqiues using nodes, is that you don't need to update all the nodes per frame. In case the framerate is low, just do less nodes. Your situation is a little bit different. If you store the g-Buffer data, you don't have to update all lights for each frame as well. But the processing of the lights in the final pass(es) is still quite a heavy operation, for each light and frame again. You could (greatly) reduce it by limiting the lights that affect a piece of geometry. The user won't be able to tell if 10 lights are indirectly shining on a wall anyway. Just pick the strongest lights.

Succes!
Rick

Share this post


Link to post
Share on other sites
Quote:
Original post by Arkhyl
There's a paper that looks similar to your method : http://www.vis.uni-stuttgart.de/~dachsbcn/download/rsm.pdf , maybe it could help you.


Bummer, looks like it's not such a new idea after all :D, still the aproach is slightly different, but will definetly help me out writing my implementation :D

Share this post


Link to post
Share on other sites
Quote:
Original post by spek
Quote:

actually no - if light points towards the wall it will be included in light gbuffer like you said latter


In the ASCII picture, the light is pointing south, to the wall at the bottom. Well, try to imagine, I couldn't make a better picture :). Anyway, the camera and light L are pointing in the opposite directions. The wall in between will never be rendered since its in none of the view frustums. Not in the final view, not in a shadowMap and/or G-buffer from light L. Therefore I guess its impossible to involve the wall for the collision detection. It will alot of light in this case, but you can't tell it from the light or final screen.

i am working on a solution for this (i have planed/hinted it already but i will have to look in to it some more ... basically its making a shadow map for reflected rays

Quote:

But... as long as you don't have a huge collection of lights and/or keep low resolutions, it won't be that much memory I think. By the way, have you considered storing all light data in a texture array or 3D texture? Might be usefully, since you don't have to swap textures for each light in that case.

only dx10 can render to 3d textures right ?

Quote:

When it comes down to performance, I think your technique can run pretty fast, as long as there are not too much lights. The nice thing about other GI techniques using nodes, is that you don't need to update all the nodes per frame. In case the frame rate is low, just do less nodes. Your situation is a little bit different. If you store the g-Buffer data, you don't have to update all lights for each frame as well. But the processing of the lights in the final pass(es) is still quite a heavy operation, for each light and frame again. You could (greatly) reduce it by limiting the lights that affect a piece of geometry. The user won't be able to tell if 10 lights are indirectly shining on a wall anyway. Just pick the strongest lights.

Succes!
Rick

Exactly the way it's done when culling standard lights for performance :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this