[EDIT] I've revised this a lot since this original post - scroll down to the post with the larger JPEG for the final algorithm.
[/EDIT]


I started trying out a bokeh idea last night inspired by Lost Planet. The approach I came up with is really fast compared to non-fake bokeh effects, but I'm facing some stability/flickering problems -- if anyone's got any hints/advice/ideas of their own, I'm all ears ;)

I'm not actually doing any DOF-blurring at the moment, I'm just trying to isolate bright pixels and apply a fake out-of-focus bokeh effect to these bright pixels only (i.e. the flaring in the shape of the camera's aperture). I should be able to add a traditional blurred-DOF effect on top of this later.

In other words, I'm not doing the blur demonstrated by this picture - I'm just trying to capture the effect that happens to the specular highlights on the glass when it goes out of focus:


Here's my recipe:

Short version:
1) Find bright pixels
2) Draw quads over them textured with hexagons/circles (multiplied with the bright pixel's colour).

Long version:
1) Render the scene.

2) Downsize the frame-buffer 16 times. Each down-sized pixel covers 256 (16x16) original pixels.
2.1) Each pixel in the down-sized buffer samples an 8x8 grid (offset half a pixel so that each sample is the average of 2x2 original pixels -- 8x8x4==256).
2.2) These 64 samples are compared to find the brightest one
2.3) The brightest sample's RGB, and it's x/y sub-offset are written to the downsized buffer (x/y are range 0-15, or 4 bits each, so they're packed into the 8 bit alpha channel).
2.4) Some kind of 'threshold' is applied (see below recipe) to determine if we actually want to apply fake bokeh to this bright pixel.

3) For each down-sized pixel over a certain brightness threshold, draw a quad at that pixel's center position, offset by it's packed x/y offset value.
3.1) The quad is textured with a "bokeh texture" - e.g. a circle/hexagon/octagon, depending on your "camera".
3.2) The quad can be scaled by the luminosity of the sample, or based on depth so out-of-focus points create bigger bokeh circles.
3.3) The "bokeh texture" is modulated with the samples RGB value and blended with the frame-buffer.


My "threshold" is currently:
1) The entire 16x16 block isn't bright (just a small bright point is present) - the brightest in the block is a certain luminosity-distance from either the dimmest or the average (not sure which to use yet).
2) The brightest is actually quite bright.
or in GLSL:
Here's some programmer-art pictures, using additively-blended pentagons and a white border so you can easily see the quads:

My problem is that it's too unstable - you can get good results in still pictures, but when the camera moves, different pixels are chosen to 'flare up' each frame, resulting in horrible flickering of these "bokeh sprites". You can see above that as I rotate the camera, the guy's white collar goes from getting 3 flares, to none, back to 2.

[Edited by - Hodgman on July 18, 2010 8:41:46 AM]

I think the reason: if the camera moves, at least one row or column will change, so 16 pixels at a time. that's 1/16 of the whole thing.

I would try with a diamond shape instead: rotate the image with 45' then downsample.
or 40', maybe an arbitrary angle would be better. (Sorry, I haven't looked into the thing in detail, just a quick idea)

Or the good old interpolate between frames. So the bokeh will be a bit sluggish, but maybe that's fine.

Quote:Original post by STufaro
The blurs happen to the whole image, and you _notice_ the specular highlights because they most obviously get blurred against a dark background.

Yeah this is purely for speed - I can't operate on every pixel, so I'm trying to prioritise the bright ones to recieve the effect because they're the most noticable. However, I tried altering it so that if a 'bright' pixel can't be found in one of my 16x16 buckets, I still perform the effect anyway and it probably looks better than ignoring the dark pixels.

Quote:The size of the blur is a function of your depth of field and your distance only, not the luminosity, if I remember right (correct me if I'm wrong though!).

Yeah using luminosity to scale the highlights actually looks pretty wrong - it should be pretty simple to tie it into the same formula that the traditional DOF blur will use to determine how out-of-focus a point is (once I add regular blurring!). However, because I'm ignoring 99% of the image during this effect (one pixel blurs out per 16x16 block) I might have to add a subtle opacity/size bias towards the bright pixels still to get it to look right.
Here's some pics with a "bad bokeh" (hard edge) texture, notice the dark circles in the second pic look a bit strange:

Quote:This sounds simple, and I am not a shader pro or familiar with GLSL, but is there a way you can operate on the specular value of your stuff (input to the pixel shader maybe? I'm not sure where that comes together, to be honest!)

In my lighting model, I actually can seperate the specular contribution from the diffuse lighting when doing post effects, thanks! Prioritising 'specular' might be a good idea -- or maybe even prioritising diffuse over specular because it's less prone to change due to camera movements? I'll have to experiment with this.

Quote:One other thing--I notice your blurs from the floor are getting drawn in front of your character. Just out of curiosity, how are you going to stop that from happening?

Yeah I haven't tackled that yet - I do know the depth of the "bokeh-ing" pixel, and the depth of the things it's drawing over though, so I should be able to do a depth-test of some kind here.

However, I do have the problem that these 'bokeh sprites' aren't sorted with regards to each other - so a distant 'bokeh blur' can still draw over the top of a close one... Anyone know of an easy GPU-side sorting method? :(
[EDIT]Actually, the problems with inter-sprite-sorting and the problems with dark pixels just go away if they're rendered additively instead of trying to kludge around with alpha blending - maybe I can do a separate render of just bokeh on a black background like below, and then composite the result with the scene later:

Quote:Original post by szecs
Or the good old interpolate between frames. So the bokeh will be a bit sluggish, but maybe that's fine.

This worked pretty well - I added interpolation over a few frames just by enabling alpha blending with a constant alpha value when writing into the down-sampled buffer (I also had to re-arrange my packing to put x/y sub-offsets into two channels, because the packed format doesn't blend properly :(). Unfortunately it doesn't remove the source of the 'popping' - it does make it a lot more bearable though =)


While I was playing around with this algorithm, I also realized that it might be useful for painterly-rendering - here's a picture with a 'paint blob texture' instead of a 'bokeh texture' - with a whole selection of different paint blob types and a way to select suitable ones you might be able to get an interesting art style from it:


[Edited by - Hodgman on July 13, 2010 8:52:40 AM]

I'm leaning more towards trying to create an actual real-time "lens blur" instead of just a few "bokeh sprites" now.

I've made a few changes that produce much better quality images:
* Removed the 'interpolate between frames' trick, it's actually causing more flicker than it's solving now.
* The sub-offset calculation is a weighted average of all sub-offsets (weighted by their luminance), instead of just using the offset of the brightest pixel. Location shifts are much more smooth.
* Carefully making sure all energy from the original image is conserved.
** Using the average luminosity of the source 16x16 block, not just the brightest pixel.
** The average luminosity of the "bokeh/lens texture" is also calculated, and then both these average lum's are used as follows:
Below is an example with a hexagonal lens sprite. As in my last post, I'm additively drawing all these "lens sprites" over a black background (instead of blending over the original image). An interesting side effect is that it causes a slight vignetting effect (because no light is scattered in from outside the screen), but I kind of like the artifact ;)


[Edited by - Hodgman on July 18, 2010 8:16:48 AM]

I know I'm kind of treating this post like a dev-blog, but I've pretty much completed the effect and may as well post the final ingredients ;P


The idea is to draw a picture of what the camera's lens aperture looks like (I'm using the texture of a hexagon below), and then use that to implement efficient lens-blur in screen-space by gathering chunks of light and then scatter those chunks by drawing textured-quads using VTF.



1) Divide the screen into 16x16 chunks. Calculate the average colour of these chunks, and a an average screen-pos weighted by luminosity (the "sub-offset"). This is so the sprites don't move in 16px jumps, but instead move smoothly across the screen.
You can of course use numbers other than 16, but this seems to be a good balance between speed and quality (plus it means you can pack the sub-offset into 8 bits if you wanted to).

2) Calculate a "DOF mask" for your scene. I used 8 bits as a near-blur-mask, 8 bits for far-blur-mask and 16 bits for linear depth.
I'm using two separate masks so in the future I can dilate the near-blur mask to get better blurry edges on close objects.

3) Additively blending to a black surface, draw one 2D quad at the location of each chunk (thats (width/16)*(height/16) quads).
In the vertex shader:
Offset the quad with the chunks "sub-offset".
Fetch the linear-depth value at the center of the quad (pass to pixel shader).
Color the quad with the chunks average color.
Use the combined DOF masks as an alpha value, and to scale the average color.
Use the combined DOF masks to scale the quad (higher near/far blur mask value means bigger quads).
In advance, measure the average luminosity of your lens texture, make sure you're scattering the right amount of energy by scaling your average up by the chunk-size, then down by the sprite-size and luminosity (code in previous post). I also clamp this energy multiplier to 0.0/1.0 so small sprites don't 'explode' in brightness.

In the pixel shader:
Fetch the linear-depth value at this pixel, compare to the one passed in from the pixel shader. Use smoothstep to create soft edges where this sprite intersects closer geometry.
Fetch the 'lens texture' and multiply with the color passed in from the vertex shader.

4) Compose the render from (3) over your scene using pre-multiplied-alpha blending.

Here's the results:

I agree with everything mokaschitta said! It looks really interesting. I'm pretty interested in your painterly effect - when your camera moves, does the screen still "pop", or did the weighted luminance fix it? I'd imagine that would be a very neat effect, but very hard on the eyes if it pops too much.

Also, even though I know it's not physically accurate - have you tried using the bokeh in a different way? It'd be very cool to see something this applied to a SSAO pass.