So Kaptein, I can add mipmaps to a FBO texture, and then every time level 0 is updated, the texture's highest mipmap is updated (and it's 1px x 1px), so I can just sample that for the average brightness.

So since the FBO uses mipmaps, can I somehow force it to use maybe it's second mipmap for redrawing to another FBO to downscale it?

Koehler, so basically you suggest just drawing the scene twice to a smaller and then a larger FBO. I'll give it a shot, thanks.

OK, so generally you draw the scene twice, once at full and then low res rather than shrinking the full res.
I'll give it a shot then, thanks.