You're using vTexel.y twice. ^^
Oh wow, thanks a lot, I really didn't notice that at all. Was not the only part of the problem though, I also have been passing in the wrong texels (target texture size, instead of input texture), and that almost fixes it. There is still one catch though, since I downscale the textures by the factor of 2:
In that jump from 5x3 to 2x1 both x and y looses one pixel, which still uses only the pixels in the middle area of the screen. Though its a lot better already since the rest of the scene is at least taken into consideration up until this point, but its still not perfect. Any ideas for that?