I continued my research and found out that I need to project my vertices of the quad to the corners of the farplane by multiplying
it with the inverse of my perspective matrix in the vertex shader.
So far so good.
vViewRay = vec3(vec4(VertexPosition, 1.0) * invPerspectiveMatrix).xyz;
Displaying viewray in the fragment shader looks like this:
I think it looks just fine with exception of the missing blue values. This could possible be the result of the z-value of my quad all being 1.0 before the inverseprojection.
So it is gone after that. Putting z of the far clipping plane (z=1000); in, didn't change anything.