Have a look at the picture I attached.

Clearly, I can see there are bands of color on the sphere instead of smooth gradual variation. This problem gets more apparent as I move away from the sphere. I checked which component of my lighting equation causes the problem and it's the computed intersection point between a ray and the sphere. I've lookat at various ray-tracing tutorials but nobody mentions this. Have you ever expierenced this problem? How would you handle it? I'm really kind of surprised that I get almost "regular" rectangulars on the sphere (in screen-space).

Ray-sphere intersection:

inline bool IntersectionRaySphere(
    const Vector3& rayStart,
    const Vector3& rayDir,
    const Vector3& sphereCenter,
    float sphereRadius,
    Vector3& intersectionPoint,
    float& distance)
{
    float a = Dot(rayDir, rayDir);
    float b = Dot(rayDir, 2.0f * (rayStart - sphereCenter));
    float c = Dot(sphereCenter, sphereCenter) + Dot(rayStart, rayStart) - 2.0f*Dot(rayStart, sphereCenter) - sphereRadius*sphereRadius;
    float delta = b*b + (-4.0f)*a*c;

    if (delta < 0.0f)
        return false;

    delta = Sqrt(delta);
    float t = -0.5f * (b + delta) / a;

    if (t > 0.0f)
    {
        distance = Sqrt(a) * t;
        intersectionPoint = rayStart + t*rayDir;
        return true;
    }
    else
    {
        return false;
    }
}

When I output, as color, the intersection point computed with this function, I get banding as well so apparently the ray-sphere intersection is be flawed.

Now the code that generates rays:

Vector2 ViewPlaneSize(float fov, float aspect, float viewPlaneDistance)
{
    Vector2 size;

    size.y = 2.0f * viewPlaneDistance * Tan(fov / 2.0f);
    size.x = aspect * size.y;

    return size;
}


class Camera
{
public:
    void SetLookParams(const Vector3& position, const Vector3& target, const Vector3& up)
    {
        this->position = position;

        lookAtTransformInverse = MatrixLookAtRH(position, target, up);
        Invert(lookAtTransformInverse);
    }

protected:
    Vector3 position;
    Matrix lookAtTransformInverse;
};
 
 
class PerspectiveCamera: public Camera
{
public:
    void SetPerspectiveParams(int width, int height, float fov, float nearPlaneDistance)
    {
        this->nearPlaneDistance = nearPlaneDistance;

        float aspect = (float)width / (float)height;

        Vector2 size = ViewPlaneSize(fov, aspect, nearPlaneDistance);

        Vector2 p1 = VectorCustom(0.0f, -size.y / 2.0f);
        Vector2 p2 = VectorCustom((float)height, size.y / 2.0f);
        yCoeffs = SolveLineCoeffs(p1, p2);

        Vector2 p3 = VectorCustom(0.0f, -size.x / 2.0f);
        Vector2 p4 = VectorCustom((float)width, size.x / 2.0f);
        xCoeffs = SolveLineCoeffs(p3, p4);
    }

    void Ray(int x, int y, Vector3& rayStart, Vector3& rayDir)
    {
        rayStart = position;

        //

        rayDir.x = xCoeffs.x*(float)x + xCoeffs.y;
        rayDir.y = yCoeffs.x*(float)y + yCoeffs.y;
        rayDir.z = -nearPlaneDistance;

        NormalizeIn(rayDir);

        rayDir = Transform(rayDir, lookAtTransformInverse);
    }

protected:
    float nearPlaneDistance;
    Vector2 yCoeffs;
    Vector2 xCoeffs;
};

SetPerspectiveParams computes line coefficients that are later used in Ray function to calculate the point on the "near plane grid" that the ray will go through. This ray is computed in view space and then transformed to world space.

I don't think the problem is precision as I rewrote the ray-sphere intersection function to use doubles to no effect. The scale should also be fine. The sphere in the pictures has radius of 1.0.

I've just created ortho camera ray generation code and to make it as simple as possible I place the camera in (0,0,0) looking down the negative Z axis. Here's the code:

    void Ray(int x, int y, Vector3& rayStart, Vector3& rayDir)
    {
        rayStart.x = 0.02f*(x-400);
        rayStart.y = 0.02f*(y-300);
        rayStart.z = 0.0f;

        rayDir.x = 0.0f;
        rayDir.y = 0.0f;
        rayDir.z = -1.0f;
    }

The screen resolution is 800x600 hence 400,300 offset.

This picture gives really horrible banding. See the pic.

Got it!

My buffer is 800x600 and I was displaying it on OpenGL texture with NEAREST filtering. The problem is that I ran everything in a window which is smaller in size than 800x600 (due to the frame and caption bar). That caused that some lines (both some vertical and horizontal) of the generated image buffer were completelly skipped, leading to sudden jumps in lighting and excessive jaggies around the spheres.

I simply saved to a PNG file my buffer and there everything was ok, cause the resolution was fixed to 800x600 and the generated buffer could be copied one-to-one.

Also, for real-time debugging I simply switched OGL texture filtering to LINEAR.