The first question I asked myself was "How does one Unit Test a renderer?" That seems fairly obvious; you generate pictures and compare them. What to compare them with? I decided I would initially generate a picture, manually verify that it is what I expect, then archive it, and from then on compare with the existing picture. This works nicely for refactoring, but the initial process isn't quite 'Testing'. However, I don't see a way around it really. Since especially the DS renderer will be low-resolution I've decided to keep these images smallish at 128x128 resolution.
To compare the images I initially wrote my own comparison routines which compared the images pixel-by-pixel. This worked well until I moved development onto a different PC. At that point the images generated by the renderer suddenly weren't matching the stored images. After looking at the image closely it turned out that in the WhiteTriangle image below the horizontal and vertical edges were each extended 1 pixel, so the edge at 45deg was one pixel to the right and one pixel up. This totally broke my image comparison. First I thought one of the images was incorrect because of driver issues, but after posting on the forums Brother Bob mentioned something about OpenGL allowing variations. After a look in the latest OpenGL spec it turns out he's correct:
Quote:OpenGL 3.0 Spec, Chapter 2.1
The GL is designed to be run on a range of graphics platforms with varying graphics capabilities and performance. To accommodate this variety, we specify ideal behavior instead of actual behavior for certain GL operations. In cases where deviation from the ideal is allowed, we also specify the rules that an implementation must obey if it is to approximate the ideal behavior usefully. This allowed variation in GL behavior implies that two distinct GL implementations may not agree pixel for pixel when presented with the same input even when run on identical framebuffer configurations.
What to do now? Clearly I needed to compare the images differently, allowing for some variation. Since I wasn't interested in writing image comparison routines I took a look on the net. Happily I found a solution: PerceptualDiff. This GPL library does some kind of perceptual-based comparison on two images. After writing the interface between my code and PerceptualDiff, the 'errant' PC happily reports the images are identical. I'll need to keep an eye on it as I don't know how different images need to be to trigger a failed comparison, but so far it's working nicely.
The process has already uncovered a latent bug in my file loading routines which has been there for years, and numerous bugs in my own image comparison code (which is scrapped now anyway). Overall I'm quite happy with how things are going. Now the process is working, I'm going to attempt to write at least one test a day (on average). This should get the total test count up pretty quickly, and hopefully the test-coverage will go up with it.
Here are the first few images I'm testing against, so far only with a basic OpenGL 1.5 era renderer.
Default clear color (set inside each renderer to be dark gray)
Custom clear color
A simple triangle (both Projection and Model/View matrix are set to identity)
A translated triangle (Projection matrix is still identity)
A scaled triangle
A rotated triangle
Combinations of the above matrix operations are nice simple tests to add in my daily additions.
An isocahedron (a 3d mesh, mostly in preparation of adding lighting in the next tests)
