Eat Your Dogfood. It's Tasty.
So a few days ago I shipped Release 13 of the Epoch programming language.
Turns out, that was a bad idea.
R13 has some seriously aggressive features in it. There's support for native algebraic sum types, type aliases (both "weak" in the sense of C/C++ typedefs and "strong" in the not-weak sense), and - most interestingly for me - templates. Moreover, R13 includes a lot of back-end refactoring to support all these features in a clean and relatively nice way.
As it happens, all of the unit tests for R13 functionality pass. So I was comfortable firing it out the gate, thinking that it was covered well enough that I could start writing some test software against R13's compiler and get some cool demo programs done.
One of the demo programs I wanted to write is a simple raytracer. This is for two reasons: first, a good raytracer can be a compact piece of elegant code. A bad raytracer can be a nightmare, as I know from firsthand experience. If Epoch allows the program to be written in a clean and elegant way, that's good; if the implementation has to be messy and gross, that's bad.
The other reason is it'll give me a nice foundation for improving the native-code JIT system, which will be crucial for keeping performance acceptable in running Epoch programs.
I had barely started working on the raytracer implementation when I found my first type system bug in the R13 compiler.
Long story short, I've spent almost all my time post-R13 fixing bugs that shouldn't have shipped in the first place.
Why did this happen? I have a suite of unit tests, including tests for all the new features. They all passed. Where did the bugs creep in?
The problem with unit testing something like a compiler is combinatorial explosion. Sure, templates work. Type aliases work. Sum types work. But once you reach the point where you need to use a templated sum type to store a type alias which refers to a sum type that might contain a templated structure instance, stuff gets messy.
Epoch is still a pretty fast-and-loose project in terms of development discipline, and I rather appreciate the freedom to fire off quick releases and move forward at my own pace. I'm interested in preserving that "culture", but at the same time, I really need to revisit my reliance on simplistic unit tests.
The only remotely non-trivial program I routinely test in Epoch is the Era IDE prototype, which isn't exactly stretching the compiler's limits at this point. Adding the raytracer to the test suite will be good in multiple ways: it will ensure that a decent cross-section of features gets tested as it would be used in real software; it will demonstrate good practices for writing software in Epoch; and it will provide a benchmark of both compiler and VM performance that I can hopefully continually improve on over time.
But that may not be enough. I think I need to start writing borderline-pathological test cases to make sure that all the cobwebs are swept out of the corners of the language - both in terms of design and in terms of implementation.