Unit Testing
Author
Now i have done a bit of Unit Testing before. On one of my modules we used unit testing to test trivial classes, these were logic gates, AND, OR, NOT etc. As you can imagine, these were very obvious to test, simple.
I have also written other unit testing code, for other not so simple but also fairly trivial classes. I have also written tests for my memory manager etc. With my more complex classes i began to find that the actual unit tests themselves were becoming larger pieces of code than that i was testing. I began to think this was pointless. Surely i can make as many, if not more, errors in writing the uni test code as i am looking for in the code being tested.
I also began to find that some code was untestable. Such as code that outputs certain things to the screen. For example, would you unit test a piece of code that procedurally generated a texture? If so, how would you do it?
So i put to you:
1) Where do you draw the line with writing unit testing code?
2) What do and don't you test?
Thanks for any comments,
Dave
Caveat: I've almost certainly written less unit tests than you. I do have experience on the other side of the fence though, doing QA work.
You're not alone, given my experience with limited or non-existant unit tests. The key though is that unit test code by and large doesn't change. Production code will. Those later changes are really where unit tests show their value, and you (or your less talented successors) are more likely to make maintenance errors than break unit tests.
1) At interfaces, like almost anything else. If you make a module that is meant to do something, others (or yourself) are going to code to that interface. Unit tests verify that contract so that you can be relatively certain that changes you make won't cause other parts grief (and that other modules your stuff interact with haven't changed).
2) Test stuff you can. Some things like the texture's contents will need to be visually inspected. You can still test the generation to ensure the return is sane, size is correct (if a limitation), invalid input is handled properly, and maybe have the unit test dump it to a file for inspection/archival.
The key point is to handle as much regression and basic functional testing so that engineering is confidant that a planned QC build is good enough to test. Nothing quite aggrivates QA like getting a handoff that keels over after 5 seconds.
Quote:Original post by Dave
I began to think this was pointless. Surely i can make as many, if not more, errors in writing the uni test code as i am looking for in the code being tested.
You're not alone, given my experience with limited or non-existant unit tests. The key though is that unit test code by and large doesn't change. Production code will. Those later changes are really where unit tests show their value, and you (or your less talented successors) are more likely to make maintenance errors than break unit tests.
Quote:
I also began to find that some code was untestable. Such as code that outputs certain things to the screen. For example, would you unit test a piece of code that procedurally generated a texture? If so, how would you do it?
So i put to you:
1) Where do you draw the line with writing unit testing code?
2) What do and don't you test?
1) At interfaces, like almost anything else. If you make a module that is meant to do something, others (or yourself) are going to code to that interface. Unit tests verify that contract so that you can be relatively certain that changes you make won't cause other parts grief (and that other modules your stuff interact with haven't changed).
2) Test stuff you can. Some things like the texture's contents will need to be visually inspected. You can still test the generation to ensure the return is sane, size is correct (if a limitation), invalid input is handled properly, and maybe have the unit test dump it to a file for inspection/archival.
The key point is to handle as much regression and basic functional testing so that engineering is confidant that a planned QC build is good enough to test. Nothing quite aggrivates QA like getting a handoff that keels over after 5 seconds.
1) Most of the time I add tests when I stumble upon bugs in "testable" code (which excludes like 90% of the bugs occur, unfortunately).
2) I write test for pretty much anything but GUI-code (and similar evils). I would test texture generators by doing pixel-by-pixel pixel-by-pixel comparisions with reference images (I currently do this with my image decoders).
Overly complex tests can be pretty dangerous, so I would just write such tests "on demand" (to test an existing bug).
2) I write test for pretty much anything but GUI-code (and similar evils). I would test texture generators by doing pixel-by-pixel pixel-by-pixel comparisions with reference images (I currently do this with my image decoders).
Overly complex tests can be pretty dangerous, so I would just write such tests "on demand" (to test an existing bug).
Author
Quote:Original post by poppeman
1) Most of the time I add tests when I stumble upon bugs in "testable" code (which excludes like 90% of the bugs occur, unfortunately).
2) I write test for pretty much anything but GUI-code (and similar evils). I would test texture generators by doing pixel-by-pixel pixel-by-pixel comparisions with reference images (I currently do this with my image decoders).
Overly complex tests can be pretty dangerous, so I would just write such tests "on demand" (to test an existing bug).
Regarding 2)
This just seems overkill to me, as the code for testing the texture is more than likely to be very similar to the code that generated it pixel for pixel.
The usual idea behind 2 is to find unexpected rendering regressions. Basically you have your test generate a bunch of images, hand-check them, save them off as "known good", and then future tests are compared against the known good images. Comparing images is trivial. It's not like you've got two totally different engines and you're checking them against each other.
It's not great, and obviously doesn't work at all if your rendering engine is actively changing. But it's better than nothing for relatively static periods.
It's not great, and obviously doesn't work at all if your rendering engine is actively changing. But it's better than nothing for relatively static periods.
Quote:Original post by Dave
Surely i can make as many, if not more, errors in writing the uni test code as i am looking for in the code being tested.
A nice thing about unit tests is that they're typically linear. You don't have the number of potential code paths doubling (or worse) every conditional. Further, they should hopefully not require much mantinence - interfaces shouldn't be changing terribly often, which is what I typically test.
Sure, you'll have just as many typeos as in any other bit of code, but those are typically easy to find/fix compile errors rather than hard to find corner cases involving Friday the 13th, werewolves, and sticky notes hanging from floss.
Quote:I also began to find that some code was untestable. Such as code that outputs certain things to the screen. For example, would you unit test a piece of code that procedurally generated a texture? If so, how would you do it?
There's two main components I'd see myself seperating that down into:
1) The generation. Toss a few sampling points in with the values they should be at, and verify they don't change (too much).
2) API Integration. Render to texture and verify the contents, or directly check the texture contents itself. That'd be something I'm more likely to eyeball-test than unit-test, however, especially if generation is tested seperately - since most/all the texture code would be centralized, a failure would be unlikely to be partial/un-noticed.
Quote:So i put to you:
1) Where do you draw the line with writing unit testing code?
2) What do and don't you test?
A bit of raw sampling, a few corner cases (-1,0,+1, 0xFFFFFFFF, 0xFFFFFFFE), a few things I know should fail/throw and a few things I know should succeed (when not dealing with particularly raw data sets). Some "identity" stuff is also useful - in a fixed point math library, for example, C / 1 == C, and C / C == 1. An example for templated unit tests would be checking these in varying corner cases for C, with varying bits for precision.
For the really complex, I sometimes end up with templated unit tests (namely when dealing with templated classes). I usually stop when I can't manage to break my own code anymore. If the code manages to break, I fix it and then apply the insights of how that break happened towards trying to break more code. Sometimes, I realize I'm unable to fix my code to fit all the constraints I've decided upon (my fixed point math classes are in suspension because I can't meet the goals of accuracy, extreme genericness (in this decided upon fashion), and efficiency all at the same time, for division).
What I don't usually test is internals. Implementation details. They're subject to rapid change and often easier to get wrong. I will try and compatmentalize those details into their own sections if they're too large, and unit test those seperately, however.
I'm also a bit lazy, I avoid compile-fail unit-tests (although I usually do an initial prodding and leave the commented lines in).
Unit testing is invaluable when used correctly. It's a small initial investment that pays off in the long run. It's perfectly normal to use the same code on both sides of the coin initially, so it seems pointless, but the implementation is constantly changing, while the test code is static, so by the end of development you might have wildly different code on the implementation side. Regardless, you'll always be able to use the test code to verify correctness of your implementation at any point, and in fact you should constantly be regression testing to get the most benefit out of unit testing.
However, if possible, have one person write the test cases and another write the implementation, otherwise it becomes easier to write incorrect test code then subsequently write implementation code that satsfies the test code but is ultimately useless.
These days I use unit testing for any serious code, especially if it's intended for or available to others. It gives me peace of mind, and more importantly, gives me direction when it comes time to work on the implementation, because I know exactly what my code needs to accomplish to be correct.
If you're using C++, Boost.Test is an excellent unit testing framework.
However, if possible, have one person write the test cases and another write the implementation, otherwise it becomes easier to write incorrect test code then subsequently write implementation code that satsfies the test code but is ultimately useless.
These days I use unit testing for any serious code, especially if it's intended for or available to others. It gives me peace of mind, and more importantly, gives me direction when it comes time to work on the implementation, because I know exactly what my code needs to accomplish to be correct.
If you're using C++, Boost.Test is an excellent unit testing framework.
Quote:Original post by Dave
Regarding 2)
This just seems overkill to me, as the code for testing the texture is more than likely to be very similar to the code that generated it pixel for pixel.
Well, not really. Save the texture to a png (or whatever you may fancy) now and test against that one in the future.
Author
Is it simply a case of do this whenever you think you can, regardless of whether the code being tested is going to change or not?
Dave
Dave
Quote:With my more complex classes i began to find that the actual unit tests themselves were becoming larger pieces of code than that i was testing. I began to think this was pointless. Surely i can make as many, if not more, errors in writing the uni test code as i am looking for in the code being tested.You need to adjust the amount of tests based on how complex or large the piece of code being tested is. For a very simple function, one or two asserts could suffice. And for a fairly complex class you should test corner cases, error cases etc.
Quote:For example, would you unit test a piece of code that procedurally generated a texture? If so, how would you do it?I'd probably just test that the returned texture is of correct size and bit depth.. Surely it's also possible to do some testing that it is visually OK, e.g. a texture shouldn't have too much variation in colors (like blue, red, green, yellow, cyan all in one texture=failure) or it's spectrum shouldn't look like white noise :P. But it's probably not really worth it, a manual check is needed anyway to know that the texture actually looks nice.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
