Sign in to follow this  
CoffeeMug

Unit Testing

Recommended Posts

I'm trying to understand unit testing. More specifically, I'm trying to understand how I would use it to improve the quality of my C++ based project. Now, I think unit testing is an amazing idea for a purely functional language. Every time I write a function I'd write a unit test over the entire input domain of that function (or some set of representative values), run the function through the test, fix cases that cause the function to return an incorrect value and move on to the next function. But how would I do this in an imperative language? At any give time (and probably more often than not) I may have a function that modifies some object's state. Usually the object is a member variable of some complex type and the function I intend to test doesn't really return a testable value: its only purpose is modification of state. How is unit testing used in this scenario (which tends to be the case vast majority of the time)?

Share this post


Link to post
Share on other sites
Hi,

Well, basically, for each test you would create a new instance of the object you are testing and then call some function that modifies its internal state, and then call some function that indicates what state the object is currently in, and test that.

For instance, suppose you are writing a Stack class. Then, you could have it look like this:


template <type T>
class Stack {
public:
void push(T number);
T pop();
T peek();
int height();

private:
//Internal representation goes here.
};







Then, you could write unit tests which test the following things:

- Push something onto the stack. Peek to make sure that we get back the same thing.
- Push something onto an empty stack. Check height to make sure it's 1.
- Peek at the stack. Make sure height didn't change.
- Push something onto the stack. Pop it back to make sure you get the same thing.
- Pop something off the stack. Check that the height is now one less.

That's sorta how it goes, only more complex for more complex objects.

There's also an article on test-driven development here:

Test Driving Expression Template Programming

Check it out. It may help you too.

Vovan

Share this post


Link to post
Share on other sites
They all answer your question, it's a series, on unit testing. Where they develop and application and examine how unit testing works in a realworld situation.

The reading is very light, and easy to understand and follow.

Share this post


Link to post
Share on other sites
You are testing the class as a whole, not individual functions. Specifically you are testing that the output of the class is correct under all possible scenerios. If you can prove that then you passed the unit test. Some function could be "wrong", but if it has no impact on the output under any scenerio you still pass your unit test. Things like variables that are written, but never read are caught by the walkthrough, not the unit test. If you miss it in the walkthrough you should catch it when developing, not running, the unit test. An example is that your unit test should test every branch of every if statement, but you can't create a scenerio where a particular branch is actually taken or you can't create a scenerio that allows you to verify the value of some variable. So you have dead code or a variable that is written, but never read or perhaps never used at all.

Share this post


Link to post
Share on other sites
I am sure there are probably many different ideas about the term unit testing, but here was what I decided to define it as and use it in a professional environment.

In essence, unit testing is the idea of testing your overall code base by exercising the individual components separately. I would not recommend using this in lieu of a formal testing process (black box/white box/functionality verification/usability) but rather as a mechanism to enhance the formal testing process.

If you skip into your "three weeks to alpha" point in development and start to fling builds to QA, you will humbled to find hundreds if not thousands of bugs (if you are a one man show – then the process is purely masochistic). Many of these problems will be resolved by one code fix, others are symptomatic of larger unknown issues. Think bugs in the memory management code, for example.

Unit testing, at least how we practiced it, was used to accomplish two goals: First, I wanted to involve the QA team as deeply into the project as possible. This meant that when we began the full test cycles, they would be better able to break the code as well as describing what broke. Second, developers extensively testing their own code is a waste of time IMHO. A quick check to ensure that the task is completed correctly is fine, but I need development building features. However, it is of great importance to thoroughly test all code paths - so we worked with QA and built simple command line interfaces to code modules and allowed them to hammer the code.

We also discovered other benefits - we began to track performance on individual pieces of the code and where we knew there would be bottlenecks were are to architect solutions with great ease. In addition, we also knew we could thoroughly trust large portions of the code base early on, thus eliminating them from scrutiny when other problems arise. This meant that our dev/qa meetings now would be informed discussions about where bugs may be and how as an overall team we could find and resolve the problems.

The level in which you decide to do you unit testing, I believe, is subjective. I would think it is certainly plausible to exercise code at the class/object level all the way to something like the "render engine" level. The true benefit is being able to find esoteric problems and boundary criteria early. it is much easier find and resolve a memory alignment problem early, vice when you are in the beta cycle.


Mind you, this is how we chose to integrate this process into our team. I am sure there are +/- to our technique, but honestly I think its success was more influenced by the fact that we all bought into the idea and tried to make it work vice the idea itself. We all came to realize that we created better code more quickly using this process – even though we had to spend more time building interfaces and test harnesses to accommodate the process. However, these tools were used all the way through the process and QA made frequent use of them when trying to isolate problems. The bug reports that QA would provide were always concise, direct and pointing to the exact area where the problem would exist. Overall – I would say I am a big believer in this technique.

I hope that helps -

#dth-0

Share this post


Link to post
Share on other sites
Quote:
Original post by xiuhcoatl
In essence, unit testing is the idea of testing your overall code base by exercising the individual components separately.

As far as I understand it, the idea behind unit testing is testing components at the lowest reasonably possible level of functional completeness (can you parse that sentence? [smile]) For functional languages this level is a function. For object oriented languages this level is believed to be a class. The problem I'm having with the whole thing is that in my mind most bugs arise from complex interaction of components and cannot be exposed by testing each class individually. If you attempt to test higher level code it becomes impractical to write unit tests because of the enormous amount of different scenarios that need to be tested.

Perhaps I can give some examples that can illustrate my point better. First of all, certain things cannot be unit tested. Code that requires user interaction is one area where unit tests fall short. All the automated tools are extremely expensive and AFAIK aren't very good. Graphical output (say of a 3D engine) cannot be united tested because I don't have nor could I acquire reference data to compare my unit tests to. Now, suppose I am writing a system like e-bay and I'd like to test some part of the API responsible for handling auctions. How would I unit test my code on that level? The process involves extremely large set of timing dependant asynchronious inputs. How would I write unit tests to emulate such an environment? I'd need to write hundreds of potential test cases. Now, if I try to test the system on a lower level (components that the auction API uses to get the job done) I won't catch most of the bugs because they arise precisely from component interaction, not from localized functionality.

So basically, how would I use unit testing to test component interaction with thousands of potential test cases (that could cause the app to take thousands of different code paths), and how would I use it in situation where it isn't possible to immediately peek at the component's state? (A single call to the scene graph modifies the render queue state which modifies the renderer state. How do I unit test that?)

Share this post


Link to post
Share on other sites
It may be that some unit testing frameworks dont work well for integration testing, but that would usually mean that the author was trying to do something that unit testing wasn't meant to do. Take a close look at the name unit testing. The idea is that you are certain that a unit works well. You do this for each unit. Then, when you combine say a few small units into a larger subsystem you write unit tests for that subsystem. You dont have to test the small ones again, as their individual unit tests take care of that. Then, as you make changes to any of the units, the unit tests can be used to do regression testing with no extra work. There is point where it becomes a diminishing returns thing, meaning that it is so time consuming to write the tests that it is no longer economically sound to do so. That is when you switch to some other testing technique for any higher level tests.
If you look at the book "Large Scale C++ Software Design," the author makes one of the best arguments for unit testing I have read, although he calls in "Heigharchical testing" because the book came out before unit testing had caught on.

Share this post


Link to post
Share on other sites
Side-effects are to be avoided when possible. Without side effects unit-testing is easy.
When you do have side-effects just load it full of Asserts, Invariants, and Pre/Post condition type of checks. Then unit test it inside of the thing that uses it, or at some higher level.

capn_midnight has hit the nail on the head though. But it takes a long time before anyone builds up the discipline to do that.

Share this post


Link to post
Share on other sites
Quote:
Original post by Washu
Craftsman to the Rescue!

Just finished reading the series. You have an amusing taste [smile] Anyway, one interesting thing I noticed that I didn't realize before is that I already develop like that, I just never keep the tests. Essentially, when I write code I also write a function that tests it and I continuously refine the test as well as the production code. Once I'm satisfied that the code runs fine I simply get rid of the test code and move on to the next task on my list. However, I acknowledge the numerous advantages of keeping the tests and organizing them into automated test suits. One potential problem I see is that many of my test functions that I write now require user interaction (either the user has to *look* at the screen to see if the test passes or fails, or the user has to provide some input to fire off the test). In many cases I just don't think it's possible to create completely automated testing functions. What do you do about these issues?

Another question I have is about organization of test suits. I plan to use boost unit testing framework but I'm not sure how to go about creating tests. Do I create a separate project or do I keep the tests in the same project as production code? If I use the same project, how do I get rid of the test code from the production build? Preprocessor? Also, suppose I have two independant classes A and B, and a class C whose functionality depends on A and B. I develop A, then B, then C. How do I structure the test suite? First create a test for A and add it to the test framework. Then do the same for B. Once I create C, do I move the tests for A and B into C and then add a test for C, or keep tests for A, B and C on the same level? I see that boost supports hierarchial test suite structure but I'm not sure how to properly take advantage of it.

Share this post


Link to post
Share on other sites
GuiUnitTesting

Martin Fowler recommends moving as much stuff out of the gui class itself into your own classes. That way the gui classes are a client of the code which does all the work (i.e. the code you want to test), and the gui class will be a thin, forwarding class, which will be very hard to get wrong and won't need testing itself.

Share this post


Link to post
Share on other sites
Quote:
Original post by xiuhcoatl
I would not recommend using this in lieu of a formal testing process (black box/white box/functionality verification/usability) but rather as a mechanism to enhance the formal testing process.


Well, blackbox and whitebox are not testing processes, as I understand it. Instead, they are ways to come up with test cases. So, they and the unit testing are not mutually exclusive (you can't use one in lieu of the other). Both apply to all testing stages. Suppose you want to do unit testing. How do you come up with test cases? Well, that's where blackbox and whitebox come in. You want to do integration testing. How do you come up with test cases? Same as before: blackbox and whitebox provide you with ideas how to make the tests.

Just being picky about terms. [smile]

Vovan

Share this post


Link to post
Share on other sites
Quote:
Original post by CoffeeMug
Quote:
Original post by Washu
Craftsman to the Rescue!

Just finished reading the series. You have an amusing taste [smile] Anyway, one interesting thing I noticed that I didn't realize before is that I already develop like that, I just never keep the tests. Essentially, when I write code I also write a function that tests it and I continuously refine the test as well as the production code. Once I'm satisfied that the code runs fine I simply get rid of the test code and move on to the next task on my list. However, I acknowledge the numerous advantages of keeping the tests and organizing them into automated test suits. One potential problem I see is that many of my test functions that I write now require user interaction (either the user has to *look* at the screen to see if the test passes or fails, or the user has to provide some input to fire off the test). In many cases I just don't think it's possible to create completely automated testing functions. What do you do about these issues?

Ok, In this case I actually abstract the portion that has to do with user input. Thus I can either supply a mock object that provides input, or an actual object that provides the input (see mock object pattern). Of course, if you are designing a gui, things get a bit more complex, but you can simulate key presses and mouse moves fairly easily.
Quote:

Another question I have is about organization of test suits. I plan to use boost unit testing framework but I'm not sure how to go about creating tests. Do I create a separate project or do I keep the tests in the same project as production code? If I use the same project, how do I get rid of the test code from the production build? Preprocessor?

I tend to use a second project (I use C# mostly now days) and build my tests there. My reasoning for this is simple, a couple of my projects are a few hundred thousand lines long. So recompiling the thing because I changed a test, or wrote a new one, tends to be a waste of time. However, there is no reason not to keep the tests as part of the code base, reason being: only the test running will execute the tests. So a good compiler (for C++) will just remove the tests if they aren't run. Not to mention, even if they do remain, they don't take away anything but minor space overhead.
Quote:

Also, suppose I have two independant classes A and B, and a class C whose functionality depends on A and B. I develop A, then B, then C. How do I structure the test suite? First create a test for A and add it to the test framework. Then do the same for B. Once I create C, do I move the tests for A and B into C and then add a test for C, or keep tests for A, B and C on the same level? I see that boost supports hierarchial test suite structure but I'm not sure how to properly take advantage of it.

No, you keep the tests seperate. The goal is to validate that A and B perform as expected. Then you write tests for C, these tests should validate that both C performs as expected, and that the use of A and B, if publicly visible in C, performs as expected.
However, you should use the heirarchy for modules, or namespace. This promotes both organization, and clarity of reading the results.

Share this post


Link to post
Share on other sites
Quote:
Original post by Washu
Ok, In this case I actually abstract the portion that has to do with user input. Thus I can either supply a mock object that provides input, or an actual object that provides the input (see mock object pattern). Of course, if you are designing a gui, things get a bit more complex, but you can simulate key presses and mouse moves fairly easily.

Properly simulating user input may get rather time consuming but it isn't the main problem. The main problem is evaluating the resulting action.
Quote:
Original post by Washu
No, you keep the tests seperate. The goal is to validate that A and B perform as expected. Then you write tests for C, these tests should validate that both C performs as expected, and that the use of A and B, if publicly visible in C, performs as expected.
However, you should use the heirarchy for modules, or namespace. This promotes both organization, and clarity of reading the results.

I'm not sure I understand. If I have a layered design that includes three main systems, each of which includes 10-20 subsystems and each of those includes 20-50 classes I have a total of about 1500 classes which means I'll have 1500 unit tests. How do I manage their *invocation*? Add 1500 invocation calls to one "StartTesting()" function? That's just rediculous. Essentially I need to A) be able to run every test without having to run others and B) be able to maintain test suites in hierarchial manner. For instance, if I am testing a subsystem S5 with classes C1 - C40 I need to be able to say TestS5() and have the test automatically test classes C1-C40 and then test class S5. Now if I need to test a sysbsystem S6 that contains classes C35 - C60, I should be able to say TestS6(). The problem now becomes that TestS6() will test classes C35 - C40 which have been tested already by TestS5(). I'd imagine this process may significantly increase test time which seems important. How are these types of problems handled?

Share this post


Link to post
Share on other sites
Quote:
Original post by CoffeeMug
I'm not sure I understand. If I have a layered design that includes three main systems, each of which includes 10-20 subsystems and each of those includes 20-50 classes I have a total of about 1500 classes which means I'll have 1500 unit tests. How do I manage their *invocation*? Add 1500 invocation calls to one "StartTesting()" function? That's just rediculous. Essentially I need to A) be able to run every test without having to run others and B) be able to maintain test suites in hierarchial manner. For instance, if I am testing a subsystem S5 with classes C1 - C40 I need to be able to say TestS5() and have the test automatically test classes C1-C40 and then test class S5. Now if I need to test a sysbsystem S6 that contains classes C35 - C60, I should be able to say TestS6(). The problem now becomes that TestS6() will test classes C35 - C40 which have been tested already by TestS5(). I'd imagine this process may significantly increase test time which seems important. How are these types of problems handled?

...
Each system is a member of the heirarchy. Along with each subsystem. Hence your tests would be organized to follow that same tree.

Share this post


Link to post
Share on other sites
Hmm, wasteful testing of shared classes and lack of selective execution seems to be a problem specific to boost testing framework. Selective testing is easy to hack in, but I'm not sure how to get around testing classes shared accross subsystems multiple times. It's not so important for my project right now as it's currently fairly small but it may become a problem in the future... I've e-mailed Boost.Test maintainer to see if he has solutions in the pipeline.

Share this post


Link to post
Share on other sites
Quote:
Original post by vovansim
How do you come up with test cases? Well, that's where blackbox and whitebox come in. You want to do integration testing. How do you come up with test cases? Same as before: blackbox and whitebox provide you with ideas how to make the tests.
Vovan


I am not sure I agree with you here - but if that is what works in your organization, or for you, then I suppose that’s ok. I don't believe in rigid laws with regards to how the job gets done - if the end result is good and the journey pleasant.. who cares.


I suppose you can make the argument that the "term" unit test, white box, black box, etc... are not a process in themselves but rather a technique or tactic. However, the idea of using blackbox testing, for example, to come up with your test cases is actually a testing mechanism (referred to as ad-hoc testing) and is not the best of ideas. Your test cases should actually come from Marketing requirements (if feasible), Engineering response document (or whatever you call the document that engineering says can really be done), architectural designs, and module design documents imho. Your formal testing process is there to validate that you built "what" you were supposed to build, it works the "way" that it was intended to, and that it cannot be exercised in an unforeseen or undesirable way (boundary criteria). If your test cases are built from using the code then you are not really testing for compliance.

This is why I wanted the QA department, and the build release department for that matter, involved with the development process from day one. While we were designing the architecture and how the code would work, QA were roughly sketching out their test cases and procedures. When we started building modules and unit test harnesses, they would script together automated testing scripts. In this manner, once we had begun the formal test process we already had baked in a lot of expertise and tools to get the job done.

In reality, no one 'tactic' is sufficient to test every possibility, which is why several tactics are used. Additionally, not every tactic is ideal for producing sufficient analysis. Rigid black-box testing is great for testing compliance to a design, but ad-hoc black-box testing is great for discovering that "whoops - we didn’t think of that" scenario. What worked for us is certainly 'not' going to work for everyone and every situation - so its up to the individual to decide what is practical and useful for their situation.


Quote:
Original post by CoffeeMug
The problem I'm having with the whole thing is that in my mind most bugs arise from complex interaction of components and cannot be exposed by testing each class individually. If you attempt to test higher-level code it becomes impractical to write unit tests because of the enormous amount of different scenarios that need to be tested.


Yep, I agree. IMHO I think unit testing at that level creates a problem where your testing code is now more complicated and harder to maintain then the code you are trying to test. We solved some of these issues by trying to decide what "functional blocks" of code could benefit from unit testing the most. Additionally, we built test harnesses into the code that, in a debug-test build, would provide a module that QA could exercise. We left "how" it got exercised to QA, whom made the determination based on our designs and meeting discussions. This removes development from having to design QA test cases (which is not what they do best) and left that for QA (which is what they do best). By making the harness (usually complicated command line interfaces that would expose underlying code) available to scripting (Perl in our case) QA could quickly assemble test cases as well as run them unattended. Eventually - these got tossed into the automated build/acceptance process, once the code base stabilized, and was used to catch low hanging fruit that crept back into the code. The problem we ran into here, was that if you made changes to this area of the code you had to be very careful to update everything else - otherwise things slip through the cracks.

Concerning larger, more complicated pieces of code that have timing issues, I (mind you without looking at it) would not make them a candidate for unit testing and would use another tactic. Off the top of my head, I would break the large process into smaller pieces and unit test them and then use functional black or white box tests to test the larger process that is not easy to automate or unit test. The idea being - hassle all the inputs to the individual pieces and make sure they work properly and handle boundary criteria correctly. Then, look at your designs and architecture and write (as in pen and paper..) test cases to exercise the larger process and use formal testing, vice automation, to exercise how everything works together. Give QA some ideas on weaknesses or fragile areas of the code and let them see if they can break it - that’s what they are most talented at doing.


Hope that helps -


#dth-0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this