# Unit Testing ftw?

This topic is 1554 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

Feel free to give some feedback and perhaps critique. It'd be nice to see some different perspectives on unit testing versus overall systems testing.

Edited by Malabyte

##### Share on other sites

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

What I've realized a lot more and also a bit through watching this video, is how important it is to code in the right way and to make it testable and to do tidbits of pinpoint testing while you code. Before I watched this video, I felt that the term "playtesting" is too vague. There's "systems testing" (which you only do at specific phases of development) and then there's the constant "functional testing" (which I erroneously called system testing for a while) of every code you write, immediately after you've written it. Just to see if it returns the values it's supposed to, or whatever.

I just felt like this video confirmed a lot of those thoughts, but I can also see that there's differences between doing solo projects and working in a team/company. Or even differences depending on how complex the program is in general, or what part of the program we're dealing with.

Edited by Malabyte

##### Share on other sites

Unit tests should always be written, it is not a widely practiced art in games development at the moment but it is gaining ground.

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

This is not true, any code written should have an associated unit test to prove that it does what it says it does. The cost for a developer to implement the unit test is far less than any other time spent trying to fix broken code.

It also improves the confidence in changes made to the system, as any changes that cause a breakage will also be flagged by the unit test.

Ofcourse, it's easy to just say "everything should have a unit test", but it takes quite a structured approach to development. And yes, there are a lot of developers in the games industry that do not write unit tests (far more don't than do to be honest).

However, unit tests will help you find bugs earlier, improve the robustness of your code and, ultimately, save you time (and money) in development.

n!

Edited by nfactorial

##### Share on other sites

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

I think he means that having all the programmers spending time (cost) on making unit tests while coding the actual functionality, combined throughout the entire project is > in cost than just having them code the functionality of the app and just do normal testing with QA people. That changes to < over x number of years your code is actually running, but most games don't really run or get maintained over a pretty small number of years.

If you are doing an MMO then this would obviously make a ton of sense because your game could be running for decades, but most games probably get a couple years of action.

I think it's easy for programmers to always say let's do unit testing, but the reality is budgets, time, etc all play into things. You may always want to make the best game possible but the people who gave you money may just want a decent game out early and on a tight budget.

ultimately, save you time in development.

But when does that "ultimately" happen? If your game is good enough to make the money it was projected, then the people who gave you money might not care.

It's situational and the real world often gets in the way. In an ideal world, sure. It's all about the money.

Edited by rpiller

##### Share on other sites

When I say "ultimately" I don't mean "at the end of development" I mean "the core benefit you gain". "Ultimately" here is a constant throughout development.

n!

##### Share on other sites

As a sidenote, I imagine that the type of unit testing that you prioritize, will change depending on circumstances as well, plus how seasoned you are as a programmer. Like, certain functions and classes would be reviewed a lot less by an experienced pro, because he's so used to writing them that he knows better what to look for and whatnot. Or that he's just able to write code more definitively than a rookie, i.e. not having to refactor as much because he can deal with much more complexity at a time. That could reduce the possible exceptions, since less code means less areas where an exception would occur.

##### Share on other sites

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes. Just as, for example, I'd be sure Carmack always starts a new engine by rendering a cube (or some simple primitive).

Whilst writing the test itself does consume time (and writing tests themselves, does not take much time) it will save much more time in the future than the tests took to write.

Also, writing a test means you get to test the API you have written and give you earlier ideas for improvement. In test driven development you write the test before you even begin the system, the test will fail. You then write the system so the test does not fail and any breaking changes in the future will be immediately caught. There are better resources on the web than I could give here, so if anyone is interested I would recommend searching the web for "Test Driven Development".

Writing tests is not about "it takes more time" but rather it saves you both time and money.

n!

##### Share on other sites

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes.

I know, I'm just talking about the ratio of it and the idea that one should first test the things that one is unfamiliar with, because that's a more likely place where errors will occur.

##### Share on other sites

I am possibly misunderstanding, there is no "ratio". If you write a system, then you write a test for it first.

But, I think I misunderstood :)

n!

##### Share on other sites

I think the sweet spot for automatic tests are at subsystem boundaries. This way you don't have to spend time writing tests for every class "just becuse", but you still verify that the subsystem, as a whole, works as expected. Which is what matters.

Of course, this assumes some kind of division into subsystems, which may not always be there.

##### Share on other sites

I'm not a game programmer but I am a commercial systems programmer, and automated unit tests are now stock-in-trade. We tend to use the tools in two ways (we use C#, so nUnit or MSTest are our toolkit)

1) To investigate the behaviour of a unit of code when writing it - code the test alongside the unit. You end up doing far less debugging step-through type stuff that way.

2) To formally test the boundary conditions of the code to make sure critical sections (like multithreaded code) works as designed, and in a way which would only be evident at runtime.

Both styles of tests are checked in to the source code control system at the same time, and we pump out statistics that give us code test coverage with each build - if it falls below a certain percentage we start to get twitchy - but there is no set limit. Typically code which is both complex (cyclomatic complexity) and not covered by unit tests, is flagged as being a hotspot and we then pick up some additional testing.

In my experience, and having come to automated testing late, I wouldn't now code without it. You dont actually need a unit test toolkit since these are just harnesses that allow you to execute a single portion of your code - you could do the same with a parameterised command line application if you wanted to. I'm not dogmatic about TDD - but in my experience programmers work faster and more accurately if they write unit tests at the same time as their line-of-business code. Its actually cheaper, and not just in the long run - any project greater than a couple of days will be made more accurate and more timely if the programmer codes tests at the same time.

As an example in the game world, in my hobby terrain project, I used unit tests to isolate and explore the behaviour of my mesh clipping and frustum culling, to make sure it was behaving as I expected. It also means I can semi-confidently change a portion of otherwise complex code, and use my unit tests as a "trigger" to spot any changes to external behaviour that I didn't know about. At its most extreme in the commercial world I've seen a big drop in the amount of paperwork specifications and "explanatory" documents needing to be produced by programmers - although this outcome isn't the goal, the fact that the behaviour of the code is explained in a series of unit tests does help incoming programmers.

The last thing to say is that using automated unit tests leads you to write better code - not just less bugs, but more structured, more isolated and more open and allows much more confident maintenance of complex systems, and less time spent in marathon debugging sessions.

Phillip

##### Share on other sites

I am possibly misunderstanding, there is no "ratio".

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

##### Share on other sites

I am possibly misunderstanding, there is no "ratio".

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

Phillip

Edited by PhillipHamlyn

##### Share on other sites

I am possibly misunderstanding, there is no "ratio".

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

Phillip

Its not really about the skill of the programmer who writes the code in the first place, in a professional setting(especially in business programming where code tend to stick around for a very long time) you will have a wide variety of people touching the code, experienced developers retire or change job, fresh graduates gets hired, code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers, if the experts don't do their job and write proper tests you can be fairly certain that someone will manage to break the code at some point without anyone noticing until the customers start complaining.

Edited by SimonForsman

##### Share on other sites

Let me give a real-world example of the business case.

Over the years I have seen many articles, perhaps even up to the hundreds, that tout the number as approximately 1:1 when it comes to comprehensive automated testing and green development. Since we track our progress carefully, I can agree with those numbers in my personal experience. It takes about twice as long to write code+tests as it does to just write code without tests.

We have a game engine that has been in use since 2005. This engine has many hundred thousand unit tests in it. The physics library has unit test, the pathfinding has unit tests, the spatial systems have unit tests, the animation components have unit tests, and so on. If someone were to introduce a significant bug in the system it could take down not just the one developer, but potentially take down the entire team for several days. Such critical bugs do sometimes get introduced, and by the time you factor in just a single day of lost productivity among all people on all teams the cost of such a bug can reach around 150 work-days lost. This is exactly the type of scenario the video is talking about. For this type of engine, a long-term project shared by many people, or a project in long-term maintenance, there is a compelling reason to write the tests.

Now let's look at the other side, the people who use the engine.

Developing a new item with this engine requires about 3 weeks of development time followed by about 2 weeks of QA time. Programmer hours are expensive, QA hours are cheap. When a new item is developed it is launched, gets put out the door, and is generally never touched again. The code is effectively thrown away.  Of course it is not completely thrown out, we routinely pull code from prior items and reuse them, and some of it gets incorporated back to the engine where tests are developed for it. Now over the course of several years and several hundred items we have only had three items where customers found significant bugs that required a re-release. The cost of these bugs is practically zero, and unit tests would not have helped avoid them. The cost to do a single QA push is far less than the additional cost of writing the automated tests. The automated tests have no lasting value because the code itself has no lasting value.

I am a strong proponent of TDD when it makes sense. I love it.  The red-green-refactor mantra plays in my head even when I writing throw-away script code because in practice I still follow it: I have a thing to change, I make the change, I test the change, I clean up, I retest, and I submit the code. However, I am an opponent of TDD when it does not make sense. It does not make sense in code which has no long-term value.

##### Share on other sites

code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers

Please name me a single traditional game engine that has 20+ years of life in it.

World of Warcraft is one of the longest running major games out there today, and it hasn't had it's 9th birthday yet.

Other long-running engines include the annual sports games like Madden that is 25 years old and FIFA that is 20 years old, but those have been rewritten many times over their lifetime.

It is true that certain long-running libraries and components are deserving of unit tests. I haven't seen anybody write that they shouldn't have unit tests.  But other parts of the game are short-lived and the code is effectively dead when the disc image is released to manufacturing, and these would be foolish to write unit tests for.

##### Share on other sites

This video killed all of my interest in automated unit testing.

##### Share on other sites

One aspect of writing tests in advance (basically either kind of test) is that you're forced to think about corner cases and special circumstances to test. More than once, the result was being aware of those special cases before writing the actual implementation, resulting in code that had far fewer bugs from the start and ended up much cleaner, because you didn't end up fiddling in bug fixes after the fact.

If you write tests, I'd suggest doing so before you finish planing your actual implementation. Plan your system tests in parallel with the requirements (sometimes during writing a test you might realize there are requirements nobody thought about yet). Unit tests should be done in parallel with planing the interface for the unit (as sometimes writing the test will make you realize that your interface is broken). Afterwards during implementation, you will most likely go back and add more tests as you write test-worthy functions or algorithm.

##### Share on other sites

Let me give a real-world example of the business case.

Over the years I have seen many articles, perhaps even up to the hundreds, that tout the number as approximately 1:1 when it comes to comprehensive automated testing and green development. Since we track our progress carefully, I can agree with those numbers in my personal experience. It takes about twice as long to write code+tests as it does to just write code without tests.

Frob, my experience is the opposite of yours - my experience is that it takes half as long to get code working when tests + code are written together. Yes, there is twice as much code, but half as many mistakes, meaning the development cycle is shortened and cheaper. It does depend on lots of factors, as you describe, but the writing of code is not the majority of the cost of getting it finished - working code is the goal, and unit tests help develop working code. Its interesting that people on the board seem to have quite divergent experiences of this, which refelects something about the industry ... but I dont know what :-)

##### Share on other sites

code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers

Please name me a single traditional game engine that has 20+ years of life in it.

World of Warcraft is one of the longest running major games out there today, and it hasn't had it's 9th birthday yet.

Other long-running engines include the annual sports games like Madden that is 25 years old and FIFA that is 20 years old, but those have been rewritten many times over their lifetime.

It is true that certain long-running libraries and components are deserving of unit tests. I haven't seen anybody write that they shouldn't have unit tests.  But other parts of the game are short-lived and the code is effectively dead when the disc image is released to manufacturing, and these would be foolish to write unit tests for.

I never mentioned games, the post i replied to was talking about business software.

##### Share on other sites

My philosophy with code is that if I haven't seen every single feature actually work, then the code isn't finished yet.

Under this definition, if writing the code+tests takes twice as long as just the code, that doesn't matter. When you've just written the code, the task isn't finished. After you've passed the tests, the task is finished (hopefully). So you're comparing the time taken to half-finish a task with the time taken to finish a task

If I'm not writing tests for something, I'll usually at least put a breakpoint in it once and step through it line by line, which is basically a semi-automated "desk check", which used to be a standard requirement for programming...

This is especially true when writing games in C++ -- all too often it's possible to write code that looks correct, and seems to behave correct from the outside, but a simple desk-check reveals an obvious off-by-one or overflow or edge-case bug.

If the code is too complex to easily step through all the possible use-cases, then I'll write a unit test instead.

The linked video about writing testable code contains good advice, btw. Not just for testing's sake, but also for writing flexible, reusable code in general.

Please name me a single traditional game engine that has 20+ years of life in it.

Yes this is not common, engines from the 80's/90's/00's are each very different from the engines of today... but 10 years of code maintenance might happen, a small bit

I bet there's still some small smidgen of code from Quake 1 hanging around in the Source engine.

When I was at Krome, the coding convention required every function to be commented with the original author and date. I saw a few nuggets deep within the 3rd rewrite of their engine that were dated year 2000, and this was in 2010.

Edited by Hodgman

##### Share on other sites

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

Usually, the point of unit testing is that if the units are well tested and correct all errors that come up in the integration between those units are due to the integration logic itself (e.g. module X crashes with the actual game engine because, unlike in X's unit tests, it isn't initialized properly) or to misunderstandings between modules (e.g. both the texture loader and the texture resource manager flip textures vertically, resulting in upside down graphics). It's much better than testing everything with a chance of bugs of any kind.

##### Share on other sites

Would you want a programmer spending time writing units tests (especially if it's 1:1 or even if it's 1/2 the amount of time) at $150k/yr+, or a QA guy test the code for$30k/yr? I think it's a valid question on where you want your time/money spent. There is no doubt that TDD results in better code, but that's not always the requirement. Sometimes 'good enough' and getting it done fast and cheap is the requirement.

If I'm not writing tests for something, I'll usually at least put a breakpoint in it once and step through it line by line, which is basically a semi-automated "desk check", which used to be a standard requirement for programming...

This is especially true when writing games in C++ -- all too often it's possible to write code that looks correct, and seems to behave correct from the outside, but a simple desk-check reveals an obvious off-by-one or overflow or edge-case bug.

In my view stepping through code is a must. Every game/app I've ever written, which haven't been massive amount of code, I've always stepped through every line at one point or another. Every time I implement a class or couple methods I step through them to make sure they do what I expect.

Your comment makes it sound like that's not a common thing to do? I have to assume it's very common and probably done more then unit testing between all the different levels of programmers around.

##### Share on other sites

In my view stepping through code is a must. Every game/app I've ever written, which haven't been massive amount of code, I've always stepped through every line at one point or another. Every time I implement a class or couple methods I step through them to make sure they do what I expect.

In my experience when I started using Unit TDD and Accepance TDD, I've never had to step through every line. That's what my tests are doing.

I'm also surprised that no one mentioned that writing unit tests improves the design of the application. It enforces separation of concerns, single responsibility principle, dependency inversion, etc., because it becomes difficult to unit test classes if they violate these principles. This, in turn, improves maintainability.