• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
Ronnie Mado Solbakken

Unit Testing ftw?

36 posts in this topic

Hey peeps, I thought I'd ask your opinions about this video, where the presenter talks about deeper testing methodology and specifically about unit testing. I think this is fundamentally going to improve my programming routines, and maybe it will help you as well, even if you're a semi-seasoned programmer:
 

http://www.youtube.com/watch?v=wEhu57pih5w

 

Feel free to give some feedback and perhaps critique. It'd be nice to see some different perspectives on unit testing versus overall systems testing.

Edited by Malabyte
0

Share this post


Link to post
Share on other sites

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

 

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

 

What I've realized a lot more and also a bit through watching this video, is how important it is to code in the right way and to make it testable and to do tidbits of pinpoint testing while you code. Before I watched this video, I felt that the term "playtesting" is too vague. There's "systems testing" (which you only do at specific phases of development) and then there's the constant "functional testing" (which I erroneously called system testing for a while) of every code you write, immediately after you've written it. Just to see if it returns the values it's supposed to, or whatever.

 

I just felt like this video confirmed a lot of those thoughts, but I can also see that there's differences between doing solo projects and working in a team/company. Or even differences depending on how complex the program is in general, or what part of the program we're dealing with.

Edited by Malabyte
0

Share this post


Link to post
Share on other sites

Unit tests should always be written, it is not a widely practiced art in games development at the moment but it is gaining ground.

 

 

 

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

 

This is not true, any code written should have an associated unit test to prove that it does what it says it does. The cost for a developer to implement the unit test is far less than any other time spent trying to fix broken code.

 

It also improves the confidence in changes made to the system, as any changes that cause a breakage will also be flagged by the unit test.

 

Ofcourse, it's easy to just say "everything should have a unit test", but it takes quite a structured approach to development. And yes, there are a lot of developers in the games industry that do not write unit tests (far more don't than do to be honest).

 

However, unit tests will help you find bugs earlier, improve the robustness of your code and, ultimately, save you time (and money) in development.

 

n!

Edited by nfactorial
0

Share this post


Link to post
Share on other sites

When I say "ultimately" I don't mean "at the end of development" I mean "the core benefit you gain". "Ultimately" here is a constant throughout development.

 

n!

0

Share this post


Link to post
Share on other sites

As a sidenote, I imagine that the type of unit testing that you prioritize, will change depending on circumstances as well, plus how seasoned you are as a programmer. Like, certain functions and classes would be reviewed a lot less by an experienced pro, because he's so used to writing them that he knows better what to look for and whatnot. Or that he's just able to write code more definitively than a rookie, i.e. not having to refactor as much because he can deal with much more complexity at a time. That could reduce the possible exceptions, since less code means less areas where an exception would occur.

0

Share this post


Link to post
Share on other sites

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes. Just as, for example, I'd be sure Carmack always starts a new engine by rendering a cube (or some simple primitive).

 

Whilst writing the test itself does consume time (and writing tests themselves, does not take much time) it will save much more time in the future than the tests took to write.

 

Also, writing a test means you get to test the API you have written and give you earlier ideas for improvement. In test driven development you write the test before you even begin the system, the test will fail. You then write the system so the test does not fail and any breaking changes in the future will be immediately caught. There are better resources on the web than I could give here, so if anyone is interested I would recommend searching the web for "Test Driven Development".

 

Writing tests is not about "it takes more time" but rather it saves you both time and money.

 

n!

0

Share this post


Link to post
Share on other sites

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes.

 

I know, I'm just talking about the ratio of it and the idea that one should first test the things that one is unfamiliar with, because that's a more likely place where errors will occur.

0

Share this post


Link to post
Share on other sites

I am possibly misunderstanding, there is no "ratio". If you write a system, then you write a test for it first.

 

But, I think I misunderstood :)

n!

-1

Share this post


Link to post
Share on other sites

I think the sweet spot for automatic tests are at subsystem boundaries. This way you don't have to spend time writing tests for every class "just becuse", but you still verify that the subsystem, as a whole, works as expected. Which is what matters.

 

Of course, this assumes some kind of division into subsystems, which may not always be there.

0

Share this post


Link to post
Share on other sites

I'm not a game programmer but I am a commercial systems programmer, and automated unit tests are now stock-in-trade. We tend to use the tools in two ways (we use C#, so nUnit or MSTest are our toolkit)

 

1) To investigate the behaviour of a unit of code when writing it - code the test alongside the unit. You end up doing far less debugging step-through type stuff that way.

2) To formally test the boundary conditions of the code to make sure critical sections (like multithreaded code) works as designed, and in a way which would only be evident at runtime.

 

Both styles of tests are checked in to the source code control system at the same time, and we pump out statistics that give us code test coverage with each build - if it falls below a certain percentage we start to get twitchy - but there is no set limit. Typically code which is both complex (cyclomatic complexity) and not covered by unit tests, is flagged as being a hotspot and we then pick up some additional testing.

 

In my experience, and having come to automated testing late, I wouldn't now code without it. You dont actually need a unit test toolkit since these are just harnesses that allow you to execute a single portion of your code - you could do the same with a parameterised command line application if you wanted to. I'm not dogmatic about TDD - but in my experience programmers work faster and more accurately if they write unit tests at the same time as their line-of-business code. Its actually cheaper, and not just in the long run - any project greater than a couple of days will be made more accurate and more timely if the programmer codes tests at the same time.

 

As an example in the game world, in my hobby terrain project, I used unit tests to isolate and explore the behaviour of my mesh clipping and frustum culling, to make sure it was behaving as I expected. It also means I can semi-confidently change a portion of otherwise complex code, and use my unit tests as a "trigger" to spot any changes to external behaviour that I didn't know about. At its most extreme in the commercial world I've seen a big drop in the amount of paperwork specifications and "explanatory" documents needing to be produced by programmers - although this outcome isn't the goal, the fact that the behaviour of the code is explained in a series of unit tests does help incoming programmers.

 

The last thing to say is that using automated unit tests leads you to write better code - not just less bugs, but more structured, more isolated and more open and allows much more confident maintenance of complex systems, and less time spent in marathon debugging sessions.

 

Phillip

1

Share this post


Link to post
Share on other sites

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

0

Share this post


Link to post
Share on other sites

 

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

 

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

 

Phillip

Edited by PhillipHamlyn
0

Share this post


Link to post
Share on other sites

 

 

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

 

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

 

Phillip

 

 

Its not really about the skill of the programmer who writes the code in the first place, in a professional setting(especially in business programming where code tend to stick around for a very long time) you will have a wide variety of people touching the code, experienced developers retire or change job, fresh graduates gets hired, code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers, if the experts don't do their job and write proper tests you can be fairly certain that someone will manage to break the code at some point without anyone noticing until the customers start complaining.

Edited by SimonForsman
0

Share this post


Link to post
Share on other sites

code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers

 

Please name me a single traditional game engine that has 20+ years of life in it.

 

World of Warcraft is one of the longest running major games out there today, and it hasn't had it's 9th birthday yet. 

 

Other long-running engines include the annual sports games like Madden that is 25 years old and FIFA that is 20 years old, but those have been rewritten many times over their lifetime. 

 

It is true that certain long-running libraries and components are deserving of unit tests. I haven't seen anybody write that they shouldn't have unit tests.  But other parts of the game are short-lived and the code is effectively dead when the disc image is released to manufacturing, and these would be foolish to write unit tests for.

2

Share this post


Link to post
Share on other sites

One aspect of writing tests in advance (basically either kind of test) is that you're forced to think about corner cases and special circumstances to test. More than once, the result was being aware of those special cases before writing the actual implementation, resulting in code that had far fewer bugs from the start and ended up much cleaner, because you didn't end up fiddling in bug fixes after the fact.

 

If you write tests, I'd suggest doing so before you finish planing your actual implementation. Plan your system tests in parallel with the requirements (sometimes during writing a test you might realize there are requirements nobody thought about yet). Unit tests should be done in parallel with planing the interface for the unit (as sometimes writing the test will make you realize that your interface is broken). Afterwards during implementation, you will most likely go back and add more tests as you write test-worthy functions or algorithm.

1

Share this post


Link to post
Share on other sites

Let me give a real-world example of the business case.

 

Over the years I have seen many articles, perhaps even up to the hundreds, that tout the number as approximately 1:1 when it comes to comprehensive automated testing and green development. Since we track our progress carefully, I can agree with those numbers in my personal experience. It takes about twice as long to write code+tests as it does to just write code without tests. 

 

 

Frob, my experience is the opposite of yours - my experience is that it takes half as long to get code working when tests + code are written together. Yes, there is twice as much code, but half as many mistakes, meaning the development cycle is shortened and cheaper. It does depend on lots of factors, as you describe, but the writing of code is not the majority of the cost of getting it finished - working code is the goal, and unit tests help develop working code. Its interesting that people on the board seem to have quite divergent experiences of this, which refelects something about the industry ... but I dont know what :-)

0

Share this post


Link to post
Share on other sites

 

code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers

 

Please name me a single traditional game engine that has 20+ years of life in it.

 

World of Warcraft is one of the longest running major games out there today, and it hasn't had it's 9th birthday yet. 

 

Other long-running engines include the annual sports games like Madden that is 25 years old and FIFA that is 20 years old, but those have been rewritten many times over their lifetime. 

 

It is true that certain long-running libraries and components are deserving of unit tests. I haven't seen anybody write that they shouldn't have unit tests.  But other parts of the game are short-lived and the code is effectively dead when the disc image is released to manufacturing, and these would be foolish to write unit tests for.

 

 

I never mentioned games, the post i replied to was talking about business software.

0

Share this post


Link to post
Share on other sites

My philosophy with code is that if I haven't seen every single feature actually work, then the code isn't finished yet.

Under this definition, if writing the code+tests takes twice as long as just the code, that doesn't matter. When you've just written the code, the task isn't finished. After you've passed the tests, the task is finished (hopefully). So you're comparing the time taken to half-finish a task with the time taken to finish a task wink.png

 

If I'm not writing tests for something, I'll usually at least put a breakpoint in it once and step through it line by line, which is basically a semi-automated "desk check", which used to be a standard requirement for programming...

 

This is especially true when writing games in C++ -- all too often it's possible to write code that looks correct, and seems to behave correct from the outside, but a simple desk-check reveals an obvious off-by-one or overflow or edge-case bug.

 

If the code is too complex to easily step through all the possible use-cases, then I'll write a unit test instead.

 

The linked video about writing testable code contains good advice, btw. Not just for testing's sake, but also for writing flexible, reusable code in general.

 

 

 


Please name me a single traditional game engine that has 20+ years of life in it.

Yes this is not common, engines from the 80's/90's/00's are each very different from the engines of today... but 10 years of code maintenance might happen, a small bit wink.png

I bet there's still some small smidgen of code from Quake 1 hanging around in the Source engine.

When I was at Krome, the coding convention required every function to be commented with the original author and date. I saw a few nuggets deep within the 3rd rewrite of their engine that were dated year 2000, and this was in 2010.

Edited by Hodgman
2

Share this post


Link to post
Share on other sites

 

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

 

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

Usually, the point of unit testing is that if the units are well tested and correct all errors that come up in the integration between those units are due to the integration logic itself (e.g. module X crashes with the actual game engine because, unlike in X's unit tests, it isn't initialized properly) or to misunderstandings between modules (e.g. both the texture loader and the texture resource manager flip textures vertically, resulting in upside down graphics). It's much better than testing everything with a chance of bugs of any kind.

0

Share this post


Link to post
Share on other sites

Would you want a programmer spending time writing units tests (especially if it's 1:1 or even if it's 1/2 the amount of time) at $150k/yr+, or a QA guy test the code for $30k/yr? I think it's a valid question on where you want your time/money spent. There is no doubt that TDD results in better code, but that's not always the requirement. Sometimes 'good enough' and getting it done fast and cheap is the requirement.

 

 


If I'm not writing tests for something, I'll usually at least put a breakpoint in it once and step through it line by line, which is basically a semi-automated "desk check", which used to be a standard requirement for programming...

 

This is especially true when writing games in C++ -- all too often it's possible to write code that looks correct, and seems to behave correct from the outside, but a simple desk-check reveals an obvious off-by-one or overflow or edge-case bug.

 

In my view stepping through code is a must. Every game/app I've ever written, which haven't been massive amount of code, I've always stepped through every line at one point or another. Every time I implement a class or couple methods I step through them to make sure they do what I expect.

 

Your comment makes it sound like that's not a common thing to do? I have to assume it's very common and probably done more then unit testing between all the different levels of programmers around.

0

Share this post


Link to post
Share on other sites


In my view stepping through code is a must. Every game/app I've ever written, which haven't been massive amount of code, I've always stepped through every line at one point or another. Every time I implement a class or couple methods I step through them to make sure they do what I expect.

In my experience when I started using Unit TDD and Accepance TDD, I've never had to step through every line. That's what my tests are doing.

 

I'm also surprised that no one mentioned that writing unit tests improves the design of the application. It enforces separation of concerns, single responsibility principle, dependency inversion, etc., because it becomes difficult to unit test classes if they violate these principles. This, in turn, improves maintainability.

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0