Jump to content

  • Log In with Google      Sign In   
  • Create Account


Unit Testing ftw?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
36 replies to this topic

#1 Malabyte   Members   -  Reputation: 583

Like
0Likes
Like

Posted 10 October 2013 - 08:30 AM

Hey peeps, I thought I'd ask your opinions about this video, where the presenter talks about deeper testing methodology and specifically about unit testing. I think this is fundamentally going to improve my programming routines, and maybe it will help you as well, even if you're a semi-seasoned programmer:
 

 

Feel free to give some feedback and perhaps critique. It'd be nice to see some different perspectives on unit testing versus overall systems testing.


Edited by Malabyte, 10 October 2013 - 08:32 AM.

- Awl you're base are belong me! -

- I don't know, I'm just a noob -


Sponsor:

#2 frob   Moderators   -  Reputation: 18390

Like
8Likes
Like

Posted 10 October 2013 - 09:30 AM

It is a cost/benefit analysis.

 

 

It is well documented that the initial cost to write automated unit tests is almost equal to the cost of writing the software when the two are written at the same time. The cost is potentially much higher if the tests are added after initial development. Drawn as a graph, the development costs look like a right-skewed distribution with a very long tail. The benefit comes in the long tail of support; during maintenance all the prior issues are immediately regressed. Often the break-even point is not reached until several years of maintenance.

 

 

For libraries that will be used extensively by many products, or shared between studios, or released publicly, or used for many years, then yes, unit tests generally make sense. The cost to manually test and verify after every change is often greater than the cost to create the automated tests.

 

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.


Check out my personal indie blog at bryanwagstaff.com.

#3 Malabyte   Members   -  Reputation: 583

Like
0Likes
Like

Posted 10 October 2013 - 09:47 AM

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

 

But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

 

What I've realized a lot more and also a bit through watching this video, is how important it is to code in the right way and to make it testable and to do tidbits of pinpoint testing while you code. Before I watched this video, I felt that the term "playtesting" is too vague. There's "systems testing" (which you only do at specific phases of development) and then there's the constant "functional testing" (which I erroneously called system testing for a while) of every code you write, immediately after you've written it. Just to see if it returns the values it's supposed to, or whatever.

 

I just felt like this video confirmed a lot of those thoughts, but I can also see that there's differences between doing solo projects and working in a team/company. Or even differences depending on how complex the program is in general, or what part of the program we're dealing with.


Edited by Malabyte, 10 October 2013 - 10:33 AM.

- Awl you're base are belong me! -

- I don't know, I'm just a noob -


#4 nfactorial   Members   -  Reputation: 712

Like
0Likes
Like

Posted 10 October 2013 - 10:42 AM

Unit tests should always be written, it is not a widely practiced art in games development at the moment but it is gaining ground.

 

 

 

However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.

 

This is not true, any code written should have an associated unit test to prove that it does what it says it does. The cost for a developer to implement the unit test is far less than any other time spent trying to fix broken code.

 

It also improves the confidence in changes made to the system, as any changes that cause a breakage will also be flagged by the unit test.

 

Ofcourse, it's easy to just say "everything should have a unit test", but it takes quite a structured approach to development. And yes, there are a lot of developers in the games industry that do not write unit tests (far more don't than do to be honest).

 

However, unit tests will help you find bugs earlier, improve the robustness of your code and, ultimately, save you time (and money) in development.

 

n!


Edited by nfactorial, 10 October 2013 - 10:43 AM.


#5 rpiller   Members   -  Reputation: 651

Like
3Likes
Like

Posted 10 October 2013 - 10:50 AM


But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.

 

I think he means that having all the programmers spending time (cost) on making unit tests while coding the actual functionality, combined throughout the entire project is > in cost than just having them code the functionality of the app and just do normal testing with QA people. That changes to < over x number of years your code is actually running, but most games don't really run or get maintained over a pretty small number of years.

 

If you are doing an MMO then this would obviously make a ton of sense because your game could be running for decades, but most games probably get a couple years of action.

 

I think it's easy for programmers to always say let's do unit testing, but the reality is budgets, time, etc all play into things. You may always want to make the best game possible but the people who gave you money may just want a decent game out early and on a tight budget.

 

 


ultimately, save you time in development.

 

But when does that "ultimately" happen? If your game is good enough to make the money it was projected, then the people who gave you money might not care.

 

It's situational and the real world often gets in the way. In an ideal world, sure. It's all about the money.


Edited by rpiller, 10 October 2013 - 10:51 AM.


#6 nfactorial   Members   -  Reputation: 712

Like
0Likes
Like

Posted 10 October 2013 - 11:25 AM

When I say "ultimately" I don't mean "at the end of development" I mean "the core benefit you gain". "Ultimately" here is a constant throughout development.

 

n!



#7 Malabyte   Members   -  Reputation: 583

Like
0Likes
Like

Posted 10 October 2013 - 12:41 PM

As a sidenote, I imagine that the type of unit testing that you prioritize, will change depending on circumstances as well, plus how seasoned you are as a programmer. Like, certain functions and classes would be reviewed a lot less by an experienced pro, because he's so used to writing them that he knows better what to look for and whatnot. Or that he's just able to write code more definitively than a rookie, i.e. not having to refactor as much because he can deal with much more complexity at a time. That could reduce the possible exceptions, since less code means less areas where an exception would occur.


- Awl you're base are belong me! -

- I don't know, I'm just a noob -


#8 nfactorial   Members   -  Reputation: 712

Like
0Likes
Like

Posted 10 October 2013 - 12:53 PM

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes. Just as, for example, I'd be sure Carmack always starts a new engine by rendering a cube (or some simple primitive).

 

Whilst writing the test itself does consume time (and writing tests themselves, does not take much time) it will save much more time in the future than the tests took to write.

 

Also, writing a test means you get to test the API you have written and give you earlier ideas for improvement. In test driven development you write the test before you even begin the system, the test will fail. You then write the system so the test does not fail and any breaking changes in the future will be immediately caught. There are better resources on the web than I could give here, so if anyone is interested I would recommend searching the web for "Test Driven Development".

 

Writing tests is not about "it takes more time" but rather it saves you both time and money.

 

n!



#9 Malabyte   Members   -  Reputation: 583

Like
0Likes
Like

Posted 10 October 2013 - 01:23 PM

You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes.

 

I know, I'm just talking about the ratio of it and the idea that one should first test the things that one is unfamiliar with, because that's a more likely place where errors will occur.


- Awl you're base are belong me! -

- I don't know, I'm just a noob -


#10 nfactorial   Members   -  Reputation: 712

Like
-1Likes
Like

Posted 10 October 2013 - 01:31 PM

I am possibly misunderstanding, there is no "ratio". If you write a system, then you write a test for it first.

 

But, I think I misunderstood :)

n!



#11 wack   Members   -  Reputation: 1214

Like
0Likes
Like

Posted 10 October 2013 - 01:51 PM

I think the sweet spot for automatic tests are at subsystem boundaries. This way you don't have to spend time writing tests for every class "just becuse", but you still verify that the subsystem, as a whole, works as expected. Which is what matters.

 

Of course, this assumes some kind of division into subsystems, which may not always be there.



#12 PhillipHamlyn   Members   -  Reputation: 454

Like
1Likes
Like

Posted 10 October 2013 - 02:12 PM

I'm not a game programmer but I am a commercial systems programmer, and automated unit tests are now stock-in-trade. We tend to use the tools in two ways (we use C#, so nUnit or MSTest are our toolkit)

 

1) To investigate the behaviour of a unit of code when writing it - code the test alongside the unit. You end up doing far less debugging step-through type stuff that way.

2) To formally test the boundary conditions of the code to make sure critical sections (like multithreaded code) works as designed, and in a way which would only be evident at runtime.

 

Both styles of tests are checked in to the source code control system at the same time, and we pump out statistics that give us code test coverage with each build - if it falls below a certain percentage we start to get twitchy - but there is no set limit. Typically code which is both complex (cyclomatic complexity) and not covered by unit tests, is flagged as being a hotspot and we then pick up some additional testing.

 

In my experience, and having come to automated testing late, I wouldn't now code without it. You dont actually need a unit test toolkit since these are just harnesses that allow you to execute a single portion of your code - you could do the same with a parameterised command line application if you wanted to. I'm not dogmatic about TDD - but in my experience programmers work faster and more accurately if they write unit tests at the same time as their line-of-business code. Its actually cheaper, and not just in the long run - any project greater than a couple of days will be made more accurate and more timely if the programmer codes tests at the same time.

 

As an example in the game world, in my hobby terrain project, I used unit tests to isolate and explore the behaviour of my mesh clipping and frustum culling, to make sure it was behaving as I expected. It also means I can semi-confidently change a portion of otherwise complex code, and use my unit tests as a "trigger" to spot any changes to external behaviour that I didn't know about. At its most extreme in the commercial world I've seen a big drop in the amount of paperwork specifications and "explanatory" documents needing to be produced by programmers - although this outcome isn't the goal, the fact that the behaviour of the code is explained in a series of unit tests does help incoming programmers.

 

The last thing to say is that using automated unit tests leads you to write better code - not just less bugs, but more structured, more isolated and more open and allows much more confident maintenance of complex systems, and less time spent in marathon debugging sessions.

 

Phillip



#13 Malabyte   Members   -  Reputation: 583

Like
0Likes
Like

Posted 10 October 2013 - 02:53 PM

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).


- Awl you're base are belong me! -

- I don't know, I'm just a noob -


#14 PhillipHamlyn   Members   -  Reputation: 454

Like
0Likes
Like

Posted 10 October 2013 - 03:25 PM

 

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

 

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

 

Phillip


Edited by PhillipHamlyn, 10 October 2013 - 03:26 PM.


#15 SimonForsman   Crossbones+   -  Reputation: 5720

Like
0Likes
Like

Posted 10 October 2013 - 04:32 PM

 

 

I am possibly misunderstanding, there is no "ratio".

 

By "ratio", I meant the ratio at which the programmer makes mistakes. A beginner makes more mistakes than a pro, and they make the mistakes in areas where I'm arguing that pros would perhaps not make mistakes as often. So as such, the need for testing a given segment of code could potentially be different for a newbie than for a pro, because chances are that the mistakes that the pro makes are going to be of a more esoteric nature (because he's exploring much deeper parts of the rabbit's hole, in his programming).

 

Interesting - I'd see the ratio as inversed - a professional and highly competent programmer would be expected to be expert in using all the tools to their advantage, and would tend to write more unit tests; a less experienced programmer would believe their code to be bug free and build an untested monument to esotericism (is that a word ?). I know that is provocative, but I've personal experience of a programmer failing an interview where they needed to show off their C# skills because it took them 30 minutes to write their first test - (I think thats stupid by the way) but automated unit tests have penetrated standard business programming to such a degree that someone representing themselves as an expert programmer who did not routinely practise automated unit tests as part of their arsenal of skills might find it difficult to get a job when compared to a more junior one that did. Maybe thats actually what you meant in your post - that the ratio of tests expected of the professional and skilled programmer is higher than that of a newbie, who will still be a hero programmer who thinks they make no mistakes.

 

Phillip

 

 

Its not really about the skill of the programmer who writes the code in the first place, in a professional setting(especially in business programming where code tend to stick around for a very long time) you will have a wide variety of people touching the code, experienced developers retire or change job, fresh graduates gets hired, code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers, if the experts don't do their job and write proper tests you can be fairly certain that someone will manage to break the code at some point without anyone noticing until the customers start complaining.


Edited by SimonForsman, 10 October 2013 - 04:33 PM.

I don't suffer from insanity, I'm enjoying every minute of it.
The voices in my head may not be real, but they have some good ideas!

#16 frob   Moderators   -  Reputation: 18390

Like
4Likes
Like

Posted 10 October 2013 - 04:42 PM

Let me give a real-world example of the business case.

 

Over the years I have seen many articles, perhaps even up to the hundreds, that tout the number as approximately 1:1 when it comes to comprehensive automated testing and green development. Since we track our progress carefully, I can agree with those numbers in my personal experience. It takes about twice as long to write code+tests as it does to just write code without tests. 

 

 

We have a game engine that has been in use since 2005. This engine has many hundred thousand unit tests in it. The physics library has unit test, the pathfinding has unit tests, the spatial systems have unit tests, the animation components have unit tests, and so on. If someone were to introduce a significant bug in the system it could take down not just the one developer, but potentially take down the entire team for several days. Such critical bugs do sometimes get introduced, and by the time you factor in just a single day of lost productivity among all people on all teams the cost of such a bug can reach around 150 work-days lost. This is exactly the type of scenario the video is talking about. For this type of engine, a long-term project shared by many people, or a project in long-term maintenance, there is a compelling reason to write the tests.

 

Now let's look at the other side, the people who use the engine.

 

Developing a new item with this engine requires about 3 weeks of development time followed by about 2 weeks of QA time. Programmer hours are expensive, QA hours are cheap. When a new item is developed it is launched, gets put out the door, and is generally never touched again. The code is effectively thrown away.  Of course it is not completely thrown out, we routinely pull code from prior items and reuse them, and some of it gets incorporated back to the engine where tests are developed for it. Now over the course of several years and several hundred items we have only had three items where customers found significant bugs that required a re-release. The cost of these bugs is practically zero, and unit tests would not have helped avoid them. The cost to do a single QA push is far less than the additional cost of writing the automated tests. The automated tests have no lasting value because the code itself has no lasting value. 

 

I am a strong proponent of TDD when it makes sense. I love it.  The red-green-refactor mantra plays in my head even when I writing throw-away script code because in practice I still follow it: I have a thing to change, I make the change, I test the change, I clean up, I retest, and I submit the code. However, I am an opponent of TDD when it does not make sense. It does not make sense in code which has no long-term value.


Check out my personal indie blog at bryanwagstaff.com.

#17 frob   Moderators   -  Reputation: 18390

Like
2Likes
Like

Posted 10 October 2013 - 04:50 PM

code written by an expert today may be modified to incorporate new features 10 or even 20+ years from now by an entierly different group of developers

 

Please name me a single traditional game engine that has 20+ years of life in it.

 

World of Warcraft is one of the longest running major games out there today, and it hasn't had it's 9th birthday yet. 

 

Other long-running engines include the annual sports games like Madden that is 25 years old and FIFA that is 20 years old, but those have been rewritten many times over their lifetime. 

 

It is true that certain long-running libraries and components are deserving of unit tests. I haven't seen anybody write that they shouldn't have unit tests.  But other parts of the game are short-lived and the code is effectively dead when the disc image is released to manufacturing, and these would be foolish to write unit tests for.


Check out my personal indie blog at bryanwagstaff.com.

#18 Khatharr   Crossbones+   -  Reputation: 2773

Like
0Likes
Like

Posted 10 October 2013 - 07:26 PM

This video killed all of my interest in automated unit testing.


void hurrrrrrrr() {__asm sub [ebp+4],5;}

There are ten kinds of people in this world: those who understand binary and those who don't.

#19 Trienco   Crossbones+   -  Reputation: 2041

Like
1Likes
Like

Posted 10 October 2013 - 09:44 PM

One aspect of writing tests in advance (basically either kind of test) is that you're forced to think about corner cases and special circumstances to test. More than once, the result was being aware of those special cases before writing the actual implementation, resulting in code that had far fewer bugs from the start and ended up much cleaner, because you didn't end up fiddling in bug fixes after the fact.

 

If you write tests, I'd suggest doing so before you finish planing your actual implementation. Plan your system tests in parallel with the requirements (sometimes during writing a test you might realize there are requirements nobody thought about yet). Unit tests should be done in parallel with planing the interface for the unit (as sometimes writing the test will make you realize that your interface is broken). Afterwards during implementation, you will most likely go back and add more tests as you write test-worthy functions or algorithm.


f@dzhttp://festini.device-zero.de

#20 PhillipHamlyn   Members   -  Reputation: 454

Like
0Likes
Like

Posted 11 October 2013 - 01:56 AM

Let me give a real-world example of the business case.

 

Over the years I have seen many articles, perhaps even up to the hundreds, that tout the number as approximately 1:1 when it comes to comprehensive automated testing and green development. Since we track our progress carefully, I can agree with those numbers in my personal experience. It takes about twice as long to write code+tests as it does to just write code without tests. 

 

 

Frob, my experience is the opposite of yours - my experience is that it takes half as long to get code working when tests + code are written together. Yes, there is twice as much code, but half as many mistakes, meaning the development cycle is shortened and cheaper. It does depend on lots of factors, as you describe, but the writing of code is not the majority of the cost of getting it finished - working code is the goal, and unit tests help develop working code. Its interesting that people on the board seem to have quite divergent experiences of this, which refelects something about the industry ... but I dont know what :-)






Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS