Hey peeps, I thought I'd ask your opinions about this video, where the presenter talks about deeper testing methodology and specifically about unit testing. I think this is fundamentally going to improve my programming routines, and maybe it will help you as well, even if you're a semi-seasoned programmer:
Feel free to give some feedback and perhaps critique. It'd be nice to see some different perspectives on unit testing versus overall systems testing.
It is a cost/benefit analysis.
It is well documented that the initial cost to write automated unit tests is almost equal to the cost of writing the software when the two are written at the same time. The cost is potentially much higher if the tests are added after initial development. Drawn as a graph, the development costs look like a right-skewed distribution with a very long tail. The benefit comes in the long tail of support; during maintenance all the prior issues are immediately regressed. Often the break-even point is not reached until several years of maintenance.
For libraries that will be used extensively by many products, or shared between studios, or released publicly, or used for many years, then yes, unit tests generally make sense. The cost to manually test and verify after every change is often greater than the cost to create the automated tests.
However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.
However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.
But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.
What I've realized a lot more and also a bit through watching this video, is how important it is to code in the right way and to make it testable and to do tidbits of pinpoint testing while you code. Before I watched this video, I felt that the term "playtesting" is too vague. There's "systems testing" (which you only do at specific phases of development) and then there's the constant "functional testing" (which I erroneously called system testing for a while) of every code you write, immediately after you've written it. Just to see if it returns the values it's supposed to, or whatever.
I just felt like this video confirmed a lot of those thoughts, but I can also see that there's differences between doing solo projects and working in a team/company. Or even differences depending on how complex the program is in general, or what part of the program we're dealing with.
Unit tests should always be written, it is not a widely practiced art in games development at the moment but it is gaining ground.
However, for most game code that is written once and then discarded, unit tests generally do not make sense. The cost to hire an room full of QA testers to verify everything at the end of the project is often less than the cost to create the automated tests.
This is not true, any code written should have an associated unit test to prove that it does what it says it does. The cost for a developer to implement the unit test is far less than any other time spent trying to fix broken code.
It also improves the confidence in changes made to the system, as any changes that cause a breakage will also be flagged by the unit test.
Ofcourse, it's easy to just say "everything should have a unit test", but it takes quite a structured approach to development. And yes, there are a lot of developers in the games industry that do not write unit tests (far more don't than do to be honest).
However, unit tests will help you find bugs earlier, improve the robustness of your code and, ultimately, save you time (and money) in development.
n!
But the way I understood it, was that unit testing was preferably something that the programmer himself does concurrently with the coding. Sort of a way to reduce the need for systems testing later on. Or maybe I misunderstood the context, here.
I think he means that having all the programmers spending time (cost) on making unit tests while coding the actual functionality, combined throughout the entire project is > in cost than just having them code the functionality of the app and just do normal testing with QA people. That changes to < over x number of years your code is actually running, but most games don't really run or get maintained over a pretty small number of years.
If you are doing an MMO then this would obviously make a ton of sense because your game could be running for decades, but most games probably get a couple years of action.
I think it's easy for programmers to always say let's do unit testing, but the reality is budgets, time, etc all play into things. You may always want to make the best game possible but the people who gave you money may just want a decent game out early and on a tight budget.
ultimately, save you time in development.
But when does that "ultimately" happen? If your game is good enough to make the money it was projected, then the people who gave you money might not care.
It's situational and the real world often gets in the way. In an ideal world, sure. It's all about the money.
When I say "ultimately" I don't mean "at the end of development" I mean "the core benefit you gain". "Ultimately" here is a constant throughout development.
n!
As a sidenote, I imagine that the type of unit testing that you prioritize, will change depending on circumstances as well, plus how seasoned you are as a programmer. Like, certain functions and classes would be reviewed a lot less by an experienced pro, because he's so used to writing them that he knows better what to look for and whatnot. Or that he's just able to write code more definitively than a rookie, i.e. not having to refactor as much because he can deal with much more complexity at a time. That could reduce the possible exceptions, since less code means less areas where an exception would occur.
You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes. Just as, for example, I'd be sure Carmack always starts a new engine by rendering a cube (or some simple primitive).
Whilst writing the test itself does consume time (and writing tests themselves, does not take much time) it will save much more time in the future than the tests took to write.
Also, writing a test means you get to test the API you have written and give you earlier ideas for improvement. In test driven development you write the test before you even begin the system, the test will fail. You then write the system so the test does not fail and any breaking changes in the future will be immediately caught. There are better resources on the web than I could give here, so if anyone is interested I would recommend searching the web for "Test Driven Development".
Writing tests is not about "it takes more time" but rather it saves you both time and money.
n!
You would generally apply unit tests (or, more specifically, test driven development) when writing a system. It doesn't matter how experienced of a programmer you are, people always can (and will) make mistakes.
I know, I'm just talking about the ratio of it and the idea that one should first test the things that one is unfamiliar with, because that's a more likely place where errors will occur.
I am possibly misunderstanding, there is no "ratio". If you write a system, then you write a test for it first.
But, I think I misunderstood :)
n!
