Appropriate number of bugs

Started by
17 comments, last by alvaro 10 years, 5 months ago

Our project Burnt Islands reminds me more and more about the project "European Air War". We have sometimes so many bugs that a huge amount of time goes to fixing them and not actually working on new features.

So, as I look at our issue tracker (Redmine) right now we have approximately 50% bugs and about 50% features and this is after fixing a whole lot of them for the past 3 days. As we like to rest and enjoy some Thursday - Saturday drinks, one of us thought that it could be motivating if we needed to fix at least 3 bugs or features in order to earn the right to get a drink!

It became a kind of competition where the one who is not working at a day job is able to do three quick bugs due to the knowledge of the code base, while the other one needed to sit for 2 hours to get a drink!

Exhausting but also a real motivation to get things done. One of the days it took so long time to figure out the fix that it was already too late to drink! Well, it resulted that complex tasks are now divided into smaller ones so that it doesn't take too much time to fix them.

Whatever works for you this what we are doing now. But is that right? How do other indie developers do things? Do you guys also have many bugs?

3-bugs-1-drink has the disadvantage that larger and more complex bugs/features aren't started after a while. Maybe it's better to work 1 - 1.5 hours on the night shift before getting a drink?

One of us works as an enterprise application programmer at her day job and here is how the big guys do things:

- all major tasks are being specified and analyzed beforehand and is then divided into smaller tasks.
- developers get those small tasks and implement them in a period of 1-3 weeks (Agile / Scrum method).
- at the end of each iteration everything that was done gets tested and bugs are being fixed on the fly.
- next iteration - new tasks.

If we were using this method for our game development then the number of bugs would grow so huge after 3 weeks that I think it could take up to 1 month just to fix them. If we used those 2-3 days at the end of the iteration and no more then the game could look pretty much like "Day one Garry's incident" game that is full of bugs and received a horrible review.

We don't want to release a bad game, so we wonder what is the best way to do things so that you implement new features and keep the bugs down?
(oh, we can't write bug-free code if anyone was wondering).

And what is the usual proportion of bugs vs new features in the game development industry?


Advertisement

I don't have a good intuition for the ratio of bugs/new features on any project I've worked on. Typically when a new features goes into the game we make sure that the people affected by the feature (designers for gameplay stuff, artists for tools or rendering features, etc) test it out and report their bugs. Those bugs are worked out as soon as possible, and the person responsible for that new feature doesn't start doing something else until the initial bug reports are cleared up. This has a number of benefits: many bugs will be caught early on, and they can be fixed quickly while the feature is still fresh in the engineer's mind, you can be relatively confident that your new features are pretty functional, and the remaining bugs will hopefully only be edge cases. Also, when you develop like this you tend to get a sense of how long a feature will take both to implement AND test, which helps you budget your time better.

Another thing to think about: are you doing all you can to catch bugs quickly? "Crash early and crash often" is a motto you'll hear every now and then. Basically: if you have an opportunity to detect that something is wrong in your code, you should do so, and crash if something is wrong. For example, if you have an array class, when people index into the array you should assert that the index is within bounds. If you have a pointer that you want to dereference you should assert that it is not null. If you're working in C# those things will throw exceptions by default, but the principle still stands: Crash early and crash often.

I have to imagine the design phase isn't very detailed. Are you guys just jumping right into code for new features or are you designing on paper in detail first the new feature and how it needs to connect with the existing codebase? How detailed is the overall project on paper? What's your project methodology? What kind of bugs are you seeing? If there is a very common bug that keeps coming up then you need to address that situation if possible. Categorize your bugs and learn from them.

You have to find out why you are getting so many bugs before you can do something about reducing them. Of course you aren't going to write bug free code, but there are a ton of things you can do to greatly reduce the amount of bugs. Are you unit testing? Have you thought about TDD? These are all in place to reduce bugs and make more robust code. You are feeling the reasons as to why these things exist.

oh, we can't write bug-free code if anyone was wondering

By saying this you are brushing under the rug the main issue. By asking how many bugs are common in the industry you are trying to almost justify your bugs and make your team feel better about their situation. If you feel there are too many bugs then you need to find out common methodologies that exist today to help you write less buggy code. The only way to fix too many bugs is to write code with less bugs. There is no getting around that.

We haven't seen any of your code and I've been on a project that had the worse possible codebase possible. I mean it was a disaster in every way shape or form. Nothing was good about it. There was no thought of the main design. It was just thrown together. It literally had zero chance to make it. The team just thought as long as they got something to work at one point in time it was good enough. Be honest and ask yourself if the games code is in this shape. Maybe get outside consulting to just look over the code design to see what their thoughts are on it. They don't have to write code or even fix it but be an objective 3rd party looking at your code. Maybe have a couple people outside look at it and take their feedback to heart.

Hi there.

Thanks for your time. The bugs we're facing are more or less "design bugs", which is as you say, a result from a lack of a detailed design document on paper. But on the other hand, when we started this project we didn't have any idea of what game we were making. We started by making a terrain generator, where you could play around (goof around) and destroy the terrain.

The code is written in C++, and we're using the Entity-Component model with messages for passing information / data between the different parts of the game. The technical part of the code is robust (namespaces, library wrapper projects, not reinventing the wheel etc).

For some of the actual bugs we're having at the current time of writing are:

- No shadow on certain objects

- Some objects intersect (physics model is smaller than the graphics model)

- The game sometimes crashes hard.

- Certain objects are not loaded when loading a savegame.

- Some Components aren't cooperating that well (the small buddies, your helpers, are just goofing around when they should be gathering resources for you)

- Player has the wrong orientation when starting / loading the game

- Trees aren't unloaded when the terrain under is unloaded.

It's just an example of some of the bugs present.

We have unit tests, but it is too time consuming to implement for many of the components of the game. There are tests for the code which is simple to test (math, vector++). It's not trivial to implement tests for unloading of objects when the terrain is unloaded, and it takes time to maintain those tests when the original code changes.

Are there any best practices when there isn't a detailed design document explaining every detail of the game?


We have unit tests, but it is too time consuming to implement for many of the components of the game.

I'm sure you see the irony of that statement now.

I would get a design doc asap. If your employees don't see the final end result it may be hard for them to really understand how to efficiently build something. Building something in isolation (which is basically what you are doing because what's next isn't known by anyone) can be hard if there isn't some master plan. Imagine building a house without a blueprint. Things wouldn't fit together very well at all and that sounds like some of the issues you have.

On the technical side I assume you have basically zero global variables? That's always a red flag for me and the first thing I look at.

Since you are using a messaging system for object communication (the heart and soul of your game) do you have a global document showing all the possible communications between all objects? If not it can get confusing as to what's actually happening as logic jumps all over the place. This document should be updated and talked about all the time as it's the way all your objects talk which in turn basically makes your entire game run the way you want it. If you don't have a map of this then you are driving blind and mistakes are easier to make. The messages have requirements/parameters and expect certain values most likely. If one person makes a message and assumes certain things and another person sends the message assuming something different then you'll get a bug. This would be a great thing to unit test. Inside each object receiving a message validate the message and it's parameters are what you would expect and the handling of the message acts appropriately, giving an error message that helps you narrow things down quickly. Error handling inside messages is often overlooked because when you write the handler you know how it should work, but when working on a project with multiple people they may not know the exact details.

If those are the kind of bugs you have, then go fix 'em. Most of those sound less like mistakes and more like incomplete features. Minor visual things like shadows and model intersections won't impede anyone's work, so they can be put off until later, but failing to load or unload objects should be fixed right away.

The helper guys not helping is just an incomplete feature, so if someone was scheduled to do it and didn't finish it, then add it to their schedule for next iteration. If they did everything that could be done for the initial implementation given the state of the rest of the game at the time, then add it to their schedule whenever all the other prerequisites are up to speed. When you have a lot of things that interact with eachother, it can take several rounds and a lot of meetings to finish them.

As you gain experience, you'll get better at the analysis phase and foresee more details before writing any code. But you should also test code as you write it. For example, the thing of facing the wrong direction when loading the game... that should be pretty obvious if you test the loading code in a few different levels to make sure it's working before checking it in to source control. And you shouldn't check "implement save/load" off the list if you know it's not done yet.

And certainly not all bugs are created equal. Personally, when I track down a really tough crash bug, it's satisfying enough just to finally understand it and add it to my knowledge base of what to look out for in the future. But if alcohol motivates you, and it takes you all night to fix a bug, then take off early the next day and party :)

Good answer from DekuTree64. A lot of your bugs sound like features that were checked in without being fully finished. It can be tempting to check something in when it's only 90% complete, because you want to keep moving forward and work on something else, but you'll find (and you HAVE found) that kind of behavior adds a burden on you in the long run. When you work that way you accrue a sort of "code debt," and like all debt, it's gotta be paid off eventually.

Slow your pace down a bit. Test as you go. Make sure your features are working before they get submitted. You'll sleep better at night.

Also, bugs in your physics system are to be expected. Make no mistake: you have to stay on top of those bugs, but don't get too discouraged if you've got a lot of physics bugs. Physics and collision detection are hard. Every project I've worked on has had a lot of physics related bugs. You just have to grind through those bugs (or use a physics library, but I've never worked on a project with a licensed physics library so I can't offer much advice about that).

One of my lecturers at the university once told a story about a programming course he'd previously taught to a mixed group of both software and hardware engineers. When reviewing the lab assignments he was astonished that the code submitted by the hardware engineers didn't have any bugs. When he asked them about it, they answered "oh, we didn't realize bugs were allowed".

There's only one answer to what's the appropriate number of bugs: ZERO. That's the mindset and the rest is just implementation.

openwar - the real-time tactical war-game platform

If you think unit testing takes to long, it's because you haven't used it enough to learn how to do it right. Once you get yourself setup with the right testing tools and get into the habit of test driven development, you will never go back. Right now the tests you are writing are probably just all wrong, because it's not something you have really bought into and made an effort to learn to do well.

In my experience the secret to good testing is to adhere strongly to the single responsibility principle, and mock/stub anything that's not part of what you are actually testing. These two things go hand in hand with writing tests that are easy to reason about, and actually valuable.

It will take you about twice as long to write your code when using test driven development, as a general rule. That seems like a lot until you realize just how costly it is to not find bugs until later on. The feedback time between when you write the code and find a bug is critical and largely determines the total cost of the bug. The longer it takes to find, the more expensive it is.

With C++ it's even worse. The best C++ developers out there have basically said that humans cannot reason about large C++ codebases, they just get too complex. So the process of tracking down a bug where you have no testing is painful and time consuming, and it will only get worse as your project grows.

Also, since integration testing seems to be pretty big in the game biz, I'll point out this really good talk that also has some good info on TDD in general:

http://www.infoq.com/presentations/integration-tests-scam

Chris

I don't think it's a good idea to mix business with pleasure - you'll end up ruining one or the other. I do like the Pavlov's dog idea and I know it does work when you just want to get things done, but if you do continue to use the beer-compensation method, I'd be careful not to make it sound like it's mandatory. Maybe some of you would rather have a coke, or a pizza instead. :)

If you think some of the developers are getting away too easily because they only fix minor bugs, you could task them with doing code review of other developer's code. i.e.: 3 bugs + 3 code reviews = 1 beer. That way, they would be obliged to understand the harder bugs as well,while the harder working guys will have it easier, because the minor bugs are easy to review.

You also mentioned that there's competition... I don't think it's so much a competition - rather sounds to me like you're allowing yourselves to speed through fixing things, just to get the tasks done. This is mostly because nobody likes fixing bugs, especially the developers with more advanced knowledge of the project - they will have the least interest in bugs, because they already know how those bugs could've been avoided in the first place. At least, I don't think a beer after work should have this sort of effect on them. :)

That being said, I never really liked the Agile development model (but this is just me), because it forces you to code fast, without thinking about possible bugs you might be introducing. I also don't like how it categorizes and splits work into fixed, periodic categories (or whatever they're called - 2 weeks planning, 2 weeks developing, 2 weeks testing, then repeat). Since you're not documenting much, just adding tasks, you will end up forgetting details of what was planned before you even start developing. Plus, when it comes to bugs, you really cannot plan for them. I always find it easier and faster to fix bugs as I find them, while I'm still involved in the task I'm working on, even if it's not related to my work. Instead, Agile just forces you to add a new task for the bug and move on.

This topic is closed to new replies.

Advertisement