Jump to content

  • Log In with Google      Sign In   
  • Create Account


Appropriate number of bugs


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
18 replies to this topic

#1 Freya   Members   -  Reputation: 801

Like
0Likes
Like

Posted 10 November 2013 - 03:05 PM

Our project Burnt Islands reminds me more and more about the project "European Air War". We have sometimes so many bugs that a huge amount of time goes to fixing them and not actually working on new features.

So, as I look at our issue tracker (Redmine) right now we have approximately 50% bugs and about 50% features and this is after fixing a whole lot of them for the past 3 days. As we like to rest and enjoy some Thursday - Saturday drinks, one of us thought that it could be motivating if we needed to fix at least 3 bugs or features in order to earn the right to get a drink!

It became a kind of competition where the one who is not working at a day job is able to do three quick bugs due to the knowledge of the code base, while the other one needed to sit for 2 hours to get a drink!

Exhausting but also a real motivation to get things done. One of the days it took so long time to figure out the fix that it was already too late to drink! Well, it resulted that complex tasks are now divided into smaller ones so that it doesn't take too much time to fix them.

Whatever works for you this what we are doing now. But is that right? How do other indie developers do things? Do you guys also have many bugs?

3-bugs-1-drink has the disadvantage that larger and more complex bugs/features aren't started after a while. Maybe it's better to work 1 - 1.5 hours on the night shift before getting a drink?

One of us works as an enterprise application programmer at her day job and here is how the big guys do things:

- all major tasks are being specified and analyzed beforehand and is then divided into smaller tasks.
- developers get those small tasks and implement them in a period of 1-3 weeks (Agile / Scrum method).
- at the end of each iteration everything that was done gets tested and bugs are being fixed on the fly.
- next iteration - new tasks.

If we were using this method for our game development then the number of bugs would grow so huge after 3 weeks that I think it could take up to 1 month just to fix them. If we used those 2-3 days at the end of the iteration and no more then the game could look pretty much like "Day one Garry's incident" game that is full of bugs and received a horrible review.

We don't want to release a bad game, so we wonder what is the best way to do things so that you implement new features and keep the bugs down?
(oh, we can't write bug-free code if anyone was wondering).

And what is the usual proportion of bugs vs new features in the game development industry?


Working on a game "Burnt islands".
Dev blog       
"Burnt islands" on Indie DB

Sponsor:

#2 Samith   Members   -  Reputation: 2115

Like
2Likes
Like

Posted 10 November 2013 - 03:20 PM

I don't have a good intuition for the ratio of bugs/new features on any project I've worked on. Typically when a new features goes into the game we make sure that the people affected by the feature (designers for gameplay stuff, artists for tools or rendering features, etc) test it out and report their bugs. Those bugs are worked out as soon as possible, and the person responsible for that new feature doesn't start doing something else until the initial bug reports are cleared up. This has a number of benefits: many bugs will be caught early on, and they can be fixed quickly while the feature is still fresh in the engineer's mind, you can be relatively confident that your new features are pretty functional, and the remaining bugs will hopefully only be edge cases. Also, when you develop like this you tend to get a sense of how long a feature will take both to implement AND test, which helps you budget your time better.

 

Another thing to think about: are you doing all you can to catch bugs quickly? "Crash early and crash often" is a motto you'll hear every now and then. Basically: if you have an opportunity to detect that something is wrong in your code, you should do so, and crash if something is wrong. For example, if you have an array class, when people index into the array you should assert that the index is within bounds. If you have a pointer that you want to dereference you should assert that it is not null. If you're working in C# those things will throw exceptions by default, but the principle still stands: Crash early and crash often.



#3 rpiller   Members   -  Reputation: 657

Like
2Likes
Like

Posted 10 November 2013 - 04:58 PM

I have to imagine the design phase isn't very detailed. Are you guys just jumping right into code for new features or are you designing on paper in detail first the new feature and how it needs to connect with the existing codebase? How detailed is the overall project on paper? What's your project methodology? What kind of bugs are you seeing? If there is a very common bug that keeps coming up then you need to address that situation if possible. Categorize your bugs and learn from them.

 

You have to find out why you are getting so many bugs before you can do something about reducing them. Of course you aren't going to write bug free code, but there are a ton of things you can do to greatly reduce the amount of bugs. Are you unit testing? Have you thought about TDD? These are all in place to reduce bugs and make more robust code. You are feeling the reasons as to why these things exist. 

 

 

 

oh, we can't write bug-free code if anyone was wondering

 

By saying this you are brushing under the rug the main issue. By asking how many bugs are common in the industry you are trying to almost justify your bugs and make your team feel better about their situation. If you feel there are too many bugs then you need to find out common methodologies that exist today to help you write less buggy code. The only way to fix too many bugs is to write code with less bugs. There is no getting around that.

 

We haven't seen any of your code and I've been on a project that had the worse possible codebase possible. I mean it was a disaster in every way shape or form. Nothing was good about it. There was no thought of the main design. It was just thrown together. It literally had zero chance to make it. The team just thought as long as they got something to work at one point in time it was good enough. Be honest and ask yourself if the games code is in this shape. Maybe get outside consulting to just look over the code design to see what their thoughts are on it. They don't have to write code or even fix it but be an objective 3rd party looking at your code. Maybe have a couple people outside look at it and take their feedback to heart.


Edited by rpiller, 10 November 2013 - 04:59 PM.


#4 Freya   Members   -  Reputation: 801

Like
0Likes
Like

Posted 10 November 2013 - 05:53 PM

Hi there.

 

Thanks for your time. The bugs we're facing are more or less "design bugs", which is as you say, a result from a lack of a detailed design document on paper. But on the other hand, when we started this project we didn't have any idea of what game we were making. We started by making a terrain generator, where you could play around (goof around) and destroy the terrain.

 

The code is written in C++, and we're using the Entity-Component model with messages for passing information / data between the different parts of the game. The technical part of the code is robust (namespaces, library wrapper projects, not reinventing the wheel etc).

 

For some of the actual bugs we're having at the current time of writing are:

 

- No shadow on certain objects

- Some objects intersect (physics model is smaller than the graphics model)

- The game sometimes crashes hard.

- Certain objects are not loaded when loading a savegame.

- Some Components aren't cooperating that well (the small buddies, your helpers, are just goofing around when they should be gathering resources for you)

- Player has the wrong orientation when starting / loading the game

- Trees aren't unloaded when the terrain under is unloaded.

 

It's just an example of some of the bugs present.

 

We have unit tests, but it is too time consuming to implement for many of the components of the game. There are tests for the code which is simple to test (math, vector++). It's not trivial to implement tests for unloading of objects when the terrain is unloaded, and it takes time to maintain those tests when the original code changes.

 

Are there any best practices when there isn't a detailed design document explaining every detail of the game?


Working on a game "Burnt islands".
Dev blog       
"Burnt islands" on Indie DB

#5 rpiller   Members   -  Reputation: 657

Like
0Likes
Like

Posted 10 November 2013 - 06:54 PM

We have unit tests, but it is too time consuming to implement for many of the components of the game.

 

I'm sure you see the irony of that statement now.

 

I would get a design doc asap. If your employees don't see the final end result it may be hard for them to really understand how to efficiently build something. Building something in isolation (which is basically what you are doing because what's next isn't known by anyone) can be hard if there isn't some master plan. Imagine building a house without a blueprint. Things wouldn't fit together very well at all and that sounds like some of the issues you have.

 

On the technical side I assume you have basically zero global variables? That's always a red flag for me and the first thing I look at.

 

Since you are using a messaging system for object communication (the heart and soul of your game) do you have a global document showing all the possible communications between all objects? If not it can get confusing as to what's actually happening as logic jumps all over the place. This document should be updated and talked about all the time as it's the way all your objects talk which in turn basically makes your entire game run the way you want it. If you don't have a map of this then you are driving blind and mistakes are easier to make. The messages have requirements/parameters and expect certain values most likely. If one person makes a message and assumes certain things and another person sends the message assuming something different then you'll get a bug. This would be a great thing to unit test. Inside each object receiving a message validate the message and it's parameters are what you would expect and the handling of the message acts appropriately, giving an error message that helps you narrow things down quickly. Error handling inside messages is often overlooked because when you write the handler you know how it should work, but when working on a project with multiple people they may not know the exact details.



#6 DekuTree64   Members   -  Reputation: 972

Like
3Likes
Like

Posted 10 November 2013 - 07:39 PM

If those are the kind of bugs you have, then go fix 'em. Most of those sound less like mistakes and more like incomplete features. Minor visual things like shadows and model intersections won't impede anyone's work, so they can be put off until later, but failing to load or unload objects should be fixed right away.

 

The helper guys not helping is just an incomplete feature, so if someone was scheduled to do it and didn't finish it, then add it to their schedule for next iteration. If they did everything that could be done for the initial implementation given the state of the rest of the game at the time, then add it to their schedule whenever all the other prerequisites are up to speed. When you have a lot of things that interact with eachother, it can take several rounds and a lot of meetings to finish them.

 

As you gain experience, you'll get better at the analysis phase and foresee more details before writing any code. But you should also test code as you write it. For example, the thing of facing the wrong direction when loading the game... that should be pretty obvious if you test the loading code in a few different levels to make sure it's working before checking it in to source control. And you shouldn't check "implement save/load" off the list if you know it's not done yet.

 

And certainly not all bugs are created equal. Personally, when I track down a really tough crash bug, it's satisfying enough just to finally understand it and add it to my knowledge base of what to look out for in the future. But if alcohol motivates you, and it takes you all night to fix a bug, then take off early the next day and party :)


Edited by DekuTree64, 10 November 2013 - 07:50 PM.


#7 Samith   Members   -  Reputation: 2115

Like
2Likes
Like

Posted 10 November 2013 - 08:42 PM

Good answer from DekuTree64. A lot of your bugs sound like features that were checked in without being fully finished. It can be tempting to check something in when it's only 90% complete, because you want to keep moving forward and work on something else, but you'll find (and you HAVE found) that kind of behavior adds a burden on you in the long run. When you work that way you accrue a sort of "code debt," and like all debt, it's gotta be paid off eventually.

 

Slow your pace down a bit. Test as you go. Make sure your features are working before they get submitted. You'll sleep better at night.

 

Also, bugs in your physics system are to be expected. Make no mistake: you have to stay on top of those bugs, but don't get too discouraged if you've got a lot of physics bugs. Physics and collision detection are hard. Every project I've worked on has had a lot of physics related bugs. You just have to grind through those bugs (or use a physics library, but I've never worked on a project with a licensed physics library so I can't offer much advice about that).



#8 Felix Ungman   Members   -  Reputation: 951

Like
9Likes
Like

Posted 11 November 2013 - 01:59 AM

One of my lecturers at the university once told a story about a programming course he'd previously taught to a mixed group of both software and hardware engineers. When reviewing the lab assignments he was astonished that the code submitted by the hardware engineers didn't have any bugs. When he asked them about it, they answered "oh, we didn't realize bugs were allowed".

 

There's only one answer to what's the appropriate number of bugs: ZERO. That's the mindset and the rest is just implementation.


openwar  - the real-time tactical war-game platform


#9 snacktime   Members   -  Reputation: 290

Like
0Likes
Like

Posted 11 November 2013 - 02:19 AM

If you think unit testing takes to long, it's because you haven't used it enough to learn how to do it right.  Once you get yourself setup with the right testing tools and get into the habit of test driven development, you will never go back.  Right now the tests you are writing are probably just all wrong, because it's not something you have really bought into and made an effort to learn to do well.

 

In my experience the secret to good testing is to adhere strongly to the single responsibility principle, and mock/stub anything that's not part of what you are actually testing.  These two things go hand in hand with writing tests that are easy to reason about, and actually valuable.

 

It will take you about twice as long to write your code when using test driven development, as a general rule.  That seems like a lot until you realize just how costly it is to not find bugs until later on.  The feedback time between when you write the code and find a bug is critical and largely determines the total cost of the bug.  The longer it takes to find, the more expensive it is.

 

With C++ it's even worse.  The best C++ developers out there have basically said that humans cannot reason about large C++ codebases, they just get too complex.  So the process of tracking down a bug where you have no testing is painful and time consuming, and it will only get worse as your project grows.

 

Also, since integration testing seems to be pretty big in the game biz, I'll point out this really good talk that also has some good info on TDD in general:

http://www.infoq.com/presentations/integration-tests-scam

 

Chris



#10 tonemgub   Members   -  Reputation: 759

Like
0Likes
Like

Posted 11 November 2013 - 03:52 AM

I don't think it's a good idea to mix business with pleasure - you'll end up ruining one or the other. I do like the Pavlov's dog idea and I know it does work when you just want to get things done, but if you do continue to use the beer-compensation method, I'd be careful not to make it sound like it's mandatory. Maybe some of you would rather have a coke, or a pizza instead. :)

 

If you think some of the developers are getting away too easily because they only fix minor bugs, you could task them with doing code review of other developer's code. i.e.: 3 bugs + 3 code reviews = 1 beer. That way, they would be obliged to understand the harder bugs as well,while the harder working guys will have it easier, because the minor bugs are easy to review.

 

You also mentioned that there's competition... I don't think it's so much a competition - rather sounds to me like you're allowing yourselves to speed through fixing things, just to get the tasks done. This is mostly because nobody likes fixing bugs, especially the developers with more advanced knowledge of the project - they will have the least interest in bugs, because they already know how those bugs could've been avoided in the first place. At least, I don't think a beer after work should have this sort of effect on them. :)

 

That being said, I never really liked the Agile development model (but this is just me), because it forces you to code fast, without thinking about possible bugs you might be introducing. I also don't like how it categorizes and splits work into fixed, periodic categories (or whatever they're called - 2 weeks planning, 2 weeks developing, 2 weeks testing, then repeat). Since you're not documenting much, just adding tasks, you will end up forgetting details of what was planned before you even start developing. Plus, when it comes to bugs, you really cannot plan for them. I always find it easier and faster to fix bugs as I find them, while I'm still involved in the task I'm working on, even if it's not related to my work. Instead, Agile just forces you to add a new task for the bug and move on.



#11 Snovi   Members   -  Reputation: 140

Like
0Likes
Like

Posted 11 November 2013 - 07:32 PM

Obviously, bugs have varying levels of severity.    You don't want show stoppers or other inconsistencies that lower the quality of the game, but if you wait to release a perfect product, you could be waiting a while.   

 

I always put new releases on Google Play first, since it's about 90 minutes to turn around an update, as opposed to a week on iOS.   



#12 jbadams   Senior Staff   -  Reputation: 17736

Like
3Likes
Like

Posted 12 November 2013 - 02:59 AM

You might try applying the concept of user pain to decide whether your game is currently in a sufficiently bug-free state.

 

Is does sound like you're letting too many bugs slip through, so it may be worth looking at your entire development process so you can try to avoid introducing so many bugs to begin with.  As others have mentioned, many of your "bugs" sound more like incomplete features.



#13 Felix Ungman   Members   -  Reputation: 951

Like
2Likes
Like

Posted 12 November 2013 - 04:13 AM


That being said, I never really liked the Agile development model (but this is just me), because it forces you to code fast, without thinking about possible bugs you might be introducing. I also don't like how it categorizes and splits work into fixed, periodic categories (or whatever they're called - 2 weeks planning, 2 weeks developing, 2 weeks testing, then repeat). Since you're not documenting much, just adding tasks, you will end up forgetting details of what was planned before you even start developing. Plus, when it comes to bugs, you really cannot plan for them. I always find it easier and faster to fix bugs as I find them, while I'm still involved in the task I'm working on, even if it's not related to my work. Instead, Agile just forces you to add a new task for the bug and move on.

 

Agile is not not cowboy coding. If you plan to implement a certain feature, you're not really done with the task until the feature works as expected (i.e. no bugs). Fixing bugs as soon as you find them is a good practice, and it doesn't contradict good agile practices. In the contrary, if you are measuring project velocity (story points per sprint or whatever your measure is) then the earlier you fix the bugs the better you'll be able to measure the "true" velocity. Otherwise you are just borrowing time from the future and adding to the hidden technological debt.

 

That said, agile may not be easy to do in practice (depending on the organizational culture), and may not be the best approach for every project. On the other hand, I've not find any negative aspects of it in my experience.


openwar  - the real-time tactical war-game platform


#14 Orymus3   Crossbones+   -  Reputation: 7040

Like
3Likes
Like

Posted 12 November 2013 - 08:32 AM

First: asking whether a number of bugs is appropriate is like saying that it takes "6 months to grieve a relationship". This is not grounded on any baseline. Therefore, its very hard to determine what an acceptable/inacceptable number of bugs really is.

 

From the looks of it though, you are creating too many bugs.  This assessment is based on the fact that the sheer amount of bugs you are faced with impedes your progress. If your velocity is near null, then there is a proble.

 

There are many reasons why this could happen:

 

- Poor code quality

One of the problems in many teams is that they "develop".

Developing new features is fun, but it inevitably changes the system you're working in. If you spend no time refactoring a bit, you'll end up debugging a lot.

If your team refactors as it moves, at a steady pace, old structures will remain relevant, and bugs will be easier to spot.

Refactoring also allows you to optimize your code and eliminate redundancies. If you follow OOP principles, you'll reduce the amount of lines of code, and thus, the amount of lines of code developers will need to read through to find what went wrong.

 

Another common problem is the lack of testing methodologies. Without delving into TTD (though I would advise it if your team is familiar with that approach), its important to test implementation under different circumpstances. It's possible that a new feature / refactor "works" under your current test conditions, but it doesn't mean it keeps status quo with all potential situations.

My personal way to assess this is to create automated test cases with "random" values. This allows me to easily run a number of cases and assess that the system works under different circumpstances. This gives me faith in the code-base, which is important before committing important changes.

 

 

- Poor work ethic

Your team, by now, should have a culture of what it means to put something to "done". Most likely, they consider it's done when they've developed it.

It shouldn't be just that though.

The DOD (definition of done) should encompass any technical debt created by the new feature.

In other words, a new feature is done only when it works fully, and hasn't caused any instabilities elsewhere.

 

Thus, a bunch of the "bugs" youy are stuck with are probably the result of technical debt: a new feature that broke something else.

If you empower your developers with the responsibility of this DOD, it will be easier to implement a reward mechanism.

 

Following your principle, I would say:

1for1: if you get 1 feature to Done (including all technical debt bugs resolved), you get to have a drink.

This allows your developers to have a hollistic vision of development. Instead of "preying on the small bugs", they'll carefully choose one feature, and own it entirely.

 

Also make sure to use the "boyscout rule". If a developer creates a new feature that leverages another part of the code that is considered risky, he must leave the place "cleaner than it was" (thus making a few optimizations to the original code of the faulty class or bit of code).

 

Lastly, to insure your most important features get done, assign an arbitrary value (0-100) to each feature, where 100 is the most important one.

When "drinks" come, insure everyone has fully completed their one feature, then, look at its value.

If it's 100, YOU get to pay their drink. If it's 50, pay half of their drink, etc.

If people want "free drinks", they're likely to start on 100-rated items (heck you could even go with 200-rated items if you want to pay them 2 drinks!).

 

This gives them incentive (drinks) to acquire ownership and responsibility (DOD).

 

Good luck!


Edited by Orymus3, 12 November 2013 - 08:45 AM.


#15 Freya   Members   -  Reputation: 801

Like
1Likes
Like

Posted 12 November 2013 - 05:21 PM

Wow. 
 
I'm amazed by the quality of the answers here at Gamedev (and that we actually get a response!). Thanks!
 
I'd like to clarify some things.
 
Our team is just two persons, who work with the project after the day job. This is kind of a hobby, but a little more serious than a hobby.
 
Sometimes the team morale isn't that good. I guess having ownership is a double edged sword. I don't have anyone except myself looking over how I'm performing, and some days are better than others. Some days I really get into the flow, other days I'm just to distracted / tired / sleepy / fogheaded to do anything good.
 
Most observations above are correct. I like to start new and cool features before I get to completely finish old features. Again, this is on the edge of the sword again. Sometimes having a partially working feature makes it easier to see if a feature is worth keeping or throwing away. And sometimes it's easier to come up with ideas and features when there is something (partially) playable.

Working on a game "Burnt islands".
Dev blog       
"Burnt islands" on Indie DB

#16 Orymus3   Crossbones+   -  Reputation: 7040

Like
4Likes
Like

Posted 12 November 2013 - 05:39 PM


Most observations above are correct. I like to start new and cool features before I get to completely finish old features. Again, this is on the edge of the sword again. Sometimes having a partially working feature makes it easier to see if a feature is worth keeping or throwing away. And sometimes it's easier to come up with ideas and features when there is something (partially) playable.

 

I'm known to live by this motto:

 

5 days dev

1 day refactor

1 day off

 

copy paste for the rest of your life.

It works for me, and brings amazing results. So far, I haven't felt like dropping out of this loop.

 

Specifically, I get no more than 1-3 hours per day to dedicate to this project, but I try to deliver at least an increment of functionality every 'dev' day.

Any 'dev' day I can't achieve this, I commit the next day as 'extra refactor' day, because it shows that my technical debt has increased.

So its not unusual for me to have 2 refactor days in the same week, but this is so that the next week, I get to make 1 increment in each of my 'dev' days.

I think its a good indicator of the health of your codebase too (being able to deliver quality, quickly, without having to debug for several hours).



#17 Squared'D   Members   -  Reputation: 2224

Like
1Likes
Like

Posted 15 November 2013 - 08:01 PM


- The game sometimes crashes hard.

 

This is a very worrisome bug. If the game suddenly crashes at unexpected times, it could mean deep problems that could be very hard to fix, and if you release/sell this game and it crashes on people, your reputation will take a dive. By not fixing it immediately, you may have built code that relies on code that needs to be removed or was poorly designed. I personally try to keep bugs to 0 and if a bug creeps up, I fix it before moving on to anything else. When building new features on buggy code bases, future bug fixes can break working code. I've been in that situation before and found it better to stop adding features and just fix everything first. In the long run, you'll save time.

 

Buggy code can kill motivation. If you remove all the bugs before adding more features, your nicely working program will provide more motivation to finish it. "Hacky" code and endless bug fixing can drive a person insane and kill an otherwise good project.

 

I hope you fix all of your bugs first. Then you'll see how much easier and enjoyable coding your project can be.


Learn all about my current projects and watch some of the game development videos that I've made.

Squared Programming Home

New Personal Journal

 


#18 Pink Horror   Members   -  Reputation: 1116

Like
0Likes
Like

Posted 21 November 2013 - 11:53 AM


Our team is just two persons, who work with the project after the day job. This is kind of a hobby, but a little more serious than a hobby.

 

I understand when a huge mismanaged project with programmers fighting deadlines and bureaucracy and apathy has bugs pile up, but I can't imagine why a two-person hobby team would let a bug database grow out of control. That's not an excuse for more bugs. It's a reason you should have fewer bugs.



#19 Álvaro   Crossbones+   -  Reputation: 12459

Like
3Likes
Like

Posted 21 November 2013 - 12:49 PM

I had a neighbor once that was a musician and didn't have a regular paycheck. Every year he would meet with his tax preparer and have a conversation that went something like:

-"How much do you want to pay in taxes this year?"

-"Let's start with zero and see where we end up..."

 

This thread reminds me of that story. smile.png


Edited by Álvaro, 21 November 2013 - 12:50 PM.





Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS