So... no one estimates? [Was: COCOMO for games?]
Author
Hi,
Recently i've been studying metrics and estimating methods for software projects. You know: Function Points, Web Object Points, COCOMO, KLOC, Cyclomatic complexity, etc... practically everything that helps you messure the size of you project and thus estimating how much [time,MM] will take to develop.
Many of theses could be applied to Game Development pretty much directly (KLOC, Cyclomatic complexity) but others i just don't see them as right options for the kind of development that games require.
So i was wondering, does anybody know any kind of method specially designed for meassuring game development? Estimating schedule, costs and size?
Something like "Game Object Points", or "COCOMO Games".
If you work in the industry, your company/team must have some sort of estimation process right? How do you estimate? Is there a specialist who just uses her experience to give the client/boss a number? Or is there a specified procedural method for analizing de requierements, the targets, the factors to consider that could affect development... etc.
Is your method uniquely developed for your company? Does your previous employee used something similar? Is the method specially for some genre or is it designed for all types of games?
I have looked around but i can't seem to find anything. I'm basically looking for methods that could be applied to "highly interactive, small database, always changing enviroment" type of software.
Any help or hint will be highly apreciated.
Bye!
[Edited by - desdemian on November 2, 2007 1:50:44 PM]
Are you talking metrics with the intent of schedule prediction?
I've worked at places where the engineer who would implement the feature put together the estimate. Tracking their predicted estimates vs actual implementation time, you can get reasonable estimates.
Most features can be decomposed into relatively small tasks which are easy to estimate. Gameplay focused tasks can be much harder, as they often involve significant amounts of iteration with design.
I would argue that this iteration is where most scheduling inaccuracies comes from. The stakeholders/clients often don't know what they want. Or designers change their minds later on.
Metrics looking at 'game design design risk' may be more useful for engineer scheduling.
Metrics such as cyclomatic complexity are occationally used to identify code that needs refactoring or redesign. I don't know if that is what you are looking for though.
I've worked at places where the engineer who would implement the feature put together the estimate. Tracking their predicted estimates vs actual implementation time, you can get reasonable estimates.
Most features can be decomposed into relatively small tasks which are easy to estimate. Gameplay focused tasks can be much harder, as they often involve significant amounts of iteration with design.
I would argue that this iteration is where most scheduling inaccuracies comes from. The stakeholders/clients often don't know what they want. Or designers change their minds later on.
Metrics looking at 'game design design risk' may be more useful for engineer scheduling.
Metrics such as cyclomatic complexity are occationally used to identify code that needs refactoring or redesign. I don't know if that is what you are looking for though.
This is not limited to games, almost no one uses software metrics. I had plenty of opportunity to work with many, many companies doing software engineering at my last position and the only people who really took software metrics seriously were defense contractors. Almost everyone uses some form of estimation but almost no one bases those estimates on software metrics.
Software metrics (in any software engineering effort, not just game design) are useless at best, and in my experience tend to be counterproductive.
If you need schedule estimates, refine your requirements to a level of detail that allows your engineers to work out a rough design. Say they break it down into N subsystems.
For each subsystem, ask the engineers to estimate the time it will take them to implement it. Take their estimate and double it. If your engineers are good designers, that should cover internal feature creep and unexpected integration bugs. If your engineers are not so good designers, quadruple the estimate.
If management/marketing creates a new requirement (external feature creep), do the same exercise to estimate the cost.
Also, since you mentioned it twice in your post, let me point out the Cyclomatic Complexity is the second-most useless software metric (LOC is the first) because it diagnoses the problem far too late. If you have areas of code with very high cyclomatic complexity, you have a bad design, and if you wait until the code is finished before you diagnose it, you're way up the creek without a paddle.
For "highly interactive, small database, always changing enviroment", look into Extreme Programming with a heavy emphasis on Test-Driven development. That style of development is especially well suited for an environment where requirements are constantly changing.
If you need schedule estimates, refine your requirements to a level of detail that allows your engineers to work out a rough design. Say they break it down into N subsystems.
For each subsystem, ask the engineers to estimate the time it will take them to implement it. Take their estimate and double it. If your engineers are good designers, that should cover internal feature creep and unexpected integration bugs. If your engineers are not so good designers, quadruple the estimate.
If management/marketing creates a new requirement (external feature creep), do the same exercise to estimate the cost.
Also, since you mentioned it twice in your post, let me point out the Cyclomatic Complexity is the second-most useless software metric (LOC is the first) because it diagnoses the problem far too late. If you have areas of code with very high cyclomatic complexity, you have a bad design, and if you wait until the code is finished before you diagnose it, you're way up the creek without a paddle.
For "highly interactive, small database, always changing enviroment", look into Extreme Programming with a heavy emphasis on Test-Driven development. That style of development is especially well suited for an environment where requirements are constantly changing.
I don't know if they invented it, but Joel Spolsky's company are using something called "Evidence Based Scheduling". He wrote an article about it: link
Author
Thanks guys, every opinion was very helpful.
I understand that metrics are not very accepted in the software industry, and i haven't had any experience with them to backup or refute that they work. But i think it could be possible to come up with a metric system that works, at least for a especific company, for a specific team. Maybe it shouldn't be a code based metric, perhaps it should be a metric over the design document... for example:
...
Rule 58: If the letters MMO are in the title, estimated_time *= 5;
=)
If anyone's interested, I found a comment about an article here wich pretty much says: "If your game prototype takes 1 to develop, your polished ready-to-ship game will take 12".
It's an intereting ratio, not necesarily true, but interesting.
If anyone else has a comment about "the way things work" (or "don't work") in this area please feel free to share!
Again, thanks.
I understand that metrics are not very accepted in the software industry, and i haven't had any experience with them to backup or refute that they work. But i think it could be possible to come up with a metric system that works, at least for a especific company, for a specific team. Maybe it shouldn't be a code based metric, perhaps it should be a metric over the design document... for example:
...
Rule 58: If the letters MMO are in the title, estimated_time *= 5;
=)
If anyone's interested, I found a comment about an article here wich pretty much says: "If your game prototype takes 1 to develop, your polished ready-to-ship game will take 12".
It's an intereting ratio, not necesarily true, but interesting.
If anyone else has a comment about "the way things work" (or "don't work") in this area please feel free to share!
Again, thanks.
The title of this thread is pretty far off base. Everyone estimates. It is a core development task. You'll notice that even the companies who say "when it's done" know at least a few months ahead of time that it's gonna be done soon. That's not a coincidence. If there isn't an explicit estimate, there's an implicit one. The manager collects or deduces those estimates, figures out what they translate into in terms of an end-of-work date for their piece, and delivers that translated notion to the people who have to work the next level of voodoo on figuring out what the end-of-work/delivery date for the entire project will be. The only tasks that don't get estimated are those that are so critical that there's no point in estimating.
As mentioned above, the key metrics are the estimates of your developers, and the historical data regarding the accuracy of their estimates. The old saw about doubling the initial estimate has, as a proviso, the following clause: Even after you take this law into account, you must still double your estimate. That's why the historical data is necessary, and that's also what FogBugz 6 uses. Its means of processing that data is not the only way, but using that data as a baseline is, fundamentally, the only meaningful way to break work out - it seems that the basic problem is that every software estimation tool breaks down at the level of programming because no two developers will give the exact same numbers for a given set of development tasks. You have to take into account what work is being done, and by whom. And the way to do that is to get estimates from the developers, and to track their ability to meaningfully estimate tasks.
Having gone to the Fogbugz demo here in Kitchener, I can say that I believe theirs is a good way to do the estimation (for a shop that has the historical data on-hand, or is making the effort to collect it), and I would agree with the fundamental philosophy of projection used in their product. They intrinsically correct for poor estimation and estimates that are given in terms of work on the task rather than work days as a whole (difference being 4 hours could be in meetings rather than on the task list), and they get rid of the explicit "dependency" concept on the theory that you can do most of a "dependent" task without having the dependency finalized, which theoretically helps shrink the schedule and reduce the complexity of task assignment.
[Edit]
Lemme give an example from my own working life, which has not been primarily game-centric, but rather enterprise apps-centric.
I was hired simultaneously with 2 other developers to work on 3 separate but related integration points between two major enterprise applications, one of them developed by the shop, and the other developed by a major corporation in the domain. We each took one integration point and began our work. The functional specs existed prior to our coming onboard, so we had basically the same starting ground for each module. We were given 6 weeks to go from the "demo" implementations to full-function implementations. The work proceeded as follows:
1. Developer 1 (not me) declared his work finished within the 6-week deadline.
2. I finished my module within 8 weeks of my initial hire date.
3. Myself and the third developer split the remainder of the work on the final module, but it took us 2 more months to really get it working properly.
4. The QA cycle rolled around. Developer 1's module didn't actually do anything when used in situ. Developer 1 had since been moved to other tasks, so I took over his original module. I tinkered with it, discussed it with QA, discussed it with the analyst, came up with a suggested fix to the functionality, implemented it, and delivered to QA. 3 more months sunk.
5. Massive Layoffs. Developer 2 went away entirely. I took over his module as well.
6. Almost a year later, we rolled out the new prototype to a few clients. It didn't work for most of them, and I spent several very long weekends trying to assist our implementation experts in getting the product to do what it was supposed to for our customers.
7. More shenanigans, further frustrations because our analysts didn't actually know much about the external app to which we were integrating, and finally the project kind-of-died.
8. A year later, using an XML layer written for our web version and some simple off-the-shelf tools, one of our developers wrote an import/export that did the job faster, better, and more reliably in somewhere around 6 months.
I guess the point of all that is this:
For every task, it is going to matter how you approach your work, who is doing the work, and what the work is. Having a series of metrics to assist estimation may help your engineers clarify what the task is, but it will not make your estimates more reliable. You have to be responsible for that all on your own, unfortunately.
[Edited by - liquiddark on December 2, 2007 1:42:25 AM]
As mentioned above, the key metrics are the estimates of your developers, and the historical data regarding the accuracy of their estimates. The old saw about doubling the initial estimate has, as a proviso, the following clause: Even after you take this law into account, you must still double your estimate. That's why the historical data is necessary, and that's also what FogBugz 6 uses. Its means of processing that data is not the only way, but using that data as a baseline is, fundamentally, the only meaningful way to break work out - it seems that the basic problem is that every software estimation tool breaks down at the level of programming because no two developers will give the exact same numbers for a given set of development tasks. You have to take into account what work is being done, and by whom. And the way to do that is to get estimates from the developers, and to track their ability to meaningfully estimate tasks.
Having gone to the Fogbugz demo here in Kitchener, I can say that I believe theirs is a good way to do the estimation (for a shop that has the historical data on-hand, or is making the effort to collect it), and I would agree with the fundamental philosophy of projection used in their product. They intrinsically correct for poor estimation and estimates that are given in terms of work on the task rather than work days as a whole (difference being 4 hours could be in meetings rather than on the task list), and they get rid of the explicit "dependency" concept on the theory that you can do most of a "dependent" task without having the dependency finalized, which theoretically helps shrink the schedule and reduce the complexity of task assignment.
[Edit]
Lemme give an example from my own working life, which has not been primarily game-centric, but rather enterprise apps-centric.
I was hired simultaneously with 2 other developers to work on 3 separate but related integration points between two major enterprise applications, one of them developed by the shop, and the other developed by a major corporation in the domain. We each took one integration point and began our work. The functional specs existed prior to our coming onboard, so we had basically the same starting ground for each module. We were given 6 weeks to go from the "demo" implementations to full-function implementations. The work proceeded as follows:
1. Developer 1 (not me) declared his work finished within the 6-week deadline.
2. I finished my module within 8 weeks of my initial hire date.
3. Myself and the third developer split the remainder of the work on the final module, but it took us 2 more months to really get it working properly.
4. The QA cycle rolled around. Developer 1's module didn't actually do anything when used in situ. Developer 1 had since been moved to other tasks, so I took over his original module. I tinkered with it, discussed it with QA, discussed it with the analyst, came up with a suggested fix to the functionality, implemented it, and delivered to QA. 3 more months sunk.
5. Massive Layoffs. Developer 2 went away entirely. I took over his module as well.
6. Almost a year later, we rolled out the new prototype to a few clients. It didn't work for most of them, and I spent several very long weekends trying to assist our implementation experts in getting the product to do what it was supposed to for our customers.
7. More shenanigans, further frustrations because our analysts didn't actually know much about the external app to which we were integrating, and finally the project kind-of-died.
8. A year later, using an XML layer written for our web version and some simple off-the-shelf tools, one of our developers wrote an import/export that did the job faster, better, and more reliably in somewhere around 6 months.
I guess the point of all that is this:
For every task, it is going to matter how you approach your work, who is doing the work, and what the work is. Having a series of metrics to assist estimation may help your engineers clarify what the task is, but it will not make your estimates more reliable. You have to be responsible for that all on your own, unfortunately.
[Edited by - liquiddark on December 2, 2007 1:42:25 AM]
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement