I realize this question is complex and subjective, but bear with me for a moment.

I firmly believe that video-game software is essentially different from, for example spreadsheet software, as from a user point of view, some games have more similarities with film art than with regular software. However, when it comes to requirements prioritization methods that help developers sort out which feature or idea is most important, there is none that facilitate the creative aspects of video games. Traditional techniques only aid regular software development where focus is on the risk and cost of requirements, and do not take into account the creative aspect of video-game requirements.

To illustrate the problem, think of a requirement (i.e. feature idea) such as the one seen in Super Paper Mario, where Mario can exit the 3D world and enter a 2D world, and vice versa in order to go around obstacles. This is a fairly challenging requirement to implement and probably costly, but definitely crucial to the core/creative vision of the game. Available prioritization methods overlook the creative/entertainment value of this requirement and mainly just estimate how costly and (technologically) risky implementation is. Of course, this is a very simplistic example, but you can imagine a case where there is a pool of great game ideas (functional and non-functional requirements) and not enough resources to realize them all. When it comes to value calculation, academic literature doesn't provide game developers with an adequate way to estimate the value of game requirements.

UPDATE / Clarification: In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and try to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the outside in vs. the inside out paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects.

I try to refine part of the preproduction process by developing a requirements prioritization method tailored to the needs of game development industry. A pivotal element in such a method is the ability to identify and estimate the creative/entertainment value of requirements (as explained above). However, in order to do that, I need to understand how game developers perceive this creative/entertainment value of requirements. In a nutshell, I am seeking answers to the following questions:

Question 1: How would you define the creative/entertainment value of video-game requirements?

Question 2: How would you measure it?

Question 3: Who should measure it?

I would love to see how these issues are perceived by game developers and I would appreciate your take on them here, but if you want to contribute to this research – receiving my eternal gratitude and a proper credit/citation in the research and all the publications that will follow, please fill up my extremely short survey (just 7 questions, 3 of which you already see above):

The extremely short survey which will cement you as my personal hero. Also, you will get a copy of my research once it's finished and the designed solutions (e.i. the Creative Requirements Prioritization method).


UPDATE / Clarification: If my intentions are still confusing, check out my explanation [color="#1D3652"]here.


Research information and trigger

This research is being conducted at Utrecht University, The Netherlands, as part of an Information Science master’s thesis.

Game development is in many ways similar to product software development as developers follow certain software development processes. Following a poor development method (or none at all) could result in longer development times, going over budget and/or delivering buggy products (Bethke, 2003). What makes video games different is the creative game vision that must be shared by the entire team to ensure that the end product is consistent and of good quality. This is especially true for full-blown game titles where from a user point of view, there are more similarities with film art than with any other software. Unfortunately, this curious creative aspect renders many software product management techniques unadoptable by the gaming industry.

I am quite interested in working towards improving the game development process in the preproduction stage, by creating a requirements prioritization method tailored to the specific needs of the game industry and I need your help! The easiest way to contribute to the research is by filling up my short survey (link above). If you find this research interesting, please contact me at a.cherv@gmail.com

Researcher information

My name is Alex Chervenkoff, an avid gamer and quite excited about this research! You can contact me at: a.cherv@gmail.com

Degree in BSc Computer Science, The University of Sheffield, UK.

Currently doing MSc Information Science at Utrecht University, The Netherlands.

Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 in under ten seconds. You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may not consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the third most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the third.

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation whatsoever between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally not quantifiable. If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers you assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone wanted to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that were fun and were original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.

Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 in under ten seconds. You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may not consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the third most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the third.

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation whatsoever between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally not quantifiable. If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers you assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone wanted to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that were fun and were original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.

[font="Arial"]Thank you for your extensive reply ApochPiQ![/font]

[font="Arial"]Rest assured criticism won’t offend me – it can only bring insight to the research at hand. I absolutely understand how subjective the issue of enjoyment is, and your “exercise” did a very good job of illustrating how an individual would fail at providing an objective, reproducible rating.[/font]

[font="Arial"]I have thought of numerous reasons why this couldn’t work, and I have talked to game developers who have also expressed why they think this is a hard task… but giving up never helped solving any challenge. Instead of looking at ways it can’t be done, I decided to focus on how something like this can work. In your answer, you rightly raised a couple of points I had already put some thought and research into:[/font]

[font="Arial"]First, one person cannot assess a piece of art in under 10 sec. However if all the stakeholders involved in making the game express their opinion on a feature (possibly by assessing a prototype, sketches etc.), there will definitely be some benefits from a prioritization point of view. Moreover, many game studios have at their exposal a group of gamers who represent their target group. They could be involved in the prioritization of features as well, and with their statistically more significant numbers than a single vote, can definitely provide a good idea of which feature is desired more than another.[/font]

[font="Arial"]Your second point, the passion and vision of the creator – I absolutely agree with you that a lot of games are pure result of the passion and vision of their creators. I actually want to make sure this vision and the passion that goes with it is translated in the list of ideas that make up the potential game. I believe that without sharing the vision and passion, chances are the end product will be disappointing. However, when overwhelmed with ideas, one needs to take a step back and assess which of these ideas resonate best with the vision of the game, and that task seems impossible to an individual, and this is why any solution that comes out of this research will rely on many stakeholders.[/font]

[font="Arial"]I do not want to degrade creativity by saying it can be quantified and represented by a simple number. Nor do I want to create a tool that “Guarantees the success of your title 100%!” However, I don’t think I have undertaken an ambitious project. I just want to take a closer look at why are so many software product management techniques unpalatable to the gaming industry. It seems it’s the creative element and the entertainment value of game software that is never taken into account in those otherwise useful software development solutions. I believe that by understanding the problem a bit better, I will be able to rework/combine/adapt some of the available requirements prioritization algorithms to better suit the specificities of the domain.[/font]

[font="Arial"]To summarize, I am not trying to devise an objective way of measuring the creative/entertainments value of game ideas. Ultimately, I am trying to lower the risk in the game development process by allowing developers and other stakeholders to adequately estimate the attractiveness of features/ideas during the preproduction stage. I’m sure you will agree this is no bad idea. Is it challenging? Yes. A master thesis needs to be.[/font]

[font="Arial"]I appreciate any feedback, suggestions and criticism, be it ways these ideas could work, or reasons why they won’t. I bet the former is more challenging and I’m sure there are people that not afraid of a challenge. If you want to be part of this research, I again encourage you to check out my 7-question survey (link in the original post above) or if you just want to share your opinion you can do it as an answer here.[/font]

[font="Arial"]Any help is highly appreciated![/font]

What you seek has a history of not being measurable despite many attempts.

There are many scholarly articles that attempt to make such predictive measurements. Malone & Lepper's work on motivation, and various papers by J.P.Gee, B.Dodge, and others, repeatedly attempt to generate such metrics.... Yet further study on every one had those metrics destroyed by counterexample.

You simply cannot measure fun. Even after it is complete, even after you have a finished product, you still cannot measure fun or creativity.

You can measure many other facets that will help in determining the profitability of a game, including comparing it to other existing product. But measurements of 'fun' is not one of them.

I have obviously confused people with my explanation, despite my attempt at the exact opposite. Let’s see if I can clarify my intentions.

In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and trying to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the outside in vs. the inside out paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects. I realize the question title is misleading if you take it without carefully inspecting the kilometric and confusing explanations I provided on my way to explain my research. I chose it because I thought it would spark the curiosity of the readers, which it did, but for all the wrong reasons.

Thank you for your replies and I hope I get more opinions. If you understand my goals by now, I kindly ask you help my research by filling out my very short (7 questions only) survey.

frob, I never said I worked in the industry. As you rightly pointed out, my view is predominantly from the outside of the industry and my assumptions are based on the findings from my initial interviews with developers and also on academic literature, which you so bluntly point out is hardly followed in reality.

Be that as it may, what I don’t understand is why the hostility? I have come to a place where people seek knowledge from people in the industry, to do exactly that. Moreover, my survey’s questions are not presupposing an answer – they are open-ended and you can easily answer “This question is irrelevant in the real world because…”, which would help my research. Instead, you choose to smugly parade your supposed superior understanding of what-actually-happens by putting effort in devising “amusing” stories about the planet Abrigreene. What is so enraging and wrong about asking how are things done?

That type of thesis would best be approached by actually having worked in both industries. If not a direct employment job, then as a consulting role so that you actually know and understand what you are talking about.

I absolutely agree that would be the best way to do this research. However, to my knowledge no one has done it. I have tried to get in a larger game studio (as a researching intern) where complex projects are developed, but sadly, the fear of leaking inside information to the competition renders such a practice virtually impossible over here (the Netherlands). The second best thing I could do is interview game developers and engage online discussions on the topic, which I am trying to do here.

I hope others will understand the limitations of my research resources and help me out here. Again, any help is highly appreciated!

I think part of the problem is that you came in making a lot of assertions and statements about how you think things are done; it was not at all clear to me that you were asking if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.