Sign in to follow this  
chervenkoff

How can I measure the "creative/entertainment value" of video-game requirements?

Recommended Posts

I realize this question is complex and subjective, but bear with me for a moment.

I firmly believe that video-game software is essentially different from, for example spreadsheet software, as from a user point of view, some games have more similarities with film art than with regular software. However, when it comes to requirements prioritization methods that help developers sort out which feature or idea is most important, there is none that facilitate the [b]creative aspects[/b] of video games. Traditional techniques only aid regular software development where focus is on the risk and cost of requirements, and do not take into account the creative aspect of video-game requirements.

To illustrate the problem, think of a requirement (i.e. feature idea) such as the one seen in Super Paper Mario, where [url="http://upload.wikimedia.org/wikipedia/en/e/e5/SPM_3D.jpg"]Mario can exit the 3D world and enter a 2D world[/url], and vice versa in order to go around obstacles. This is a fairly challenging requirement to implement and probably costly, but definitely crucial to the core/creative vision of the game. Available prioritization methods overlook the [b]creative/entertainment value[/b] of this requirement and mainly just estimate how costly and (technologically) risky implementation is. Of course, this is a very simplistic example, but you can imagine a case where there is a pool of great game ideas (functional and non-functional requirements) and not enough resources to realize them all. When it comes to value calculation, academic literature doesn't provide game developers with an adequate way to estimate the value of game requirements.

[b]UPDATE / Clarification:[/b] In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and try to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the[b] outside in[/b] vs. the [b]inside out[/b] paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision.[b] By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.[/b]

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects.

I try to refine part of the preproduction process by developing a requirements prioritization method tailored to the needs of game development industry. A pivotal element in such a method is the ability to identify and estimate the creative/entertainment value of requirements (as explained above). However, in order to do that, I need to understand how game developers perceive this creative/entertainment value of requirements. In a nutshell, [b]I am seeking answers to the following questions[/b]:

Question 1: [b]How would you define the creative/entertainment value of video-game requirements?[/b]

Question 2: [b]How would you measure it?[/b]

Question 3: [b]Who should measure it?[/b]

I would love to see how these issues are perceived by game developers and I would appreciate your take on them here, but if you want to contribute to this research – receiving my eternal gratitude and a proper credit/citation in the research and all the publications that will follow, please fill up my extremely short survey (just 7 questions, 3 of which you already see above):

[url="https://spreadsheets.google.com/spreadsheet/viewform?formkey=dFJfVUIwb2dKSkNWYnZfcTFnakZTcFE6MQ"]The extremely short survey[/url] which will cement you as my personal hero. Also, you will get a copy of my research once it's finished and the designed solutions (e.i. the Creative Requirements Prioritization method).


[b]UPDATE / Clarification:[/b] If my intentions are still confusing, check out my explanation [color="#1D3652"][size="2"][url="http://www.gamedev.net/topic/608428-how-can-i-measure-the-creativeentertainment-value-of-video-game-requirements/page__view__findpost__p__4852014"]here[/url].[/size][/color]
[b]
[/b]
[b]Research information and trigger[/b]

This research is being conducted at Utrecht University, The Netherlands, as part of an Information Science master’s thesis.

Game development is in many ways similar to product software development as developers follow certain software development processes. Following a poor development method (or none at all) could result in longer development times, going over budget and/or delivering buggy products (Bethke, 2003). What makes video games different is the creative game vision that must be shared by the entire team to ensure that the end product is consistent and of good quality. This is especially true for full-blown game titles where from a user point of view, there are more similarities with film art than with any other software. Unfortunately, this curious creative aspect renders many software product management techniques unadoptable by the gaming industry.

I am quite interested in working towards improving the game development process in the preproduction stage, by creating a requirements prioritization method tailored to the specific needs of the game industry and [b]I need your help![/b] The easiest way to contribute to the research is by filling up my short survey (link above). If you find this research interesting, please contact me at a.cherv@gmail.com

[b]Researcher information[/b]

My name is Alex Chervenkoff, an avid gamer and quite excited about this research! You can contact me at: a.cherv@gmail.com

Degree in BSc Computer Science, The University of Sheffield, UK.

Currently doing MSc Information Science at Utrecht University, The Netherlands.

Share this post


Link to post
Share on other sites
Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 [i]in under ten seconds.[/i] You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may [b]not[/b] consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the [i]third[/i] most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the [i]third[/i].

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation [i]whatsoever[/i] between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally [i]not quantifiable.[/i] If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers [i]you [/i]assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone [i]wanted[/i] to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that [i]were[/i] fun and [i]were[/i] original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1313378470' post='4849225']
Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 [i]in under ten seconds.[/i] You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may [b]not[/b] consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the [i]third[/i] most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the [i]third[/i].

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation [i]whatsoever[/i] between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally [i]not quantifiable.[/i] If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers [i]you [/i]assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone [i]wanted[/i] to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that [i]were[/i] fun and [i]were[/i] original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.
[/quote]

[font="Arial"]Thank you for your extensive reply ApochPiQ![/font]

[font="Arial"]Rest assured criticism won’t offend me – it can only bring insight to the research at hand. I absolutely understand how subjective the issue of enjoyment is, and your “exercise” did a very good job of illustrating how an individual would fail at providing an objective, reproducible rating.[/font]

[font="Arial"]I have thought of numerous reasons why this couldn’t work, and I have talked to game developers who have also expressed why they think this is a hard task… but giving up never helped solving any challenge. Instead of looking at ways it can’t be done, I decided to focus on how something like this can work. In your answer, you rightly raised a couple of points I had already put some thought and research into:[/font]

[font="Arial"]First, one person cannot assess a piece of art in under 10 sec. However if all the stakeholders involved in making the game express their opinion on a feature (possibly by assessing a prototype, sketches etc.), there will definitely be some benefits from a prioritization point of view. Moreover, many game studios have at their exposal a group of gamers who represent their target group. They could be involved in the prioritization of features as well, and with their statistically more significant numbers than a single vote, can definitely provide a good idea of which feature is desired more than another.[/font]

[font="Arial"]Your second point, the passion and vision of the creator – I absolutely agree with you that a lot of games are pure result of the passion and vision of their creators. I actually want to make sure this vision and the passion that goes with it is translated in the list of ideas that make up the potential game. I believe that without sharing the vision and passion, chances are the end product will be disappointing. However, when overwhelmed with ideas, one needs to take a step back and assess which of these ideas resonate best with the vision of the game, and that task seems impossible to an individual, and this is why any solution that comes out of this research will rely on many stakeholders.[/font]

[font="Arial"]I do not want to degrade creativity by saying it can be quantified and represented by a simple number. Nor do I want to create a tool that “Guarantees the success of your title 100%!” However, I don’t think I have undertaken an ambitious project. I just want to take a closer look at why are so many software product management techniques unpalatable to the gaming industry. It seems it’s the creative element and the entertainment value of game software that is never taken into account in those otherwise useful software development solutions. I believe that by understanding the problem a bit better, I will be able to rework/combine/adapt some of the available requirements prioritization algorithms to better suit the specificities of the domain.[/font]

[font="Arial"]To summarize, I am not trying to devise an objective way of measuring the creative/entertainments value of game ideas. Ultimately, I am trying to lower the risk in the game development process by allowing developers and other stakeholders to adequately estimate the attractiveness of features/ideas during the preproduction stage. I’m sure you will agree this is no bad idea. Is it challenging? Yes. A master thesis needs to be.[/font]

[font="Arial"]I appreciate any feedback, suggestions and criticism, be it ways these ideas could work, or reasons why they won’t. I bet the former is more challenging and I’m sure there are people that not afraid of a challenge. If you want to be part of this research, I again encourage you to check out my 7-question survey (link in the original post above) or if you just want to share your opinion you can do it as an answer here.[/font]

[font="Arial"]Any help is highly appreciated![/font]

Share this post


Link to post
Share on other sites
[quote name='chervenkoff' timestamp='1313415683' post='4849374']
[font="Arial"]I do not want to degrade creativity by saying it can be quantified and represented by a simple number. Nor do I want to create a tool that “Guarantees the success of your title 100%!” However, I don’t think I have undertaken an ambitious project. ... It seems it’s the creative element and the entertainment value of game software that is never taken into account in those otherwise useful software development solutions. I believe that by understanding the problem a bit better, I will be able to rework/combine/adapt some of the available requirements prioritization algorithms to better suit the specificities of the domain.[/quote][/font]
[font="Arial"]In other words, you admit it cannot be quantified, and then say you want to quantify it.
[/font]

[quote][font="Arial"]To summarize, I am not trying to devise an objective way of measuring the creative/entertainments value of game ideas. Ultimately, I am trying to lower the risk in the game development process by allowing developers and other stakeholders to adequately estimate the attractiveness of features/ideas during the preproduction stage. I’m sure you will agree this is no bad idea. Is it challenging? Yes. A master thesis needs to be.[/quote][/font]
[font="Arial"]Again, you say you are not trying to quantify it. Yet your next statement is that you want a way to estimate (or quantify) it.
[/font]


What you seek [url="http://adage.com/article/guest-columnists/viewpoint-metrics-killing-creativity-advertising/142600/"]has[/url] [url="http://prairieskygroup.com/tag/thought-leadership/"]a history[/url] [url="http://www.jerrythepunkrat.com/2010/03/can-you-really-quantify-creativity.html"]of not[/url] [url="http://gameful.org/groups/gameful-challenge-1-awe-inspiring-improvement-aka-make-lord-kelvin-proud/forum/topic/measuring-creative-connections/"]being[/url] [url="http://www.sas.com/resources/whitepaper/wp_17098.pdf"]measurable[/url] [url="http://blog.thinkforachange.com/2010/03/21/how-do-you-measure-an-idea.aspx"]despite[/url] [url="http://books.google.com/books?id=oyAwg8qSyJYC&pg=PA97&lpg=PA97&dq=creativity+unmeasurable&source=bl&ots=qDSEfPFPHO&sig=L0R67uOkfQDxKbjWryHUe_MMMDk&hl=en&ei=JE5JTo1wp8qIArTYpf8B&sa=X&oi=book_result&ct=result&resnum=9&sqi=2&ved=0CEoQ6AEwCA#v=onepage&q=creativity%20unmeasurable&f=false"]many[/url] [url="http://www.jhrobbins.com/Measuring%20the%20Unmeasurable.PDF"]attempts[/url].

There are [url="http://scholar.google.com/scholar?hl=en&q=measuring+fun"]many scholarly articles[/url] that attempt to make such predictive measurements. Malone & Lepper's work on motivation, and various papers by J.P.Gee, B.Dodge, and others, repeatedly attempt to generate such metrics.... Yet further study on every one had those metrics destroyed by counterexample.

You simply cannot measure fun. Even after it is complete, even after you have a finished product, you still cannot measure fun or creativity.

You can measure many other facets that will help in determining the profitability of a game, including comparing it to other existing product. But measurements of 'fun' is not one of them.

Share this post


Link to post
Share on other sites
[quote name='frob' timestamp='1313427651' post='4849453']
What you seek [url="http://adage.com/article/guest-columnists/viewpoint-metrics-killing-creativity-advertising/142600/"]has[/url] [url="http://prairieskygroup.com/tag/thought-leadership/"]a history[/url] [url="http://www.jerrythepunkrat.com/2010/03/can-you-really-quantify-creativity.html"]of not[/url] [url="http://gameful.org/groups/gameful-challenge-1-awe-inspiring-improvement-aka-make-lord-kelvin-proud/forum/topic/measuring-creative-connections/"]being[/url] [url="http://www.sas.com/resources/whitepaper/wp_17098.pdf"]measurable[/url] [url="http://blog.thinkforachange.com/2010/03/21/how-do-you-measure-an-idea.aspx"]despite[/url] [url="http://books.google.com/books?id=oyAwg8qSyJYC&pg=PA97&lpg=PA97&dq=creativity+unmeasurable&source=bl&ots=qDSEfPFPHO&sig=L0R67uOkfQDxKbjWryHUe_MMMDk&hl=en&ei=JE5JTo1wp8qIArTYpf8B&sa=X&oi=book_result&ct=result&resnum=9&sqi=2&ved=0CEoQ6AEwCA#v=onepage&q=creativity%20unmeasurable&f=false"]many[/url] [url="http://www.jhrobbins.com/Measuring%20the%20Unmeasurable.PDF"]attempts[/url].

There are [url="http://scholar.google.com/scholar?hl=en&q=measuring+fun"]many scholarly articles[/url] that attempt to make such predictive measurements. Malone & Lepper's work on motivation, and various papers by J.P.Gee, B.Dodge, and others, repeatedly attempt to generate such metrics.... Yet further study on every one had those metrics destroyed by counterexample.

You simply cannot measure fun. Even after it is complete, even after you have a finished product, you still cannot measure fun or creativity.

You can measure many other facets that will help in determining the profitability of a game, including comparing it to other existing product. But measurements of 'fun' is not one of them.
[/quote]

I have obviously confused people with my explanation, despite my attempt at the exact opposite. Let’s see if I can clarify my intentions.

In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and trying to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the [b]outside in[/b] vs. the [b]inside out[/b] paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects. I realize the question title is misleading if you take it without carefully inspecting the kilometric and confusing explanations I provided on my way to explain my research. I chose it because I thought it would spark the curiosity of the readers, which it did, but for all the wrong reasons.

Thank you for your replies and I hope I get more opinions. If you understand my goals by now, I kindly ask you help my research by filling out my very short (7 questions only) survey.

Share this post


Link to post
Share on other sites
[quote name='chervenkoff' timestamp='1313434103' post='4849491']
With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the [b]outside in[/b] vs. the [b]inside out[/b] paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.
[/quote]
I want to say: WTF?!


I think perhaps you should spend some time actually in the industry before trying to write up what the industry does.

My suspicion is that your only exposure is what you have seen on the outside, while asking questions from within your academic studies.


1> Have you actually worked at a game studio designing games from conception to completion? Your questions clearly announce "No"! It seems clear you simply do not understand the process. It is complex and varied, and [b]never [/b]has the steps you list. Saying again: [b]Your guesses about the game industry design process are fundamentally invalid.[/b]

2> Have you actually worked at one of the "other industries" you mention in such a designing role? I want to know what industries specifically you are comparing with. A few major companies have published their processes, but what they write about vs what it means when sitting in a cubicle are entirely different. It is complex and varied, and never works as the textbooks claim.

That type of thesis would best be approached by [i]actually having worked in both industries[/i]. If not a direct employment job, then as a consulting role so that you actually know and understand what you are talking about.



Your questions, and your survey (as far as I cared to enter it, you ask questions that I don't feel comfortable answering because they are utter nonsense) show that you have no idea what it actually takes to make a game in a professional environment. You might as well include the process of making grilled parepuls while on the planet Abrigreene as it has the same level of experience.

Share this post


Link to post
Share on other sites
Choice is a good metric for entertainment value. Players get bored when theres nothing to do and it doesnt take much to play the game, but when there are interesting choices it becomes fun. Players try to maximize some value based on their choices.

3D Mario offers a new dimension of choice. The rewards of the choices are also mote engrossing.

Share this post


Link to post
Share on other sites
[quote]I want to say: WTF?!


I think perhaps you should spend some time actually in the industry before trying to write up what the industry does.[/quote]
You haven't dealt with academia much, have you?



Here is about the only metric that anyone will pay any attention to: [url="http://www.eurogamer.net/articles/insomniac-60fps-no-more"]sales figures[/url].

In short - does it sell more copies? Then it's good. If it doesn't, then it's not.

For [url="http://grattisfaction.com/2010/01/how-zynga-does-customer-development-minimum-viable-product/"]service centric development[/url], an iterative process is used. Measure everything, iterate fast, keep what improves conversion, kill everything else fast.

[quote]What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision.[/quote]
Most developers today would cry "waterfall". And in many ways, it is an obsolete practice in time of infinite computing power.

Process for just about everything looks like this:
- launch yesterday
- gather metrics on everything, a gigabyte per day is good, terabyte is better
- analyze what resulted in most conversions (aka revenue), identify funnels (steps at which the conversions stop), fix and improve
- go to beginning

This should be done several times a day, preferably more.

[quote] In video-game software, it is the creative vision that drives the requirements.[/quote]
No, sorry.

Sales figures. Nothing else drives the requirements. The handful of very rare projects that succeed on "creative vision" (aka, perfect storm of luck, viral marketing and viable monetization) are flukes.


Regarding the intention - it's in the right direction. But the algorithm used has been put into practice cca. 1998 by a small company called Google. Since then, with increasingly cheap computing power, the methodology has been adopted by every single tech startup and is even applied by VC and angel investors at level of companies themselves.

Short release cycle to actual market, measure, improve vs. kill, repeat. The faster, the better.

Welcome to the future, where engineers or developers are irrelevant to success.

Share this post


Link to post
Share on other sites
frob, I never said I worked in the industry. As you rightly pointed out, my view is predominantly from the outside of the industry and my assumptions are based on the findings from my initial interviews with developers and also on academic literature, which you so bluntly point out is hardly followed in reality.

Be that as it may, what I don’t understand is why the hostility? I have come to a place where people seek knowledge from people in the industry, to do exactly that. Moreover, my survey’s questions are not presupposing an answer – they are open-ended and you can easily answer “This question is irrelevant in the real world because…”, which would help my research. Instead, you choose to smugly parade your supposed superior understanding of what-actually-happens by putting effort in devising “amusing” stories about the planet Abrigreene. What is so enraging and wrong about asking how are things done?

[quote name='frob' timestamp='1313427651' post='4849453'] That type of thesis would best be approached by actually having worked in both industries. If not a direct employment job, then as a consulting role so that you actually know and understand what you are talking about. [/quote]

I absolutely agree that would be the best way to do this research. However, to my knowledge no one has done it. I have tried to get in a larger game studio (as a researching intern) where complex projects are developed, but sadly, the fear of leaking inside information to the competition renders such a practice virtually impossible over here (the Netherlands). The second best thing I could do is interview game developers and engage online discussions on the topic, which I am trying to do here.

I hope others will understand the limitations of my research resources and help me out here. Again, any help is highly appreciated!

Share this post


Link to post
Share on other sites
I think part of the problem is that you came in making a lot of assertions and statements about how [i]you think[/i] things are done; it was not at all clear to me that you were [i]asking[/i] if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.

Share this post


Link to post
Share on other sites
[quote name='Antheus' timestamp='1313446694' post='4849595']
Here is about the only metric that anyone will pay any attention to: [url="http://www.eurogamer.net/articles/insomniac-60fps-no-more"]sales figures[/url].

In short - does it sell more copies? Then it's good. If it doesn't, then it's not.

For [url="http://grattisfaction.com/2010/01/how-zynga-does-customer-development-minimum-viable-product/"]service centric development[/url], an iterative process is used. Measure everything, iterate fast, keep what improves conversion, kill everything else fast.

[quote]What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision.[/quote]
Most developers today would cry "waterfall". And in many ways, it is an obsolete practice in time of infinite computing power.

Process for just about everything looks like this:
- launch yesterday
- gather metrics on everything, a gigabyte per day is good, terabyte is better
- analyze what resulted in most conversions (aka revenue), identify funnels (steps at which the conversions stop), fix and improve
- go to beginning

This should be done several times a day, preferably more.

[quote] In video-game software, it is the creative vision that drives the requirements.[/quote]
No, sorry.

Sales figures. Nothing else drives the requirements. The handful of very rare projects that succeed on "creative vision" (aka, perfect storm of luck, viral marketing and viable monetization) are flukes.


Regarding the intention - it's in the right direction. But the algorithm used has been put into practice cca. 1998 by a small company called Google. Since then, with increasingly cheap computing power, the methodology has been adopted by every single tech startup and is even applied by VC and angel investors at level of companies themselves.

Short release cycle to actual market, measure, improve vs. kill, repeat. The faster, the better.

Welcome to the future, where engineers or developers are irrelevant to success.
[/quote]

Thanks for the reply Antheus!

I have been hearing this view on the matter by quite some developers. The focus seems to be entirely on what would sell best. However, I still wonder what happens when a new, original idea for a game is presented. Is there a methodical way to deal with a pile of ideas that comprise this game? What would the the process of picking the most important/core requirements be?

The "ghetto testing" you referred to is also quite an interesting method. Excuse my ignorance, but I am not sure which algorithm/methodology you refer to when talking about Google.

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1313499978' post='4849828']
I think part of the problem is that you came in making a lot of assertions and statements about how [i]you think[/i] things are done; it was not at all clear to me that you were [i]asking[/i] if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.
[/quote]

I didn't do a great job at presenting my problem. The statements were formulated based on my limited exposure to industry expertise from the few interviews I had with game developers and academic literature on both game development and regular software development. It has become clear that the general best practices for product software management do not resonate well with game development.

I would love to improve the post (and my understanding of the issues discussed) if anyone points out wrong assumptions/statements and suggests what the real-world case is. Your first reply helped me understand I have not presented my questions unambiguously. I have edited the post since and I hope it makes sense now and my research seems less ambitious (e.i. than trying to quantify creativity) than it originally did.

Share this post


Link to post
Share on other sites
[quote name='chervenkoff' timestamp='1313505278' post='4849867']
[quote name='ApochPiQ' timestamp='1313499978' post='4849828']
I think part of the problem is that you came in making a lot of assertions and statements about how [i]you think[/i] things are done; it was not at all clear to me that you were [i]asking[/i] if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.
[/quote]

I didn't do a great job at presenting my problem. The statements were formulated based on my limited exposure to industry expertise from the few interviews I had with game developers and academic literature on both game development and regular software development. It has become clear that the general best practices for product software management do not resonate well with game development.

I would love to improve the post (and my understanding of the issues discussed) if anyone points out wrong assumptions/statements and suggests what the real-world case is. Your first reply helped me understand I have not presented my questions unambiguously. I have edited the post since and I hope it makes sense now and my research seems less ambitious (e.i. than trying to quantify creativity) than it originally did.
[/quote]
It isn't that they don't resonate, it is that they generally don't apply. The are invalid.

First, I really do understand your background in academics. I went to graduate school too, and did the whole thesis thing. Of course, I did it in a technical field on algorithms and based my research as an extension to various SIGGRAPH and InfoVis papers, so it required new mathematical research, not business research.

Over the years I've worked in game development, and in business backend software, in embedded hardware development, and in various presentation industries (broadcast tv software, broadcast TV software, etc.)

In business backend software where I was one of the leads, we did not follow the practices you originally stated as fact. We were told by the company owner what the business needed, such as integration to SalesForce CRM system. We studied out the differences between our current workflow and the SalesForce workflow. We spent months writing ETL jobs, developing a plan to phase over the various other workers, moving our data over, then retiring our older servers. Similarly for other projects we were never asked to prioritize features. They were always created by business need and scheduled based on how they affect profit.

In hardware the software need to calibrate and display in real-time the hardware sensor data. There was no major prioritization of requirements. If the sensor was able to detect it, we needed to show it. If the sensor could be calibrated in any way, we needed to expose it. Again, this was not based at all in what you stated as fact for project management.

In the presentation software, in every case we were asked by the executives to write specific things, such as a PowerPoint plugin to display in-meeting poll questions and results, or to display traffic data on broadcast TV, or to render other data, and so on. Everything was sorted based on funding and ad sales. All that mattered were how many eyeballs were watching the screen and maximizing that number to get commercials. In no case did we follow those business practices that you stated as fact.


For all the business software in all industries I've been in, the driving forces were business need and financial statements. There is very little involved in "requirements prioritization" at the project level. They tended to be "We are falling behind and need this immediately to stay competitive" as specified by the highest level executives, or "We already made the decision based on profitability, now you need to make it work."



Similarly in games development was everything based on money. Design is based on estimated sales figures. Production is based on estimated sales figures. Developers base everything on the budget they are given. Marketing is based on estimated sales figures.

For games software, there are pitches created by designers and passed to studio executives. The executives review it based on profitability of comparable products. Then a demo is made. The executives review it again, and base it on the profitability of comparable products, legality, marketability, and so on. If it seems like a profitable idea, the product enters full design, reviewed periodically for profitability vs risk, hopefully not cancelled, and goes through the rest of production and post-production and marketing and eventual sales cycle, all based on profitability. During implementation, we follow scrum practices where features are implemented in order of what is required to build the game, not based on the values you cited as being important. Instead it was based on features like spawning objects, inventory systems, creation of core objects, core UI, core rendering, core networking. Later on were the creation of clones of objects, 'nice to have' effects, 'nice to have' networking features, and so on. The first round is based on what is required to make a game. The second round was filling up the content until time ran out. Those were sorted based on complexity and risk, not so much on how creative the item was.

Nowhere in the corporate process of making games is the "creative/entertainment value" a consideration. During production some managers will use it to help prioritize the lesser object creation order, but again that is a minor side consideration fairly late in the cycle, and only used as a minor detail.

Share this post


Link to post
Share on other sites
Frob, you don't seem to be taking into account the fact that a game design is not monolithic. A game designer may present a design to an executive or group of executives, but there is pretty much 0% chance that the final produced game will look identical to the design. And the first batch of changes is going to be an evaluation of whether anything in the design is going to be more expensive than it's worth to implement. Making a movie is really a better comparison to the process of making a game than making productivity software is. (Or at least you should be comparing to a BIG piece of productivity software with lots of features like Photoshop or Microsoft Office.) The original screenplay for a movie may be purchased by a studio, but will then have many changes made to it by several different layers of people in the process of production, down to actors ad-libbing their lines and whole scenes possibly ending up on the cutting room floor.

Yes the overall goal is to make the most money and secondarily to win free advertising via having pleased viewers recommend the movie to their acquaintances, as well as the possibility of winning a major award which generates a lot of free publicity and further sales. But everyone involved, from the writer and director to the actors, makeup and costume artists, stunt men, etc. has an idea of what makes a good movie, and they are trying to make a good movie. Audiences may pay for a movie based on high concept and advertising alone, but if the movie's entertainment value doesn't satisfy that audience it will negatively impact the future earning potential of the IP involved as well as the studio, director, and any actors the audience identifies as being a part of the problem. Executives may try to always look at the bottom line, but when you have a staff that includes artists and entertainers, those people's decision-making processes are going to be motivated by their urge to create art and be entertaining.

Share this post


Link to post
Share on other sites
frob,

The solutions, and generally the academic literature we study in Software Product Management (SPM) at Utrecht University are rarely present in regular software businesses, but I have personally observed numerous software product Dutch companies in the process of implementing SPM practices. Six months ago, I and a team of colleagues conducted several interviews as part of an SPM research with a product manager in a rapidly growing software company (their main product was a CRM system), who was trying to bring up the maturity level of his (40+ FTEs) organization. I was surprised to see that the problems he and his company had, have solutions in SPM literature, but he was totally unaware of them. Solutions that have empirical evidence for helping success rate of projects in terms of schedule predictability and quality. Similarly, the other groups in the class conducting SPM research at other software companies reported the same situation.

What I am trying to say is, I am not surprised any more that the solutions in academic literature are not more widely applied. Whether it's that they are relatively new; it's too expensive for companies to reorganize and implement them; companies don't trust the return of investment; or they just don't know about them (as was the case with the 8 companies researched in our SPM class), I honestly don't know. The truth is, there is a lot of academic literature on valid problems that is rarely paid attention to.

It is true that the research I am conducting deals with concepts (i.e. the infamous "creative/entertainment value") that are not in use, and I am very well aware of this. I also realize it might seem too idealistic and even naive to focus on the creative vision while developing a game and use it as one of the factors for requirements prioritization, but as a scientist, I am not afraid of debunking an idea. In fact, I try not to favor any side and just see what the expert would say when presented with the idea.

What I am doing with my research will most probably not apply for smaller projects (e.g. casual games). Whatever prioritization method I come up, I think will probably focus on a fraction of a project. It is a relatively small improvement I am aiming at, but that's the way academia works - in very small steps. In the process I hope I will get a good idea of the current industry practices, as I must take that into account.

The reason I got excited about it in the first place was due to the fact that games are rarely viewed as tool for dealing with problems. They are creative, they are entertaining, some might go to the length of arguing they are art. The best examples have similarities with film art. A logical continuation of thought would suggest that the creative/core vision should play [b]a role[/b] in the development process, as it is the case with any form of art. Part of my research deals with the "[b]What if[/b]" the creative vision is taken into account, and "[b]How to[/b]" actually do that. The What if and How to are reflected in my survey, and the reason I asked game developers to try and answer these questions - so I can see what game developers think about it, and how they would approach the problem.

I am not arguing this is what needs to be done, nor what is actually done in the industry. I hope it is clearer why I am asking those silly questions now.

P.S. I appreciate the tone of your last reply, and the explanations you provide.

Share this post


Link to post
Share on other sites
[quote name='sunandshadow' timestamp='1313520347' post='4849977']
Frob, you don't seem to be taking into account the fact that a game design is not monolithic. A game designer may present a design to an executive or group of executives, but there is pretty much 0% chance that the final produced game will look identical to the design. And the first batch of changes is going to be an evaluation of whether anything in the design is going to be more expensive than it's worth to implement. Making a movie is really a better comparison to the process of making a game than making productivity software is. (Or at least you should be comparing to a BIG piece of productivity software with lots of features like Photoshop or Microsoft Office.) The original screenplay for a movie may be purchased by a studio, but will then have many changes made to it by several different layers of people in the process of production, down to actors ad-libbing their lines and whole scenes possibly ending up on the cutting room floor.

Yes the overall goal is to make the most money and secondarily to win free advertising via having pleased viewers recommend the movie to their acquaintances, as well as the possibility of winning a major award which generates a lot of free publicity and further sales. But everyone involved, from the writer and director to the actors, makeup and costume artists, stunt men, etc. has an idea of what makes a good movie, and they are trying to make a good movie. Audiences may pay for a movie based on high concept and advertising alone, but if the movie's entertainment value doesn't satisfy that audience it will negatively impact the future earning potential of the IP involved as well as the studio, director, and any actors the audience identifies as being a part of the problem. Executives may try to always look at the bottom line, but when you have a staff that includes artists and entertainers, those people's decision-making processes are going to be motivated by their urge to create art and be entertaining.
[/quote]

Exactly my point! Thank you :) I was starting to think I am going crazy.

Share this post


Link to post
Share on other sites
I'm glad if I said anything helpful. :)

I also wanted to comment on ApochPiQ's art gallery exercise. I think this is a great exercise, except I think the results and conclusions I would draw would be totally different. The field of demographics exists for a reason - it is to some extant predictable what people will like and not like. Me personally I could tell you with maybe 80% accuracy what art in a gallery I am going to like and not like [b]before I go in the gallery[/b]. Maybe I have more predictable taste than most people, or maybe because I come from an art background I've spent a lot of time studying my own preferences and I have enough experience to make a guess at what kinds of things are going to be in an art gallery. But on the other hand I could make the same sort of prediction for, say, my mother with at least 50% accuracy.

Here are some of my predictions of what I would and wouldn't like:
- dislike any piece portraying a person or animal looking like they are in pain or terror, or dead
- neutral or mild dislike toward any piece which is abstract; more strong dislike for things that look like violent random scratches or clusters, more neutral for things which look like orderly geometric shapes or fractals.
- neutral to most photographs, with exceptions for the two points below this one:
- strongly like any piece portraying pretty flowers, butterflies, dragonflies, horses, deer, dogs, cats, birds, fish, all sorts of animals
- mildly like any piece portraying a person with a positive facial expression in an impressive costume (historical, futuristic, specialized use like a ballet costume or suit of armor, etc.
- strongly like pieces which use a neon/bold color palette
- strongly like pieces which use a jeweltone/pearl/pastel/metallic color palette
- mildly like pieces which use the sunlight color palette involving warm browns, soft blues, and crisp greens
- mildly like black and white, sepia tone, and partially colorized black and white pieces
- dislike pieces using the formal color palette involving tan, navy blue, gray-green, and power red

Share this post


Link to post
Share on other sites
The important part of the art exercise has nothing to do with what kind of art you like, or whether or not you can personally assign scores to something.

The point was in the [i]comparison[/i] to other people's opinions, and even the comparison between your immediate reaction to a piece, your short-term memory of that piece, and your lasting impressions of the same piece. If you personally don't think that people would have varying scores at those three time points, I would argue that you have either uncommonly flat taste, or uncommonly acute perception of artistic merit. I would also argue that you cannot generalize from self in this instance; that was, again, part of the point of the exercise. Just because you will react to art in one way does not in any way suggest that you can conclude what any arbitrary other person would think of the same exact work.

For instance, I find pastels to be irritating, and prefer grayscale photography and muted colors over anything with highly vibrant blues or greens. The point of the art gallery exercise is not that I can tell you those things a priori of walking into the gallery; the point is that (1) I may find a piece that does not fit my preconceived ideas of what I "like" and yet still appeals to me; (2) I may change my mind about a piece upon reflection; and (3) I'm going to have wildly different numbers than you do.



As for software practices: there are two kinds of software companies. One kind is rigidly attached to process, often informed by academic research, and spends tremendous amounts of money and time on things like certifications, process compliance, and so on.

The other kind is not a hellhole.


(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the [i]people[/i]. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1313606711' post='4850428']
(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the [i]people[/i]. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)
[/quote]

Generally speaking, I think you are quite right. However, at a certain point, one needs to choose the lesser evil - failing to deliver under the pressure of overwhelmingly complex projects, or bow to the creativity-repelling processes and hope the initial spark that started it all doesn't die. I thought I'd give it a go and try and fix one of those processes so it takes care of creativity and vision, instead of killing them.

Share this post


Link to post
Share on other sites
That's the thing - we already have great ways of delivering software without killing creativity. Hundreds of games ship every year using those techniques.

Why are those sequences of procedure less valid than your concept of a "process"?

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1313615377' post='4850472']
That's the thing - we already have great ways of delivering software without killing creativity. Hundreds of games ship every year using those techniques.

Why are those sequences of procedure less valid than your concept of a "process"?
[/quote]

You are saying everything is perfect now?

Share this post


Link to post
Share on other sites
The point is that there isn't, and never can be, a god killer technique or process that delivers creative results in a production environment. There isn't, and never [i]will be[/i], a beats-them-all technique for effective prioritization of requirements/ideas into a neatly folded list - there is too much interdependency and uncertain quantities to prove which ideas are the more profitable and effective ones. Even if you base it upon years of data mined from other successful titles nothing, [i]nothing[/i], guarantees that the same mix will prove viable another year ahead.

That's why we have this whole 'creative process' to begin with. Prioritization is another matter entirely and must be decided on a case-by-case basis, not by ultra-god-process-5000.

Share this post


Link to post
Share on other sites
[quote name='DarklyDreaming' timestamp='1313618222' post='4850498']
The point is that there isn't, and never can be, a god killer technique or process that delivers creative results in a production environment. There isn't, and never [i]will be[/i], a beats-them-all technique for effective prioritization of requirements/ideas into a neatly folded list - there is too much interdependency and uncertain quantities to prove which ideas are the more profitable and effective ones. Even if you base it upon years of data mined from other successful titles nothing, [i]nothing[/i], guarantees that the same mix will prove viable another year ahead.

That's why we have this whole 'creative process' to begin with. Prioritization is another matter entirely and must be decided on a case-by-case basis, not by ultra-god-process-5000.
[/quote]

I'm not convinced you've read what has been said until now. And it was a lot. I never made such claims. Nor are any of my intentions even close to what you suggest.

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1313606711' post='4850428']
The important part of the art exercise has nothing to do with what kind of art you like, or whether or not you can personally assign scores to something.

The point was in the [i]comparison[/i] to other people's opinions, and even the comparison between your immediate reaction to a piece, your short-term memory of that piece, and your lasting impressions of the same piece. If you personally don't think that people would have varying scores at those three time points, I would argue that you have either uncommonly flat taste, or uncommonly acute perception of artistic merit. I would also argue that you cannot generalize from self in this instance; that was, again, part of the point of the exercise. Just because you will react to art in one way does not in any way suggest that you can conclude what any arbitrary other person would think of the same exact work.

For instance, I find pastels to be irritating, and prefer grayscale photography and muted colors over anything with highly vibrant blues or greens. The point of the art gallery exercise is not that I can tell you those things a priori of walking into the gallery; the point is that (1) I may find a piece that does not fit my preconceived ideas of what I "like" and yet still appeals to me; (2) I may change my mind about a piece upon reflection; and (3) I'm going to have wildly different numbers than you do.



As for software practices: there are two kinds of software companies. One kind is rigidly attached to process, often informed by academic research, and spends tremendous amounts of money and time on things like certifications, process compliance, and so on.

The other kind is not a hellhole.


(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the [i]people[/i]. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)
[/quote]
I agree with the majority of this except for one very important point - it's [i]never[/i] about an arbitrary other person. It's always about a big group of people, otherwise known as a target audience or demographic, about whom we [i]can[/i] have a priori statistical knowledge. That's the core principle of marketing. It's also not usually about making a product in the 20% of things people are surprised to like, it's about making a product that's in the 80% of things people almost always like because that's a much safer bet, and the people who like a product against type are less likely to buy sequels and tie-ins and all that than the people who are always looking for more of the same type of thing.

Share this post


Link to post
Share on other sites
[quote name='DarklyDreaming' timestamp='1313618222' post='4850498']That's why we have this whole 'creative process' to begin with. Prioritization is another matter entirely and must be decided on a case-by-case basis, not by ultra-god-process-5000.
[/quote]
So let's say you are the lead designer or producer of a game, whatever title but the point is that it's your job to decide prioritization. The game design you bought says "we need either feature A or feature B here". Feature A and feature B would cost about the same amount to implement. You don't have a personal preference between the two. How do you decide? You have to decide because otherwise it's a stumbling block for your own personal work efficiency and the whole project. I think you need to have some overall strategy or rule of thumb for making this kind of decision.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this