Jump to content
  • Advertisement
Sign in to follow this  
chervenkoff

How can I measure the "creative/entertainment value" of video-game requirements?

This topic is 2496 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I realize this question is complex and subjective, but bear with me for a moment.

I firmly believe that video-game software is essentially different from, for example spreadsheet software, as from a user point of view, some games have more similarities with film art than with regular software. However, when it comes to requirements prioritization methods that help developers sort out which feature or idea is most important, there is none that facilitate the creative aspects of video games. Traditional techniques only aid regular software development where focus is on the risk and cost of requirements, and do not take into account the creative aspect of video-game requirements.

To illustrate the problem, think of a requirement (i.e. feature idea) such as the one seen in Super Paper Mario, where Mario can exit the 3D world and enter a 2D world, and vice versa in order to go around obstacles. This is a fairly challenging requirement to implement and probably costly, but definitely crucial to the core/creative vision of the game. Available prioritization methods overlook the creative/entertainment value of this requirement and mainly just estimate how costly and (technologically) risky implementation is. Of course, this is a very simplistic example, but you can imagine a case where there is a pool of great game ideas (functional and non-functional requirements) and not enough resources to realize them all. When it comes to value calculation, academic literature doesn't provide game developers with an adequate way to estimate the value of game requirements.

UPDATE / Clarification: In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and try to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the outside in vs. the inside out paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects.

I try to refine part of the preproduction process by developing a requirements prioritization method tailored to the needs of game development industry. A pivotal element in such a method is the ability to identify and estimate the creative/entertainment value of requirements (as explained above). However, in order to do that, I need to understand how game developers perceive this creative/entertainment value of requirements. In a nutshell, I am seeking answers to the following questions:

Question 1: How would you define the creative/entertainment value of video-game requirements?

Question 2: How would you measure it?

Question 3: Who should measure it?

I would love to see how these issues are perceived by game developers and I would appreciate your take on them here, but if you want to contribute to this research – receiving my eternal gratitude and a proper credit/citation in the research and all the publications that will follow, please fill up my extremely short survey (just 7 questions, 3 of which you already see above):

The extremely short survey which will cement you as my personal hero. Also, you will get a copy of my research once it's finished and the designed solutions (e.i. the Creative Requirements Prioritization method).


UPDATE / Clarification: If my intentions are still confusing, check out my explanation [color="#1D3652"]here.


Research information and trigger

This research is being conducted at Utrecht University, The Netherlands, as part of an Information Science master’s thesis.

Game development is in many ways similar to product software development as developers follow certain software development processes. Following a poor development method (or none at all) could result in longer development times, going over budget and/or delivering buggy products (Bethke, 2003). What makes video games different is the creative game vision that must be shared by the entire team to ensure that the end product is consistent and of good quality. This is especially true for full-blown game titles where from a user point of view, there are more similarities with film art than with any other software. Unfortunately, this curious creative aspect renders many software product management techniques unadoptable by the gaming industry.

I am quite interested in working towards improving the game development process in the preproduction stage, by creating a requirements prioritization method tailored to the specific needs of the game industry and I need your help! The easiest way to contribute to the research is by filling up my short survey (link above). If you find this research interesting, please contact me at a.cherv@gmail.com

Researcher information

My name is Alex Chervenkoff, an avid gamer and quite excited about this research! You can contact me at: a.cherv@gmail.com

Degree in BSc Computer Science, The University of Sheffield, UK.

Currently doing MSc Information Science at Utrecht University, The Netherlands.

Share this post


Link to post
Share on other sites
Advertisement
Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 in under ten seconds. You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may not consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the third most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the third.

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation whatsoever between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally not quantifiable. If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers you assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone wanted to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that were fun and were original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.

Share this post


Link to post
Share on other sites

Here's an exercise for you.


Go to a local art gallery, preferably one doing a new showing by an artist you are not familiar with (this is a lot more effective if it's the first encounter you've ever had with their artwork). Bring a notepad and a pen.

You have three assignments.

First, every time you see a piece, rate it from 1-100 in under ten seconds. You do not get to think long and hard, or study the piece, or anything; just make snap judgments. If you're doing it right, this should be difficult.

Second, when you leave the show, rate the ten most interesting pieces you saw, and write a two-sentence description of each. You may not consult your snap-judgment scores; this must be done independently of any consideration you have made of the pieces thus far. Include a rating from 1-100 of each piece, again, independently of the snap-judgment scores. If you're doing it right, this should be extremely hard.

Third, wait a week. Do not look at your snap judgments, do not read your two-sentence summaries, nothing. Forget about the show, in effect. Then, after one week, pick the third most interesting piece you saw (that you can remember) and write one paragraph describing it. Now score it from 1-100. Remember: not the most interesting or impressive or memorable piece, the third.

If you did that right, it should be damn near impossible.

For bonus points: go back and look to see if there is any correlation whatsoever between the scorings you did in each of the three steps. I will bet you a nice beer that you would have equivalent results just randomly picking numbers without actually going to the show at all.

Extra bonus points: get a friend to do the same assignment with you, and never share your numbers. This works best if your friend does not have similar tastes in art. After two weeks, show each other your lists and scores. See how well they line up.

Mega bonus points: see if your two sets of scores correlate at all with the pieces the artist had the most fun creating, or the pieces that their agent decided were good enough to merit getting them a gallery showing in the first place.


My point here is that creativity and "enjoyment" are fundamentally not quantifiable. If we could attach numbers to these things, rest assured that someone would have invented the analogue of an actuarial position for rating subjective qualities, and there would already be a thriving industry that was already extremely good at this. The fact that no such industry exists should be a strong sign that we do not have, even at a theoretical level, any way to quantify these things.

Worse, the numbers you assign to something's enjoyment-factor may vary wildly from someone else's. They may also have nothing to do with how much someone wanted to make the thing you're trying to enjoy; a lot of games get made because people are passionate about them, not because they will peak out the fun-o-meter for a lot of customers. The fact that other people enjoy those games is usually just a bonus. But look at some of the truly most creative and original games out there: you will find wildly different opinions about how "fun" or even how "creative" those games are. I bet you can name a few games that had really innovative concepts but were executed so poorly that the game wasn't fun; or, worse, games that were fun and were original and had no sales.

The trump card, though, is this: I bet you can name a few games that are neither creative, original, or particularly well-executed, and yet are among the most financially successful franchises on the planet.


People try every year to quantify the artistry and magic that makes games successful. They have consistently failed. Unless you bring a radical new insight into human nature to the table, you will also fail. Frankly, simply by asking this question, you are showing that you don't have that novel insight yet - and therefore, your attempts will probably not get you anywhere.

Please don't take that as any kind of slight against you or anything; there is no shame in failing to solve a very hard problem. Just understand that the odds weigh painfully against you.


[font="Arial"]Thank you for your extensive reply ApochPiQ![/font]

[font="Arial"]Rest assured criticism won’t offend me – it can only bring insight to the research at hand. I absolutely understand how subjective the issue of enjoyment is, and your “exercise” did a very good job of illustrating how an individual would fail at providing an objective, reproducible rating.[/font]

[font="Arial"]I have thought of numerous reasons why this couldn’t work, and I have talked to game developers who have also expressed why they think this is a hard task… but giving up never helped solving any challenge. Instead of looking at ways it can’t be done, I decided to focus on how something like this can work. In your answer, you rightly raised a couple of points I had already put some thought and research into:[/font]

[font="Arial"]First, one person cannot assess a piece of art in under 10 sec. However if all the stakeholders involved in making the game express their opinion on a feature (possibly by assessing a prototype, sketches etc.), there will definitely be some benefits from a prioritization point of view. Moreover, many game studios have at their exposal a group of gamers who represent their target group. They could be involved in the prioritization of features as well, and with their statistically more significant numbers than a single vote, can definitely provide a good idea of which feature is desired more than another.[/font]

[font="Arial"]Your second point, the passion and vision of the creator – I absolutely agree with you that a lot of games are pure result of the passion and vision of their creators. I actually want to make sure this vision and the passion that goes with it is translated in the list of ideas that make up the potential game. I believe that without sharing the vision and passion, chances are the end product will be disappointing. However, when overwhelmed with ideas, one needs to take a step back and assess which of these ideas resonate best with the vision of the game, and that task seems impossible to an individual, and this is why any solution that comes out of this research will rely on many stakeholders.[/font]

[font="Arial"]I do not want to degrade creativity by saying it can be quantified and represented by a simple number. Nor do I want to create a tool that “Guarantees the success of your title 100%!” However, I don’t think I have undertaken an ambitious project. I just want to take a closer look at why are so many software product management techniques unpalatable to the gaming industry. It seems it’s the creative element and the entertainment value of game software that is never taken into account in those otherwise useful software development solutions. I believe that by understanding the problem a bit better, I will be able to rework/combine/adapt some of the available requirements prioritization algorithms to better suit the specificities of the domain.[/font]

[font="Arial"]To summarize, I am not trying to devise an objective way of measuring the creative/entertainments value of game ideas. Ultimately, I am trying to lower the risk in the game development process by allowing developers and other stakeholders to adequately estimate the attractiveness of features/ideas during the preproduction stage. I’m sure you will agree this is no bad idea. Is it challenging? Yes. A master thesis needs to be.[/font]

[font="Arial"]I appreciate any feedback, suggestions and criticism, be it ways these ideas could work, or reasons why they won’t. I bet the former is more challenging and I’m sure there are people that not afraid of a challenge. If you want to be part of this research, I again encourage you to check out my 7-question survey (link in the original post above) or if you just want to share your opinion you can do it as an answer here.[/font]

[font="Arial"]Any help is highly appreciated![/font]

Share this post


Link to post
Share on other sites

[font="Arial"]I do not want to degrade creativity by saying it can be quantified and represented by a simple number. Nor do I want to create a tool that “Guarantees the success of your title 100%!” However, I don’t think I have undertaken an ambitious project. ... It seems it’s the creative element and the entertainment value of game software that is never taken into account in those otherwise useful software development solutions. I believe that by understanding the problem a bit better, I will be able to rework/combine/adapt some of the available requirements prioritization algorithms to better suit the specificities of the domain.
[/font]
[font="Arial"]In other words, you admit it cannot be quantified, and then say you want to quantify it.
[/font]

[font="Arial"]To summarize, I am not trying to devise an objective way of measuring the creative/entertainments value of game ideas. Ultimately, I am trying to lower the risk in the game development process by allowing developers and other stakeholders to adequately estimate the attractiveness of features/ideas during the preproduction stage. I’m sure you will agree this is no bad idea. Is it challenging? Yes. A master thesis needs to be.[/quote][/font]
[font="Arial"]Again, you say you are not trying to quantify it. Yet your next statement is that you want a way to estimate (or quantify) it.
[/font]


What you seek has a history of not being measurable despite many attempts.

There are many scholarly articles that attempt to make such predictive measurements. Malone & Lepper's work on motivation, and various papers by J.P.Gee, B.Dodge, and others, repeatedly attempt to generate such metrics.... Yet further study on every one had those metrics destroyed by counterexample.

You simply cannot measure fun. Even after it is complete, even after you have a finished product, you still cannot measure fun or creativity.

You can measure many other facets that will help in determining the profitability of a game, including comparing it to other existing product. But measurements of 'fun' is not one of them.

Share this post


Link to post
Share on other sites

What you seek has a history of not being measurable despite many attempts.

There are many scholarly articles that attempt to make such predictive measurements. Malone & Lepper's work on motivation, and various papers by J.P.Gee, B.Dodge, and others, repeatedly attempt to generate such metrics.... Yet further study on every one had those metrics destroyed by counterexample.

You simply cannot measure fun. Even after it is complete, even after you have a finished product, you still cannot measure fun or creativity.

You can measure many other facets that will help in determining the profitability of a game, including comparing it to other existing product. But measurements of 'fun' is not one of them.


I have obviously confused people with my explanation, despite my attempt at the exact opposite. Let’s see if I can clarify my intentions.

In my research, I study available software product management solutions (more specifically requirements prioritization algorithms) and trying to find out why are they not suitable for game development. It seems that it is the creative or entertainment nature of the software itself that introduces this incompatibility. It is the (available) techniques inadequacy, to recognize the core value of the software and its conceptually different purpose.

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the outside in vs. the inside out paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision. This will ultimately provide a creative value, but it is relative to the central idea and the stakeholders’ ability to subjectively assess the requirements. This is just one side of the RP algorithm, as other factors such as risk and cost need to be taken into account as well, but the available RP solutions already offer adequate ways to do that, and they are compatible with the needs of game development.

The reason I’m writing here is because I am trying to see how developers deal with these problems (PR and focusing on the core idea) when dealing with more complex projects. I realize the question title is misleading if you take it without carefully inspecting the kilometric and confusing explanations I provided on my way to explain my research. I chose it because I thought it would spark the curiosity of the readers, which it did, but for all the wrong reasons.

Thank you for your replies and I hope I get more opinions. If you understand my goals by now, I kindly ask you help my research by filling out my very short (7 questions only) survey.

Share this post


Link to post
Share on other sites

With regular software, it is the user or his/her needs that most often help estimate the value of requirements. Moreover, it is the user that produces most requirements. In video-game software, it is the creative vision that drives the requirements. With regards to requirements origin, it is the outside in vs. the inside out paradigm. Having established this fundamental difference, we can deduce that if requirements prioritization (RP) algorithms focus on user ideas/needs in order to estimate the value of requirements for regular software, then in the case of video-game software, the RP algorithm should satisfy the creative vision. By creative/entertainment value of a requirement, I refer to the degree to which the core vision relies on this particular requirement.

I want to say: WTF?!


I think perhaps you should spend some time actually in the industry before trying to write up what the industry does.

My suspicion is that your only exposure is what you have seen on the outside, while asking questions from within your academic studies.


1> Have you actually worked at a game studio designing games from conception to completion? Your questions clearly announce "No"! It seems clear you simply do not understand the process. It is complex and varied, and never has the steps you list. Saying again: Your guesses about the game industry design process are fundamentally invalid.

2> Have you actually worked at one of the "other industries" you mention in such a designing role? I want to know what industries specifically you are comparing with. A few major companies have published their processes, but what they write about vs what it means when sitting in a cubicle are entirely different. It is complex and varied, and never works as the textbooks claim.

That type of thesis would best be approached by actually having worked in both industries. If not a direct employment job, then as a consulting role so that you actually know and understand what you are talking about.



Your questions, and your survey (as far as I cared to enter it, you ask questions that I don't feel comfortable answering because they are utter nonsense) show that you have no idea what it actually takes to make a game in a professional environment. You might as well include the process of making grilled parepuls while on the planet Abrigreene as it has the same level of experience.

Share this post


Link to post
Share on other sites
Choice is a good metric for entertainment value. Players get bored when theres nothing to do and it doesnt take much to play the game, but when there are interesting choices it becomes fun. Players try to maximize some value based on their choices.

3D Mario offers a new dimension of choice. The rewards of the choices are also mote engrossing.

Share this post


Link to post
Share on other sites
I want to say: WTF?!


I think perhaps you should spend some time actually in the industry before trying to write up what the industry does.[/quote]
You haven't dealt with academia much, have you?



Here is about the only metric that anyone will pay any attention to: sales figures.

In short - does it sell more copies? Then it's good. If it doesn't, then it's not.

For service centric development, an iterative process is used. Measure everything, iterate fast, keep what improves conversion, kill everything else fast.

What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision.[/quote]
Most developers today would cry "waterfall". And in many ways, it is an obsolete practice in time of infinite computing power.

Process for just about everything looks like this:
- launch yesterday
- gather metrics on everything, a gigabyte per day is good, terabyte is better
- analyze what resulted in most conversions (aka revenue), identify funnels (steps at which the conversions stop), fix and improve
- go to beginning

This should be done several times a day, preferably more.

In video-game software, it is the creative vision that drives the requirements.[/quote]
No, sorry.

Sales figures. Nothing else drives the requirements. The handful of very rare projects that succeed on "creative vision" (aka, perfect storm of luck, viral marketing and viable monetization) are flukes.


Regarding the intention - it's in the right direction. But the algorithm used has been put into practice cca. 1998 by a small company called Google. Since then, with increasingly cheap computing power, the methodology has been adopted by every single tech startup and is even applied by VC and angel investors at level of companies themselves.

Short release cycle to actual market, measure, improve vs. kill, repeat. The faster, the better.

Welcome to the future, where engineers or developers are irrelevant to success.

Share this post


Link to post
Share on other sites
frob, I never said I worked in the industry. As you rightly pointed out, my view is predominantly from the outside of the industry and my assumptions are based on the findings from my initial interviews with developers and also on academic literature, which you so bluntly point out is hardly followed in reality.

Be that as it may, what I don’t understand is why the hostility? I have come to a place where people seek knowledge from people in the industry, to do exactly that. Moreover, my survey’s questions are not presupposing an answer – they are open-ended and you can easily answer “This question is irrelevant in the real world because…”, which would help my research. Instead, you choose to smugly parade your supposed superior understanding of what-actually-happens by putting effort in devising “amusing” stories about the planet Abrigreene. What is so enraging and wrong about asking how are things done?

That type of thesis would best be approached by actually having worked in both industries. If not a direct employment job, then as a consulting role so that you actually know and understand what you are talking about.


I absolutely agree that would be the best way to do this research. However, to my knowledge no one has done it. I have tried to get in a larger game studio (as a researching intern) where complex projects are developed, but sadly, the fear of leaking inside information to the competition renders such a practice virtually impossible over here (the Netherlands). The second best thing I could do is interview game developers and engage online discussions on the topic, which I am trying to do here.

I hope others will understand the limitations of my research resources and help me out here. Again, any help is highly appreciated!

Share this post


Link to post
Share on other sites
I think part of the problem is that you came in making a lot of assertions and statements about how you think things are done; it was not at all clear to me that you were asking if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!