How can I measure the "creative/entertainment value" of video-game requirements?

Started by
52 comments, last by Antheus 12 years, 8 months ago

I agree with the majority of this except for one very important point - it's never about an arbitrary other person. It's always about a big group of people, otherwise known as a target audience or demographic, about whom we can have a priori statistical knowledge. That's the core principle of marketing. It's also not usually about making a product in the 20% of things people are surprised to like, it's about making a product that's in the 80% of things people almost always like because that's a much safer bet, and the people who like a product against type are less likely to buy sequels and tie-ins and all that than the people who are always looking for more of the same type of thing.




Are you meaning to suggest that blanket application of statistics invalidates the existence of individualized personal preference? I can't really understand that opinion.

I feel like you're falling prey to a common disastrous mistake in thinking, which is assuming that the existence of trends obviates the need for individual data points. Demographics are comprised of individual people. Statistics are lossy information conveyances; "80% of things that people almost-always like" discards a fascinating amount of knowledge about the specific people and their specific tastes.

I think it's a tragedy to become so obsessed with "holistic" perspectives and aggregates that one loses sight of the constituencies.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

Advertisement

[quote name='chervenkoff' timestamp='1313616504' post='4850480']
[quote name='ApochPiQ' timestamp='1313615377' post='4850472']
That's the thing - we already have great ways of delivering software without killing creativity. Hundreds of games ship every year using those techniques.

Why are those sequences of procedure less valid than your concept of a "process"?


You are saying everything is perfect now?
[/quote]

I can't help but feel like you're just being combative at this point, which is disappointing.

I never said anything was "perfect." Just that we have good systems for producing creatively rich software.

I also asked you a question, which you unfortunately did not answer.
[/quote]

At the moment of replying, emotion might have gotten the better of me, for which I apologize. I guess I was fighting with a one-man army here for too long.

My rhetorical question was aimed at pointing out that there is always need for improvement. Pointing at successful projects and saying everything is OK is easy, however the truth is that more often than it should, projects fail. My research does not aim to replace the processes and practices already in use, but to complement them, without introducing significant overhead.
If you have any questions, please e-mail me (a.cherv@gmail.com), PM me or just post here. Thank you!

[quote name='sunandshadow' timestamp='1313625569' post='4850540']
I agree with the majority of this except for one very important point - it's never about an arbitrary other person. It's always about a big group of people, otherwise known as a target audience or demographic, about whom we can have a priori statistical knowledge. That's the core principle of marketing. It's also not usually about making a product in the 20% of things people are surprised to like, it's about making a product that's in the 80% of things people almost always like because that's a much safer bet, and the people who like a product against type are less likely to buy sequels and tie-ins and all that than the people who are always looking for more of the same type of thing.


Are you meaning to suggest that blanket application of statistics invalidates the existence of individualized personal preference? I can't really understand that opinion.

I feel like you're falling prey to a common disastrous mistake in thinking, which is assuming that the existence of trends obviates the need for individual data points. Demographics are comprised of individual people. Statistics are lossy information conveyances; "80% of things that people almost-always like" discards a fascinating amount of knowledge about the specific people and their specific tastes.

I think it's a tragedy to become so obsessed with "holistic" perspectives and aggregates that one loses sight of the constituencies.
[/quote]
A game design is more or less a piece of art, and as such it can only arise from the personal preferences of the game designer. My designs first and foremost exist to please me. But the designer is only one person, if they have a few preferences which are way out of line with the target demographic it's completely appropriate for these to get overridden during the development process even though that will probably annoy the designer. Being an artist is like that, trying to turn art into a commercial product always requires a bit of hack-work and pandering. If you're a professional you suck it up and move on, or if the proposed changes are too large you realize you are working with the wrong group and try to find people who have more tastes in common with you. As far as individual players go, no any one individual doesn't matter any more than one other individual; surveying them, especially a survey with one or more free-response boxes, is the best practical way to find out the opinions of a large number of individuals.

I want to help design a "sandpark" MMO. Optional interactive story with quests and deeply characterized NPCs, plus sandbox elements like player-craftable housing and lots of other crafting. If you are starting a design of this type, please PM me. I also love pet-breeding games.


At the moment of replying, emotion might have gotten the better of me, for which I apologize. I guess I was fighting with a one-man army here for too long.

My rhetorical question was aimed at pointing out that there is always need for improvement. Pointing at successful projects and saying everything is OK is easy, however the truth is that more often than it should, projects fail. My research does not aim to replace the processes and practices already in use, but to complement them, without introducing significant overhead.


The fact that projects fail does not suggest in any way that the process for managing creatively rich software is at fault.

Are you actually interested in project failure, or are you fixated on the notion of trying to repair something which is not broken? In the former case, perhaps you should actually study why projects really fail instead of just assuming (very incorrectly) that the demands of managing creativity has much to do with it. In the latter case, you continue to display precisely the wrong attitude and ask precisely the wrong questions.

If I was doing your research, my first priority would be to actually learn something about how real game development works. You are operating with a minimum of such knowledge and this is killing your credibility and leading you to waste a painful amount of time on diversionary arguments which have nothing to do with what you are trying to find out. You need to do research from a point of humility - always assume that you know nothing before you assume that you know anything. Having talked to a few developers does not an expert make; the fact that you've received so much criticism for your incorrect assertions about developmental realities should be a strong indicator of this problem.

Once that is done, you can move on to the real question, which is "what needs to be improved?" You're walking into this discussion with a preconceived notion of what needs to be improved, and despite being told repeatedly by several people that your preconception is wrong, you cling to it. A much more appropriate response for a researcher is to discard your incorrect view entirely and embrace reality. Honest discovery in research is never accomplished by slavishly working towards wrong assumptions. The really interesting breakthroughs come from people who refine their views progressively over time as new information comes to light. Half the cool stuff that researchers discover has nothing to do with what they originally sought to research!

I guess, in a nutshell, you need to be willing to walk away from your misguided current tack of research, and explore something different which is actually a valid problem. Depending on your motivations for asking these questions to begin with, that could either be very easy (e.g. you genuinely want to see fewer game projects fail, for instance) or it could be very difficult (i.e. you had a pet project and you're committed to it even if it proves out to be a waste of time).

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]


A game design is more or less a piece of art, and as such it can only arise from the personal preferences of the game designer. My designs first and foremost exist to please me. But the designer is only one person, if they have a few preferences which are way out of line with the target demographic it's completely appropriate for these to get overridden during the development process even though that will probably annoy the designer. Being an artist is like that, trying to turn art into a commercial product always requires a bit of hack-work and pandering. If you're a professional you suck it up and move on, or if the proposed changes are too large you realize you are working with the wrong group and try to find people who have more tastes in common with you. As far as individual players go, no any one individual doesn't matter any more than one other individual; surveying them, especially a survey with one or more free-response boxes, is the best practical way to find out the opinions of a large number of individuals.


*sigh*

I didn't hatch yesterday, you know. There's no need to treat me like I don't comprehend the basic concept of popular marketing.


My point is orthogonal to marketing, and always has been. My point has always been that individual variations exist and are interesting in their own right and should be considered interesting in their own right, quite independent of the pragmatics of marketing. I'm not trying to argue that marketing doesn't (or shouldn't) exist, nor am I ignorant of the reasons why it does exist, nor am I too stupid to appreciate the process itself by which statistical aggregation turns individuality into an exploitable mathematical model.

I really don't understand your need to disagree with me over something that is totally unrelated to your (stated) reasons for disagreement.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]


[quote name='sunandshadow' timestamp='1313794136' post='4851416']
A game design is more or less a piece of art, and as such it can only arise from the personal preferences of the game designer. My designs first and foremost exist to please me. But the designer is only one person, if they have a few preferences which are way out of line with the target demographic it's completely appropriate for these to get overridden during the development process even though that will probably annoy the designer. Being an artist is like that, trying to turn art into a commercial product always requires a bit of hack-work and pandering. If you're a professional you suck it up and move on, or if the proposed changes are too large you realize you are working with the wrong group and try to find people who have more tastes in common with you. As far as individual players go, no any one individual doesn't matter any more than one other individual; surveying them, especially a survey with one or more free-response boxes, is the best practical way to find out the opinions of a large number of individuals.


*sigh*

I didn't hatch yesterday, you know. There's no need to treat me like I don't comprehend the basic concept of popular marketing.


My point is orthogonal to marketing, and always has been. My point has always been that individual variations exist and are interesting in their own right and should be considered interesting in their own right, quite independent of the pragmatics of marketing. I'm not trying to argue that marketing doesn't (or shouldn't) exist, nor am I ignorant of the reasons why it does exist, nor am I too stupid to appreciate the process itself by which statistical aggregation turns individuality into an exploitable mathematical model.

I really don't understand your need to disagree with me over something that is totally unrelated to your (stated) reasons for disagreement.
[/quote]
I don't think you're stupid or anything like that. The situation is that I am baffled by what you perceive to be the problem. So I figured if I spelled out my own thoughts as clearly as possible, and used the same terms you were using like creativity and individuals, you'd be able to point at one of my pieces and say, there, that one's the one I disagree with. Individuals are interesting, but games are a mass market product, so I just don't see the relevance or usefulness of focusing on variations in the context of game production and marketing.

I want to help design a "sandpark" MMO. Optional interactive story with quests and deeply characterized NPCs, plus sandbox elements like player-craftable housing and lots of other crafting. If you are starting a design of this type, please PM me. I also love pet-breeding games.


The fact that projects fail does not suggest in any way that the process for managing creatively rich software is at fault.

I agree. Moreover, I never said projects fail due to weak creative vision management. I am doing this research because as I said many times already, video games resemble film art (and other arts), where the creative/core vision needs to be shared by all stakeholders involved in the creation. It seemed logical this aspect needs to be reflected in the development of games as well. Many developers agree, academicians have encouraged my research, and some of the replies here agree as well.


Are you actually interested in project failure, or are you fixated on the notion of trying to repair something which is not broken? In the former case, perhaps you should actually study why projects really fail instead of just assuming (very incorrectly) that the demands of managing creativity has much to do with it. In the latter case, you continue to display precisely the wrong attitude and ask precisely the wrong questions.

If I was doing your research, my first priority would be to actually learn something about how real game development works. You are operating with a minimum of such knowledge and this is killing your credibility and leading you to waste a painful amount of time on diversionary arguments which have nothing to do with what you are trying to find out. You need to do research from a point of humility - always assume that you know nothing before you assume that you know anything. Having talked to a few developers does not an expert make; the fact that you've received so much criticism for your incorrect assertions about developmental realities should be a strong indicator of this problem.

Once that is done, you can move on to the real question, which is "what needs to be improved?" You're walking into this discussion with a preconceived notion of what needs to be improved, and despite being told repeatedly by several people that your preconception is wrong, you cling to it. A much more appropriate response for a researcher is to discard your incorrect view entirely and embrace reality. Honest discovery in research is never accomplished by slavishly working towards wrong assumptions. The really interesting breakthroughs come from people who refine their views progressively over time as new information comes to light. Half the cool stuff that researchers discover has nothing to do with what they originally sought to research!

I guess, in a nutshell, you need to be willing to walk away from your misguided current tack of research, and explore something different which is actually a valid problem. Depending on your motivations for asking these questions to begin with, that could either be very easy (e.g. you genuinely want to see fewer game projects fail, for instance) or it could be very difficult (i.e. you had a pet project and you're committed to it even if it proves out to be a waste of time).

[color="#171E29"][font="Arial"]I am not trying to investigate why projects fail – there is academic literature on this and I have reviewed it (and I’m still reading more). The literature further motivates me to do this research, but since you repeatedly questioned my knowledge, I have nothing else left to do but to cite contemporary research on the topic: [/font]


[color="#891917"][font="Arial"]Software[/font][color="#891917"][font="Arial"] Engineering Challenges inGame Development[/font]

[color="#27500B"][font="Arial"]C M Kanode[/font][color="#535353"][font="Arial"], [/font][color="#27500B"][font="Arial"]H M Haddad [/font][color="#535353"][font="Arial"]in[/font] [color="#262626"][font="Arial"]Information Technology NewGenerations 2009 ITNG 09 Sixth International Conference on[/font] [color="#262626"][font="Arial"](2009)[/font]

[color="#262626"][font="Arial"] [color="#171E29"][font="Arial"]Some highlights from Kanode and Haddad’s research:[/font]

[color="#171E29"][font="Arial"]"A major issue leveled against the games industry is that most adopt a poor methodology for software creation. Petrillo et al. refers to data collected by the Standish Group. Only 16% of projects are actually completed on time and on budget. Clearly, there is a problem. Based on the statistics gathered by Petrillo et al. of completed games, the errors with the greatest occurrence (over the 50% mark) fall roughly under project management, requirements engineering, and risk management."[/font]

[color="#171E29"][font="Arial"]First paragraph of the conclusion:[/font]

[color="#171E29"][font="Arial"]"Game development has unique characteristics that represent challenges to this industry. Applying SE principles and sound practices can help overcome these challenges. Development companies must invest in adopting proven methods, found in traditional SE, to fit with the peculiarities of game development such as the management of multimedia assets and the need for game play exploration. Many issues in videogame development point to project management. Development companies need to invest in grooming skilled management through training (not only managing people, but teaching solid project management skills)."[/font]

[color="#171E29"][font="Arial"]Note: As I said many times over, that’s exactly what I’m trying to do – study and adapt popular solutions. [/font]
[/font]

[font="Arial"][color="#262626"][color="#891917"][font="Arial"]Houston[/font][color="#891917"][font="Arial"], we have a problem...: A Survey of Actual Problems in Computer Games Development[/font]

[color="#262626"][color="#27500B"][font="Arial"]Fabio Petrillo[/font][color="#535353"][font="Arial"], [/font][color="#27500B"][font="Arial"]Marcelo Pimenta[/font][color="#535353"][font="Arial"], [/font][color="#27500B"][font="Arial"]Francisco Trindade[/font][color="#535353"][font="Arial"], [/font][color="#27500B"][font="Arial"]Carlos Dietrich [/font][color="#535353"][font="Arial"]in[/font] [color="#262626"][font="Arial"]SAC2008 23rd Annual ACMSymposium on Applied Computing[/font] [color="#262626"][font="Arial"](2008)[/font][/font][font="Arial"][color="#262626"]
[color="#262626"] [/font]
[font="Arial"][color="#262626"][color="#171E29"][font="Arial"]Some highlights of Petrillo et al.’s research:[/font]

[color="#262626"][color="#171E29"][font="Arial"]"When we analyze this results more closely, we see that the most cited problems were the unreal or ambitious scope and features creep with 75% (15 of 20) of projects reporting this problems. After that, demonstrating the coherence of the postmortems, 70% of projects citing the cutting features during development process. The other most found were problems in the design phase and delay or optimistic schedule, with 65%.Also the technological problems, with 60% can be highlighted. Figure 1 presents the histogram of occurrence of problems in sequence decreasing, in which we can make a graphical comparison ofthese results."[/font]

Note: Find figure 1 at the bottom.

[color="#262626"][color="#171E29"][font="Arial"]First paragraph of the conclusion:[/font]

[color="#262626"][color="#171E29"][font="Arial"]"Our work shows that indeed all the main problems of traditional software industry are also found in the games industry, and it is possible to affirm that they are very related. In both contexts, for example, the unreal scope was pointed out as critical, as the problems with requirements definition. This contrasts with the one of so called universal problems of games industry, like crunch time, which had a relatively low occurrence rate, with 45% for crunch time. Theproblems with over budget and the loss of professionals have had the lowest incidence among all the problems analyzed, with an occurrence rate ofonly 25%. Budget problems have a considerable discrepancy, probably due to the greater emphasis in the technological and management aspects given by the stories authors, minimizing the financial aspects of the project. We can note similarities with respect to classification (same problems with same importance) and frequency of some problems, as schedule delays and requirement problems, having almost equal rates. Thus, one possible consequence of this similarity is obvious: solutions adopted successfully to traditional software industry to solve some problems can be also adopted by the games industry."[/font]

[color="#262626"][color="#171E29"][font="Arial"]Note: the boldface highlights are done by the author.[/font][/font]

[font="Arial"]A diagram from the research:

unledpuh.jpg

[color="#171e29"] [color="#171E29"][font="Arial"]My research is relevant to the top 5 problems you see in the diagram. Make your own conclusions.[/font]

[color="#171E29"][font="Arial"]If you still don’t believe in the necessity of my research, that’s fine, I’ll sleep just as well. However, don’t expect me to quit my research. There are people on both sides of this fence, and as an unbiased researcher, I hope to be able to reflect every aspect and view.[/font]
[/font]
P.S. If anyone needs the two publications, just e-mail/PM me, I'll gladly send them to you.
If you have any questions, please e-mail me (a.cherv@gmail.com), PM me or just post here. Thank you!
OK, let's try this from the beginning.

1. What are you actually trying to do here?
Your story has changed pretty much every other post in this thread, and I'm really losing track of all the different things you claim to be interested in and not interested in at various turns. So why can't we just get one straight answer on what you're trying to do, and we'll call it good?

2. Why do you think your research is relevant to those five problems?
Nothing you've talked about thus far has anything to do with typical solutions to the five top problems in the chart you posted. Moreover, you haven't shown any indication that you know how those problems are successfully addressed by the teams who do overcome them. All I see here is what I already know of the industry: most teams are mismanaged and run by people who are ignorant of the existing solutions to all of these issues. The minority of teams - the ones who are consistently successful - have solved these issues many, many, many years ago.

3. Why do you think there is any need for novel approaches to any of these problems, versus simply educating those who are unaware of the known solutions?
You keep talking as if you are trying to invent something new here, when it is not at all clear that the problem is that we need anything new. The problem is that people are not in the loop with how the actual successful teams operate; it's not as if a few people get magically lucky and don't understand why.

Wielder of the Sacred Wands
[Work - ArenaNet] [Epoch Language] [Scribblings]

I have nothing else left to do but to cite contemporary research on the topic:[/quote]

Quoting reference is the *very first* thing to do, not last resort.

video games resemble film art (and other arts), where the creative/core vision needs to be shared by all stakeholders involved in the creation.[/quote]

Hollywood, Bollywood, indie, home cinema, user generated content, small and medium budget production, commercial productions, cinema, DVD, online productions, ....

Each of this is a world on its own, with different rules. Hollywood and Bollywood are both "AAA", yet they could not be more dissimilar in everything. What works in one type of production in one of them has nothing to do with another.

Same for software industry. Of course budget will be a reason for failure. The number seems surprisingly low, given the total number of companies that go bankrupt every year.

A good question to ask would be: How does number of projects cancelled or failed due to budget reasons compare to number of bankruptcies filed. Any statistically significant deviation between the two would indicate difference in behavior which could lead to applicable solutions.

"Unreal of ambitious scope" - I don't understand how this can be a failure. It simply means that project was completed by vastly exceeding original goals. If it failed, it did not fail due to that - the cause was: missed dead-line, running out of funding, being cancelled by publisher (due to deadline or running out of funding).

Another failure mode that can be examine is negative ROI. A project might not have failed (launched, released, sold), but investment did not cover the development.

Same for the other 4. None of those are *failures*. A project fails when it's not complete by some deadline or when it is cancelled, but this means funding. So all projects that fail do so due to funding. And vice-versa - if project failed due to "budget", it means its scope was too broad for available funds.

And we're back to money. Whether you like it or not.

So now the first step is to study the relationship between publishers and developers and the process involved.

Hopefully this also indicates the generally negative reception. The metrics measured by such work simply doesn't do anything useful and is merely stating the obvious in convoluted way: Companies cancel projects that will not make them profit.
Many good points made in this discussion, I think if the goal is to improve the current process you must have a solid grasp of the current process. I've noticed that each company has their own "way" of doing things. Some are very much are hierarchical and ideas only flow from the top down, no exceptions. Others are more egalitarianism, ideas can come from the lowest person on the ladder ( a temporary tester for instance ) and are seriously considered and evaluated without prejudice. The thing is both systems work, if you have a core of highly competent and creative individuals they really don't need external "assistance", or if you have an open environment where all share in the creation of a product this too can work.

Let's discuss the top 3 issues on why game development fails according to [color="#171E29"][font="Arial"]Petrillo..[/font]

Unreal or ambitious scope
Feature creep
Cutting features

Consider that the top 3 problems are related to "features and scope" why is that? Because the games industry is highly competitive, it forces competitors to add features to keep pace with their peers. Games which just throw out features (key features) or never adapt to the changing market, are at a competitive disadvantage.. No Coop? no Multiplayer? no DLC? no sale.. Of course each game is different so not all this applies.. I personally don't think Angry Bird needs Coop etc..

Mature and successful teams know there really isn't anything like "Unreal or ambitions scope", they have enough competence to achieve even their wildest ambitions, see Steam, see RDR2, see WOW, etc.. The issue is of course time and money. Experienced leads and project managers can estimate a features cost within a reasonable time scale.. This is the first point of failure, incorrect assessment of time and effort.

Feature Creep? This isn't really a point of failure from my experience. This comes back to the fact that games are a collection of interacting systems.. after an initial play through of a rough slice of the game, if it's no fun that's a bad sign. Ultimately it is a game and for it to sell, it has to be fun. Weight that against the schedule would you rather spend the remaining time finishing a product which is no fun or invest every effort to make it fun? Every year 100's of un-fun games are released, they don't make back their return and are forgotten. Sure the project completed but that's really no achievement if the company goes out of business afterwards or the team is laid off.. Companies which can afford to (ie Rockstar, Blizzard, Sony, Valve etc..), invest the time to make it fun and are not slave to the schedule. Those who can't press on and hope for the best. What can be done about this? I think the reason why things don't turn out to be "fun" is because lack of shared vision and creative leadership. People usually have an idea of what is "fun".. but for something to be "fun" alot of factors have to align, and without a shared vision of why or what is fun in a game, people can very often put in less than satisfactory effort in the needed aspects to make it fun..

Cutting features, i think this is needed actually. Back to the previous point games are a collection of systems, like the reverse analog of "feature creep", sometimes what it takes to make a game fun is to remove a feature. That's just the reality, after play through, after some insight, maybe you don't need a super smart horse AI? Maybe.. :) Even that being said, cutting the wrong feature for the wrong reasons can equally kill a game and I've seen that happen too. So how do u know what to cut? Again goes back to above you need to have a shared vision and creative leadership on what makes the game fun.

Again this comes back to the people, the team.. Good teams, more often than not they will make the right decision on what to add, what to cut, and what extent to take a feature or scope of a game within the time frame they have..

Hope this gives you some insights into the process.. Again i'll reiterate, each team is different and for some these top 3 are not their biggest stumbling blocks at all.. etc..

Good Luck!

-ddn

This topic is closed to new replies.

Advertisement