Here is about the only metric that anyone will pay any attention to: sales figures.
In short - does it sell more copies? Then it's good. If it doesn't, then it's not.
For service centric development, an iterative process is used. Measure everything, iterate fast, keep what improves conversion, kill everything else fast.
What I try to do is find a way to prioritize the requirements according to their relevance/importance to this core creative vision.
Most developers today would cry "waterfall". And in many ways, it is an obsolete practice in time of infinite computing power.
Process for just about everything looks like this:
- launch yesterday
- gather metrics on everything, a gigabyte per day is good, terabyte is better
- analyze what resulted in most conversions (aka revenue), identify funnels (steps at which the conversions stop), fix and improve
- go to beginning
This should be done several times a day, preferably more.
In video-game software, it is the creative vision that drives the requirements.[/quote]
No, sorry.
Sales figures. Nothing else drives the requirements. The handful of very rare projects that succeed on "creative vision" (aka, perfect storm of luck, viral marketing and viable monetization) are flukes.
Regarding the intention - it's in the right direction. But the algorithm used has been put into practice cca. 1998 by a small company called Google. Since then, with increasingly cheap computing power, the methodology has been adopted by every single tech startup and is even applied by VC and angel investors at level of companies themselves.
Short release cycle to actual market, measure, improve vs. kill, repeat. The faster, the better.
Welcome to the future, where engineers or developers are irrelevant to success.
[/quote]
Thanks for the reply Antheus!
I have been hearing this view on the matter by quite some developers. The focus seems to be entirely on what would sell best. However, I still wonder what happens when a new, original idea for a game is presented. Is there a methodical way to deal with a pile of ideas that comprise this game? What would the the process of picking the most important/core requirements be?
The "ghetto testing" you referred to is also quite an interesting method. Excuse my ignorance, but I am not sure which algorithm/methodology you refer to when talking about Google.
How can I measure the "creative/entertainment value" of video-game requirements?
Author
Author
I think part of the problem is that you came in making a lot of assertions and statements about how you think things are done; it was not at all clear to me that you were asking if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.
I didn't do a great job at presenting my problem. The statements were formulated based on my limited exposure to industry expertise from the few interviews I had with game developers and academic literature on both game development and regular software development. It has become clear that the general best practices for product software management do not resonate well with game development.
I would love to improve the post (and my understanding of the issues discussed) if anyone points out wrong assumptions/statements and suggests what the real-world case is. Your first reply helped me understand I have not presented my questions unambiguously. I have edited the post since and I hope it makes sense now and my research seems less ambitious (e.i. than trying to quantify creativity) than it originally did.
[quote name='ApochPiQ' timestamp='1313499978' post='4849828']
I think part of the problem is that you came in making a lot of assertions and statements about how you think things are done; it was not at all clear to me that you were asking if those assumptions were correct. From what you posted, it certainly sounded like you were convinced of the truth of your statements, not trying to verify some information.
I didn't do a great job at presenting my problem. The statements were formulated based on my limited exposure to industry expertise from the few interviews I had with game developers and academic literature on both game development and regular software development. It has become clear that the general best practices for product software management do not resonate well with game development.
I would love to improve the post (and my understanding of the issues discussed) if anyone points out wrong assumptions/statements and suggests what the real-world case is. Your first reply helped me understand I have not presented my questions unambiguously. I have edited the post since and I hope it makes sense now and my research seems less ambitious (e.i. than trying to quantify creativity) than it originally did.
[/quote]
It isn't that they don't resonate, it is that they generally don't apply. The are invalid.
First, I really do understand your background in academics. I went to graduate school too, and did the whole thesis thing. Of course, I did it in a technical field on algorithms and based my research as an extension to various SIGGRAPH and InfoVis papers, so it required new mathematical research, not business research.
Over the years I've worked in game development, and in business backend software, in embedded hardware development, and in various presentation industries (broadcast tv software, broadcast TV software, etc.)
In business backend software where I was one of the leads, we did not follow the practices you originally stated as fact. We were told by the company owner what the business needed, such as integration to SalesForce CRM system. We studied out the differences between our current workflow and the SalesForce workflow. We spent months writing ETL jobs, developing a plan to phase over the various other workers, moving our data over, then retiring our older servers. Similarly for other projects we were never asked to prioritize features. They were always created by business need and scheduled based on how they affect profit.
In hardware the software need to calibrate and display in real-time the hardware sensor data. There was no major prioritization of requirements. If the sensor was able to detect it, we needed to show it. If the sensor could be calibrated in any way, we needed to expose it. Again, this was not based at all in what you stated as fact for project management.
In the presentation software, in every case we were asked by the executives to write specific things, such as a PowerPoint plugin to display in-meeting poll questions and results, or to display traffic data on broadcast TV, or to render other data, and so on. Everything was sorted based on funding and ad sales. All that mattered were how many eyeballs were watching the screen and maximizing that number to get commercials. In no case did we follow those business practices that you stated as fact.
For all the business software in all industries I've been in, the driving forces were business need and financial statements. There is very little involved in "requirements prioritization" at the project level. They tended to be "We are falling behind and need this immediately to stay competitive" as specified by the highest level executives, or "We already made the decision based on profitability, now you need to make it work."
Similarly in games development was everything based on money. Design is based on estimated sales figures. Production is based on estimated sales figures. Developers base everything on the budget they are given. Marketing is based on estimated sales figures.
For games software, there are pitches created by designers and passed to studio executives. The executives review it based on profitability of comparable products. Then a demo is made. The executives review it again, and base it on the profitability of comparable products, legality, marketability, and so on. If it seems like a profitable idea, the product enters full design, reviewed periodically for profitability vs risk, hopefully not cancelled, and goes through the rest of production and post-production and marketing and eventual sales cycle, all based on profitability. During implementation, we follow scrum practices where features are implemented in order of what is required to build the game, not based on the values you cited as being important. Instead it was based on features like spawning objects, inventory systems, creation of core objects, core UI, core rendering, core networking. Later on were the creation of clones of objects, 'nice to have' effects, 'nice to have' networking features, and so on. The first round is based on what is required to make a game. The second round was filling up the content until time ran out. Those were sorted based on complexity and risk, not so much on how creative the item was.
Nowhere in the corporate process of making games is the "creative/entertainment value" a consideration. During production some managers will use it to help prioritize the lesser object creation order, but again that is a minor side consideration fairly late in the cycle, and only used as a minor detail.
Frob, you don't seem to be taking into account the fact that a game design is not monolithic. A game designer may present a design to an executive or group of executives, but there is pretty much 0% chance that the final produced game will look identical to the design. And the first batch of changes is going to be an evaluation of whether anything in the design is going to be more expensive than it's worth to implement. Making a movie is really a better comparison to the process of making a game than making productivity software is. (Or at least you should be comparing to a BIG piece of productivity software with lots of features like Photoshop or Microsoft Office.) The original screenplay for a movie may be purchased by a studio, but will then have many changes made to it by several different layers of people in the process of production, down to actors ad-libbing their lines and whole scenes possibly ending up on the cutting room floor.
Yes the overall goal is to make the most money and secondarily to win free advertising via having pleased viewers recommend the movie to their acquaintances, as well as the possibility of winning a major award which generates a lot of free publicity and further sales. But everyone involved, from the writer and director to the actors, makeup and costume artists, stunt men, etc. has an idea of what makes a good movie, and they are trying to make a good movie. Audiences may pay for a movie based on high concept and advertising alone, but if the movie's entertainment value doesn't satisfy that audience it will negatively impact the future earning potential of the IP involved as well as the studio, director, and any actors the audience identifies as being a part of the problem. Executives may try to always look at the bottom line, but when you have a staff that includes artists and entertainers, those people's decision-making processes are going to be motivated by their urge to create art and be entertaining.
Yes the overall goal is to make the most money and secondarily to win free advertising via having pleased viewers recommend the movie to their acquaintances, as well as the possibility of winning a major award which generates a lot of free publicity and further sales. But everyone involved, from the writer and director to the actors, makeup and costume artists, stunt men, etc. has an idea of what makes a good movie, and they are trying to make a good movie. Audiences may pay for a movie based on high concept and advertising alone, but if the movie's entertainment value doesn't satisfy that audience it will negatively impact the future earning potential of the IP involved as well as the studio, director, and any actors the audience identifies as being a part of the problem. Executives may try to always look at the bottom line, but when you have a staff that includes artists and entertainers, those people's decision-making processes are going to be motivated by their urge to create art and be entertaining.
Author
frob,
The solutions, and generally the academic literature we study in Software Product Management (SPM) at Utrecht University are rarely present in regular software businesses, but I have personally observed numerous software product Dutch companies in the process of implementing SPM practices. Six months ago, I and a team of colleagues conducted several interviews as part of an SPM research with a product manager in a rapidly growing software company (their main product was a CRM system), who was trying to bring up the maturity level of his (40+ FTEs) organization. I was surprised to see that the problems he and his company had, have solutions in SPM literature, but he was totally unaware of them. Solutions that have empirical evidence for helping success rate of projects in terms of schedule predictability and quality. Similarly, the other groups in the class conducting SPM research at other software companies reported the same situation.
What I am trying to say is, I am not surprised any more that the solutions in academic literature are not more widely applied. Whether it's that they are relatively new; it's too expensive for companies to reorganize and implement them; companies don't trust the return of investment; or they just don't know about them (as was the case with the 8 companies researched in our SPM class), I honestly don't know. The truth is, there is a lot of academic literature on valid problems that is rarely paid attention to.
It is true that the research I am conducting deals with concepts (i.e. the infamous "creative/entertainment value") that are not in use, and I am very well aware of this. I also realize it might seem too idealistic and even naive to focus on the creative vision while developing a game and use it as one of the factors for requirements prioritization, but as a scientist, I am not afraid of debunking an idea. In fact, I try not to favor any side and just see what the expert would say when presented with the idea.
What I am doing with my research will most probably not apply for smaller projects (e.g. casual games). Whatever prioritization method I come up, I think will probably focus on a fraction of a project. It is a relatively small improvement I am aiming at, but that's the way academia works - in very small steps. In the process I hope I will get a good idea of the current industry practices, as I must take that into account.
The reason I got excited about it in the first place was due to the fact that games are rarely viewed as tool for dealing with problems. They are creative, they are entertaining, some might go to the length of arguing they are art. The best examples have similarities with film art. A logical continuation of thought would suggest that the creative/core vision should play a role in the development process, as it is the case with any form of art. Part of my research deals with the "What if" the creative vision is taken into account, and "How to" actually do that. The What if and How to are reflected in my survey, and the reason I asked game developers to try and answer these questions - so I can see what game developers think about it, and how they would approach the problem.
I am not arguing this is what needs to be done, nor what is actually done in the industry. I hope it is clearer why I am asking those silly questions now.
P.S. I appreciate the tone of your last reply, and the explanations you provide.
The solutions, and generally the academic literature we study in Software Product Management (SPM) at Utrecht University are rarely present in regular software businesses, but I have personally observed numerous software product Dutch companies in the process of implementing SPM practices. Six months ago, I and a team of colleagues conducted several interviews as part of an SPM research with a product manager in a rapidly growing software company (their main product was a CRM system), who was trying to bring up the maturity level of his (40+ FTEs) organization. I was surprised to see that the problems he and his company had, have solutions in SPM literature, but he was totally unaware of them. Solutions that have empirical evidence for helping success rate of projects in terms of schedule predictability and quality. Similarly, the other groups in the class conducting SPM research at other software companies reported the same situation.
What I am trying to say is, I am not surprised any more that the solutions in academic literature are not more widely applied. Whether it's that they are relatively new; it's too expensive for companies to reorganize and implement them; companies don't trust the return of investment; or they just don't know about them (as was the case with the 8 companies researched in our SPM class), I honestly don't know. The truth is, there is a lot of academic literature on valid problems that is rarely paid attention to.
It is true that the research I am conducting deals with concepts (i.e. the infamous "creative/entertainment value") that are not in use, and I am very well aware of this. I also realize it might seem too idealistic and even naive to focus on the creative vision while developing a game and use it as one of the factors for requirements prioritization, but as a scientist, I am not afraid of debunking an idea. In fact, I try not to favor any side and just see what the expert would say when presented with the idea.
What I am doing with my research will most probably not apply for smaller projects (e.g. casual games). Whatever prioritization method I come up, I think will probably focus on a fraction of a project. It is a relatively small improvement I am aiming at, but that's the way academia works - in very small steps. In the process I hope I will get a good idea of the current industry practices, as I must take that into account.
The reason I got excited about it in the first place was due to the fact that games are rarely viewed as tool for dealing with problems. They are creative, they are entertaining, some might go to the length of arguing they are art. The best examples have similarities with film art. A logical continuation of thought would suggest that the creative/core vision should play a role in the development process, as it is the case with any form of art. Part of my research deals with the "What if" the creative vision is taken into account, and "How to" actually do that. The What if and How to are reflected in my survey, and the reason I asked game developers to try and answer these questions - so I can see what game developers think about it, and how they would approach the problem.
I am not arguing this is what needs to be done, nor what is actually done in the industry. I hope it is clearer why I am asking those silly questions now.
P.S. I appreciate the tone of your last reply, and the explanations you provide.
Author
Frob, you don't seem to be taking into account the fact that a game design is not monolithic. A game designer may present a design to an executive or group of executives, but there is pretty much 0% chance that the final produced game will look identical to the design. And the first batch of changes is going to be an evaluation of whether anything in the design is going to be more expensive than it's worth to implement. Making a movie is really a better comparison to the process of making a game than making productivity software is. (Or at least you should be comparing to a BIG piece of productivity software with lots of features like Photoshop or Microsoft Office.) The original screenplay for a movie may be purchased by a studio, but will then have many changes made to it by several different layers of people in the process of production, down to actors ad-libbing their lines and whole scenes possibly ending up on the cutting room floor.
Yes the overall goal is to make the most money and secondarily to win free advertising via having pleased viewers recommend the movie to their acquaintances, as well as the possibility of winning a major award which generates a lot of free publicity and further sales. But everyone involved, from the writer and director to the actors, makeup and costume artists, stunt men, etc. has an idea of what makes a good movie, and they are trying to make a good movie. Audiences may pay for a movie based on high concept and advertising alone, but if the movie's entertainment value doesn't satisfy that audience it will negatively impact the future earning potential of the IP involved as well as the studio, director, and any actors the audience identifies as being a part of the problem. Executives may try to always look at the bottom line, but when you have a staff that includes artists and entertainers, those people's decision-making processes are going to be motivated by their urge to create art and be entertaining.
Exactly my point! Thank you
I was starting to think I am going crazy.
I'm glad if I said anything helpful.
I also wanted to comment on ApochPiQ's art gallery exercise. I think this is a great exercise, except I think the results and conclusions I would draw would be totally different. The field of demographics exists for a reason - it is to some extant predictable what people will like and not like. Me personally I could tell you with maybe 80% accuracy what art in a gallery I am going to like and not like before I go in the gallery. Maybe I have more predictable taste than most people, or maybe because I come from an art background I've spent a lot of time studying my own preferences and I have enough experience to make a guess at what kinds of things are going to be in an art gallery. But on the other hand I could make the same sort of prediction for, say, my mother with at least 50% accuracy.
Here are some of my predictions of what I would and wouldn't like:
- dislike any piece portraying a person or animal looking like they are in pain or terror, or dead
- neutral or mild dislike toward any piece which is abstract; more strong dislike for things that look like violent random scratches or clusters, more neutral for things which look like orderly geometric shapes or fractals.
- neutral to most photographs, with exceptions for the two points below this one:
- strongly like any piece portraying pretty flowers, butterflies, dragonflies, horses, deer, dogs, cats, birds, fish, all sorts of animals
- mildly like any piece portraying a person with a positive facial expression in an impressive costume (historical, futuristic, specialized use like a ballet costume or suit of armor, etc.
- strongly like pieces which use a neon/bold color palette
- strongly like pieces which use a jeweltone/pearl/pastel/metallic color palette
- mildly like pieces which use the sunlight color palette involving warm browns, soft blues, and crisp greens
- mildly like black and white, sepia tone, and partially colorized black and white pieces
- dislike pieces using the formal color palette involving tan, navy blue, gray-green, and power red
I also wanted to comment on ApochPiQ's art gallery exercise. I think this is a great exercise, except I think the results and conclusions I would draw would be totally different. The field of demographics exists for a reason - it is to some extant predictable what people will like and not like. Me personally I could tell you with maybe 80% accuracy what art in a gallery I am going to like and not like before I go in the gallery. Maybe I have more predictable taste than most people, or maybe because I come from an art background I've spent a lot of time studying my own preferences and I have enough experience to make a guess at what kinds of things are going to be in an art gallery. But on the other hand I could make the same sort of prediction for, say, my mother with at least 50% accuracy.
Here are some of my predictions of what I would and wouldn't like:
- dislike any piece portraying a person or animal looking like they are in pain or terror, or dead
- neutral or mild dislike toward any piece which is abstract; more strong dislike for things that look like violent random scratches or clusters, more neutral for things which look like orderly geometric shapes or fractals.
- neutral to most photographs, with exceptions for the two points below this one:
- strongly like any piece portraying pretty flowers, butterflies, dragonflies, horses, deer, dogs, cats, birds, fish, all sorts of animals
- mildly like any piece portraying a person with a positive facial expression in an impressive costume (historical, futuristic, specialized use like a ballet costume or suit of armor, etc.
- strongly like pieces which use a neon/bold color palette
- strongly like pieces which use a jeweltone/pearl/pastel/metallic color palette
- mildly like pieces which use the sunlight color palette involving warm browns, soft blues, and crisp greens
- mildly like black and white, sepia tone, and partially colorized black and white pieces
- dislike pieces using the formal color palette involving tan, navy blue, gray-green, and power red
The important part of the art exercise has nothing to do with what kind of art you like, or whether or not you can personally assign scores to something.
The point was in the comparison to other people's opinions, and even the comparison between your immediate reaction to a piece, your short-term memory of that piece, and your lasting impressions of the same piece. If you personally don't think that people would have varying scores at those three time points, I would argue that you have either uncommonly flat taste, or uncommonly acute perception of artistic merit. I would also argue that you cannot generalize from self in this instance; that was, again, part of the point of the exercise. Just because you will react to art in one way does not in any way suggest that you can conclude what any arbitrary other person would think of the same exact work.
For instance, I find pastels to be irritating, and prefer grayscale photography and muted colors over anything with highly vibrant blues or greens. The point of the art gallery exercise is not that I can tell you those things a priori of walking into the gallery; the point is that (1) I may find a piece that does not fit my preconceived ideas of what I "like" and yet still appeals to me; (2) I may change my mind about a piece upon reflection; and (3) I'm going to have wildly different numbers than you do.
As for software practices: there are two kinds of software companies. One kind is rigidly attached to process, often informed by academic research, and spends tremendous amounts of money and time on things like certifications, process compliance, and so on.
The other kind is not a hellhole.
(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the people. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)
The point was in the comparison to other people's opinions, and even the comparison between your immediate reaction to a piece, your short-term memory of that piece, and your lasting impressions of the same piece. If you personally don't think that people would have varying scores at those three time points, I would argue that you have either uncommonly flat taste, or uncommonly acute perception of artistic merit. I would also argue that you cannot generalize from self in this instance; that was, again, part of the point of the exercise. Just because you will react to art in one way does not in any way suggest that you can conclude what any arbitrary other person would think of the same exact work.
For instance, I find pastels to be irritating, and prefer grayscale photography and muted colors over anything with highly vibrant blues or greens. The point of the art gallery exercise is not that I can tell you those things a priori of walking into the gallery; the point is that (1) I may find a piece that does not fit my preconceived ideas of what I "like" and yet still appeals to me; (2) I may change my mind about a piece upon reflection; and (3) I'm going to have wildly different numbers than you do.
As for software practices: there are two kinds of software companies. One kind is rigidly attached to process, often informed by academic research, and spends tremendous amounts of money and time on things like certifications, process compliance, and so on.
The other kind is not a hellhole.
(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the people. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)
Author
(I kid. But only partially - being strangled by process and procedure is a surefire way to kill off creative drive and innovation. I think that's probably the root of the incompatibility that is showing up between process-oriented mentalities and game development in particular; it has nothing to do with the software, and everything to do with the people. Creative, driven, interesting people do not like to work in process-heavy environments, generally speaking. And people who are content to live within the confines of ivory-tower process constraints are not creative or interesting, generally speaking.)
Generally speaking, I think you are quite right. However, at a certain point, one needs to choose the lesser evil - failing to deliver under the pressure of overwhelmingly complex projects, or bow to the creativity-repelling processes and hope the initial spark that started it all doesn't die. I thought I'd give it a go and try and fix one of those processes so it takes care of creativity and vision, instead of killing them.
That's the thing - we already have great ways of delivering software without killing creativity. Hundreds of games ship every year using those techniques.
Why are those sequences of procedure less valid than your concept of a "process"?
Why are those sequences of procedure less valid than your concept of a "process"?
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
