• entries
    18
  • comments
    69
  • views
    552433

Week Of Awesome III - Postmortem

Sign in to follow this  

759 views

This is my postmortem for the WoA III. I have been mulling over what many of the contestants have said, what has been discussed. I tried to reflect more on how to improve things for next year. The one word i've seen thrown around quite a bit is "disappointment", not in the contest itself, but each competitor's feelings toward what they created, and where they placed. This is not a word I want associated with this contest. Everyone that competed should feel a sense of accomplishment for what they've done is no small task, however it seems to be the biggest thing i've noticed in nearly everyone's initial reactions. This I think is caused by the nature of what this week is. It's a competition, and as such placing poorly can be disheartening, I understand this very much. It makes me wonder if I should bother ranking any entrys at all(and to be honest it is something I'm going to give alot of thought to for next year). I want to reward the truly excellent entries, but not make others feel like what they worked on was for naught. I think another thing that might have been a problem is an overly large prize pool. yes it was nice being able to hand out such large prizes, but I don't like that for anyone that didn't place, they could only see disappointment from the results(this of course isn't true for everyone, but for most it seems to be the common theme). In this regard I think I will institute a monetary cap to prizes, so that it does not become so much about winning.

Secondly the theme for this year, I felt it was much better than last years, but it's nowhere near perfect. One of two major complaints were seen. First off there was the relatively obvious "use your death to advance", which many contestants seemed to want to avoid using, and I do think many succeeded in that regard. I do believe that when looked into deeply their are a number of possible interpretations, but overall it was still kind of restricting. So with this in mind, I have decided that next year multiple themes will be presented, and you well be tasked with incorporating at least 2 of the themes into your game(likely a total of 4 themes will be presented). This will also most likely mean more abstract theme concepts, but we'll see what I can collect for next years.

On the topic of judging itself, I feel some work is to be done here. Eck himself created a great scale, and I attempted to make it as a guideline for each judge to use, but I don't believe each judge followed it adequately. Secondly I also put forth some basic guidelines for how a judge is to treat games. However still I think this requires more refining to get right. Judges are still going to have their individual takes on each game, but I'd like to narrow the margin between how widely different a game is scored by two different judges. Perhaps I need to take a firmer role when I see how each judge is addressing individual games, If i see a score that is vastly different from how the rest of the judges have scored a game I should perhaps ask for the judge to reevaluate the score. Or alternatively perhaps i should only average the 3 highest scores. I plan to give this thought on how I might improve the way judging is done.

Next, I had hoped by adding a reward for reviewers, it would have given more incentive to others to give feedback and review games. Unfortunately there wasn't any real strong uptake in what was seen in terms of feedback. I'm not sure if i will keep this reward in for the future, and was a bit surprised to see so few actually partake in it. I'm really not sure what to do to encourage others to share there feedback.

I have much to think about, and I doubt i'll get it all right next year either, but hopefully next year well be an improvement over this years. Thank you for everyone that competed, supported, and helped this year! Till next year's contest!
Sign in to follow this  


8 Comments


Recommended Comments

Averaging the three highest scores is a good idea, but I dislike the idea of asking a judge to "reevaluate" a score.

 

To encourage more feedback to participants what if, after the contest ends, each entrant is given one game to review (which does not contribute to the score) and is required to provide a full paragraph or more of feedback?

 

Also, the judges were told to "provide a sentence or two" of feedback - I think that might have been taken too literally. If each judge provided "a paragraph" worth of feedback, or "a sentence or two per category", it would've helped alot - but maybe that would've added too much work for the judges. I did I think about two or three paragraphs per game, and it was pretty tiring.

 

My reviews were overly negative, which is weird, considering I'm normally encouraging and very positive; I tended to focus on what could've been improved instead of highlighting what did go well, giving critical feedback for if people want to improve their skills - but people aren't participating in the contest to improve their skills, they are participating for enjoyment.

Share this comment


Link to comment

Averaging the three highest scores is a good idea, but I dislike the idea of asking a judge to "reevaluate" a score.

 

To encourage more feedback to participants what if, after the contest ends, each entrant is given one game to review (which does not contribute to the score) and is required to provide a full paragraph or more of feedback?

 

Also, the judges were told to "provide a sentence or two" of feedback - I think that might have been taken too literally. If each judge provided "a paragraph" worth of feedback, or "a sentence or two per category", it would've helped alot - but maybe that would've added too much work for the judges. I did I think about two or three paragraphs per game, and it was pretty tiring.

 

My reviews were overly negative, which is weird, considering I'm normally encouraging and very positive; I tended to focus on what could've been improved instead of highlighting what did go well, giving critical feedback for if people want to improve their skills - but people aren't participating in the contest to improve their skills, they are participating for enjoyment.

 

I dont fully agree with this either. Some people enjoy just doing the competition for sure, but some people also use this competition to practice skills that they would not normally use. One of the things I love best about the competition is how it drives our community to work together with new an interesting people. I would have never met Bytroll, Shadow, and a few others if it had not been for the competition. These relationships have been lasting and very rewarding in many ways. So, I dont think it is just about going out to have fun.

 

With that said, they teach us in the military leadership courses that you must always find time to praise your personnel. If all they ever hear is negative it brings down the environment and fosters hostility. So, it cant hurt to throw in a few positive things as well.

 

I do, however, feel that all judges should put some effort into their writeup about their reviews. It might be exhausting but is crucial for the person(s) who made the game. If judges dont have time to do this than perhaps they shouldn't be a judge :P.

 

Like I said, I thought you guys did great this year and it was certainly better than last year! We had loads of fun and produced something that is commercially viable. As stormy said, in a span of 7 days that in itself is a huge thing. I say we keep building on what we got and we can end up with something really great.

Share this comment


Link to comment

I'd like to narrow the margin between how widely different a game is scored by two different judges. Perhaps I need to take a firmer role when I see how each judge is addressing individual games, If i see a score that is vastly different from how the rest of the judges have scored a game I should perhaps ask for the judge to reevaluate the score. Or alternatively perhaps i should only average the 3 highest scores. I plan to give this thought on how I might improve the way judging is done.

It's an interesting dilemma.

I don't entirely agree with a goal to try and standardise the judging so that two judges would rate the same game with the same score. Personally I think a judge should be free to judge with whatever slant they want as long as it is applied fairly across the board - so a particular judge might have a critical eye for artwork, while another might tend to high-standards for gameplay, etc. Otherwise, you could argue that if all judges are marking to the same criteria then there is little benefit in having an entry judged by more than one judge.

Arriving at an overall score is a different problem however. If the judges are using different criteria then a simple average will produce a fairly meaningless result and contestants might feel their score was dragged down if a single judge gave them a lower score in a particular category.

Averaging only the highest 3 scores might be one way to mitigate that, the problem with that approach would be if there was a judge who routinely issued scores that are lower than the other judges then this judge's score will never be factored in and he/she might as well not be a judge at all!

Another possibility is to first normalise the results from each judge in some fashion, for example they could be linearly mapped to another range (i.e. lowest score is mapped to 0 and the highest score is mapped to, say, 10). With the judges scores normalised you can now average them to produce a combined, normalised score.

How you combine scores across categories to produce an overall rank is yet another problem!
One thing you might consider is to not combine scores across categories at all and instead split the prize pools across the categories, so that 'Best in Graphics' will win something and so will 'Best in Audio', etc.

Food for thought :-)

Share this comment


Link to comment
Firstly I think as the others have said you are being harsh on yourself people will be disappointed in a comp if they dont finish 1st but from what i got from most of the blogs after they were happy with what they achieved. I like the idea of multiple themes but is this going to hurt someone if they choose only one of the four? As dmatter suggested maybe have some best of categories? All in all i am sure people had a good time with it

Share this comment


Link to comment

Arriving at an overall score is a different problem however. If the judges are using different criteria then a simple average will produce a fairly meaningless result and contestants might feel their score was dragged down if a single judge gave them a lower score in a particular category.


Averaging only the highest 3 scores might be one way to mitigate that, the problem with that approach would be if there was a judge who routinely issued scores that are lower than the other judges then this judge's score will never be factored in and he/she might as well not be a judge at all!

Another possibility is to first normalise the results from each judge in some fashion, for example they could be linearly mapped to another range (i.e. lowest score is mapped to 0 and the highest score is mapped to, say, 10). With the judges scores normalised you can now average them to produce a combined, normalised score.

 

One thing to keep in mind is that each judge only judges half the games. Even if all judges stick to their personal criteria for scoring, if one consistently gives higher scores, an entry is lucky to get to them instead of the judge that gives consistently lower scores. I think a judging guideline like Eck provided would help with this, without forcing judges to give scores similar to each other.

Share this comment


Link to comment

Yay! I helped! :)

 

@Stormynature - Very well said. 

 

When people say they're disappointed that could mean a couple of things. They could be disappointed in the judging process, but they could also mean they're disappointed in themselves. So, I wouldn't take it personally slicer4ever. I think this year's contest was even better than last year's. Good job!

 

If you want to find out more, you can always ask the disappointed people for more information. "Why were you disappointed? Which of your scores seems way out of whack? Which game do you think you should have beaten?"

 

I wrote the criteria to try and help judges give a consistent scoring. I think the guidelines make the scores a little closer to each other, but more importantly, I hopped to help make judge's scores self-consistent. If one judge is more critical of graphics, it doesn't really matter so long as he's assigning points based on the scale as he interprets it. As DifferentName points out though, this would only be true if the judges tested ALL the games. Next year, that might not be possible.

 

I did test all the games and I had a really tough time. After I made it about half-way through the entries, I had difficulty remembering which games corresponded to the scores I had given. I wound up having to re-fire up the games and think about which one was more fun, which one's graphics were better, etc. And I struggled with trying to be fair.

 

We can talk about different scoring systems, but we're never going to please everyone. If you're interested, I recommend starting a thread in the lounge so people can offer up ideas.

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now