This is my postmortem for the WoA III. I have been mulling over what many of the contestants have said, what has been discussed. I tried to reflect more on how to improve things for next year. The one word i've seen thrown around quite a bit is "disappointment", not in the contest itself, but each competitor's feelings toward what they created, and where they placed. This is not a word I want associated with this contest. Everyone that competed should feel a sense of accomplishment for what they've done is no small task, however it seems to be the biggest thing i've noticed in nearly everyone's initial reactions. This I think is caused by the nature of what this week is. It's a competition, and as such placing poorly can be disheartening, I understand this very much. It makes me wonder if I should bother ranking any entrys at all(and to be honest it is something I'm going to give alot of thought to for next year). I want to reward the truly excellent entries, but not make others feel like what they worked on was for naught. I think another thing that might have been a problem is an overly large prize pool. yes it was nice being able to hand out such large prizes, but I don't like that for anyone that didn't place, they could only see disappointment from the results(this of course isn't true for everyone, but for most it seems to be the common theme). In this regard I think I will institute a monetary cap to prizes, so that it does not become so much about winning.

I am not sure I fully agree. Part of what makes the competition rewarding and fun is the chance of success or failure. Try not to beat yourself up when you read the word "disappointed" because this is in part what makes a person try harder next year. Competition is crucial for development and growth and by removing the win/lose aspect you remove that crucial element of participation.

I just want to make sure you know that what you are doing is great. I enjoy these competitions and the people I have spoken with are happy this community has them. Some of the other online ones are so large you get lost in the games... this is still small enough it is manageable and fun to participate in. Keep up the great work and keep working to refine the process! <3

You are being overly harsh upon yourself. The number of entrants was higher, the number of successful entries was higher, the prize pool was larger, the number journal entries (participation) was higher, etc etc and so on. I did not see anywhere someone saying "I wish I had never gone into the competition" or similar.

Was there disappointment? Sure, it is to be expected in any competition where you invest time, effort and other things. One always hopes to succeed and taste of disappointment will always accompany not walking away with the 1st place. The outcome of this competition has led to a couple of entries being explored for future development into commercially viable products, this is most definitively demonstrated by the impact that Try, Die, Repeat has already had with multiple youtube "lets play" video and overwhelmingly positive feedback which in the context of a simple 7 day competition entry is big.

This year is only really the second time that the competition has been operated as a deliberately orchestrated event. It is, by all measures, a greater success than last years and I see no reason why next years won't be even better. The only thing to be wary of is making so many changes that it becomes something no one wants to be a part of.

I throw money in as a sponsor to this competition as I see it as an important social feature of Gamedev.net, People talking with each other, joining together in teams and basically having a great time, not to mention the possible financial benefits that might arise from winning or releasing a game, the increase in skill levels etc are all things which strengthen the community. We as a community need such things as they effectively are what will help foster growth on the site and help maintain it's viability into the future. I could keep going on for ages but the reality is simple: You were overly harsh in your criticism on yourself! :P

Averaging the three highest scores is a good idea, but I dislike the idea of asking a judge to "reevaluate" a score.

To encourage more feedback to participants what if, after the contest ends, each entrant is given one game to review (which does not contribute to the score) and is required to provide a full paragraph or more of feedback?

Also, the judges were told to "provide a sentence or two" of feedback - I think that might have been taken too literally. If each judge provided "a paragraph" worth of feedback, or "a sentence or two per category", it would've helped alot - but maybe that would've added too much work for the judges. I did I think about two or three paragraphs per game, and it was pretty tiring.

My reviews were overly negative, which is weird, considering I'm normally encouraging and very positive; I tended to focus on what could've been improved instead of highlighting what did go well, giving critical feedback for if people want to improve their skills - but people aren't participating in the contest to improve their skills, they are participating for enjoyment.

Averaging the three highest scores is a good idea, but I dislike the idea of asking a judge to "reevaluate" a score.

To encourage more feedback to participants what if, after the contest ends, each entrant is given one game to review (which does not contribute to the score) and is required to provide a full paragraph or more of feedback?

Also, the judges were told to "provide a sentence or two" of feedback - I think that might have been taken too literally. If each judge provided "a paragraph" worth of feedback, or "a sentence or two per category", it would've helped alot - but maybe that would've added too much work for the judges. I did I think about two or three paragraphs per game, and it was pretty tiring.

My reviews were overly negative, which is weird, considering I'm normally encouraging and very positive; I tended to focus on what could've been improved instead of highlighting what did go well, giving critical feedback for if people want to improve their skills - but people aren't participating in the contest to improve their skills, they are participating for enjoyment.

I dont fully agree with this either. Some people enjoy just doing the competition for sure, but some people also use this competition to practice skills that they would not normally use. One of the things I love best about the competition is how it drives our community to work together with new an interesting people. I would have never met Bytroll, Shadow, and a few others if it had not been for the competition. These relationships have been lasting and very rewarding in many ways. So, I dont think it is just about going out to have fun.

With that said, they teach us in the military leadership courses that you must always find time to praise your personnel. If all they ever hear is negative it brings down the environment and fosters hostility. So, it cant hurt to throw in a few positive things as well.

I do, however, feel that all judges should put some effort into their writeup about their reviews. It might be exhausting but is crucial for the person(s) who made the game. If judges dont have time to do this than perhaps they shouldn't be a judge :P.

Like I said, I thought you guys did great this year and it was certainly better than last year! We had loads of fun and produced something that is commercially viable. As stormy said, in a span of 7 days that in itself is a huge thing. I say we keep building on what we got and we can end up with something really great.

I'd like to narrow the margin between how widely different a game is scored by two different judges. Perhaps I need to take a firmer role when I see how each judge is addressing individual games, If i see a score that is vastly different from how the rest of the judges have scored a game I should perhaps ask for the judge to reevaluate the score. Or alternatively perhaps i should only average the 3 highest scores. I plan to give this thought on how I might improve the way judging is done.

Arriving at an overall score is a different problem however. If the judges are using different criteria then a simple average will produce a fairly meaningless result and contestants might feel their score was dragged down if a single judge gave them a lower score in a particular category.

Averaging only the highest 3 scores might be one way to mitigate that, the problem with that approach would be if there was a judge who routinely issued scores that are lower than the other judges then this judge's score will never be factored in and he/she might as well not be a judge at all!

Another possibility is to first normalise the results from each judge in some fashion, for example they could be linearly mapped to another range (i.e. lowest score is mapped to 0 and the highest score is mapped to, say, 10). With the judges scores normalised you can now average them to produce a combined, normalised score.

One thing to keep in mind is that each judge only judges half the games. Even if all judges stick to their personal criteria for scoring, if one consistently gives higher scores, an entry is lucky to get to them instead of the judge that gives consistently lower scores. I think a judging guideline like Eck provided would help with this, without forcing judges to give scores similar to each other.

Yay! I helped! :)

@Stormynature - Very well said.

When people say they're disappointed that could mean a couple of things. They could be disappointed in the judging process, but they could also mean they're disappointed in themselves. So, I wouldn't take it personally slicer4ever. I think this year's contest was even better than last year's. Good job!

If you want to find out more, you can always ask the disappointed people for more information. "Why were you disappointed? Which of your scores seems way out of whack? Which game do you think you should have beaten?"

I wrote the criteria to try and help judges give a consistent scoring. I think the guidelines make the scores a little closer to each other, but more importantly, I hopped to help make judge's scores self-consistent. If one judge is more critical of graphics, it doesn't really matter so long as he's assigning points based on the scale as he interprets it. As DifferentName points out though, this would only be true if the judges tested ALL the games. Next year, that might not be possible.

I did test all the games and I had a really tough time. After I made it about half-way through the entries, I had difficulty remembering which games corresponded to the scores I had given. I wound up having to re-fire up the games and think about which one was more fun, which one's graphics were better, etc. And I struggled with trying to be fair.

We can talk about different scoring systems, but we're never going to please everyone. If you're interested, I recommend starting a thread in the lounge so people can offer up ideas.