• entries
    383
  • comments
    1075
  • views
    352189

Almost judging time

Sign in to follow this  
Trapper Zoid

61 views

As of now I've received emails for two MAGIC games, so it is now officially a "contest". I haven't played more than a couple of seconds of each one to make sure that they work so I can review them all on an equally fresh basis, but I'm looking forward to playing them all tomorrow.

For me tomorrow is the hard part of the contest; where I have to rank all the games to see who is the winner. While I mostly think everyone is a winner for entering the contest (only mostly because I have an urge to kick my own arse for writing something so trite), I do have to do the judge thing for this to be a contest. I had a scoring scheme already worked out with Prozak to enable us to come to a consensus on which game we thought was best, but now since I'm the only judge things have got a fair bit simpler. I'll still use the same scheme since I put a fair bit of effort in devising it, but it means the final scores will be arrived at much quicker without the need for consultation between the judges.

The only issue I'm yet to make up my mind on is how to return the results. My judging scheme involves giving each game a numeric score to help aid communication between judges as to their opinion on the game (back then I wasn't sure how many judges I'd have). So when I release the results, do I just list all the entries in rank order, first to last, or do I also release their final score?

The reason I'm undecided on this is that the scores themselves have absolutely no purpose other than as a metric for ranking games. And unfortunately the laughable excuse for game critics in most publications (on-line and off-line) have re-enforced the "high school English paper" mindset of grading, where games get at least 50% just for running on the target platform. I find it annoying that a film critic would give a good, well-crafted fun film three stars out of five, whereas a game critic gives the game equivalent something like 92%. So there's a danger that people will misconstrue the purpose of the scoring and start making assumptions that just aren't true; for example, I see nothing wrong with giving the average game in a contest a score like 2/10 if it does the job of ranking games in order.

So if the scores are essentially just a ranking mechanism, why even consider releasing them? Well, as game programmers with INT stats of 16 and above I guess we all just like allocating mostly meaningless numbers to things. And I suspect people might be interested in exactly how their game went.

There's also the option of releasing exactly how I derived the scores, but I've been advised by other contest organisers that that might lead to more problems than it's worth. There's always a subjective element with grading something creative, so a lot of the scoring categories have very vague fuzzy boundaries where a judge can use their discretion. I've had enough experience arguing with students on projects on why they should have got 3.5 marks instead of 3 in category D to know the potential headaches with going with a complete score breakdown.

In any case, if I do decide to release scores you can be sure that they won't be out of something as conventional as 10 or 100 where there's an automatic assumption of a "passing grade". I might as well scale them all to be out of 1337.

Any opinion on whether releasing the final scores is a good or bad idea? As I'm teetering between the the two right now your arguments could sway my opinion.
Sign in to follow this  


3 Comments


Recommended Comments

I'd really like to see some scoring and rationales behind the games (and said scores). One of the big irks that I had about the contest I entered Admiral Overalls into was that the contest ended with a "this game won, and you guys all lost" with no idea of how I did versus the other entries, or what the judges thought about it. Numbers may be meaningless overall, but they feel good for the developer. [smile]

By the way, I just want to let you know that I've been really impressed with how you've managed the contest so far. I'd really like to be involved in your next one, be it as a contestant or the administrative side, so give me a poke some time if you want to brainstorm about something in the future. I'm looking really forward to playing all of the entries!

Share this comment


Link to comment
Quote:
Original post by HopeDagger
I'd really like to see some scoring and rationales behind the games (and said scores). One of the big irks that I had about the contest I entered Admiral Overalls into was that the contest ended with a "this game won, and you guys all lost" with no idea of how I did versus the other entries, or what the judges thought about it. Numbers may be meaningless overall, but they feel good for the developer. [smile]

By the way, I just want to let you know that I've been really impressed with how you've managed the contest so far. I'd really like to be involved in your next one, be it as a contestant or the administrative side, so give me a poke some time if you want to brainstorm about something in the future. I'm looking really forward to playing all of the entries!

I should have added in all that diatribe I posted above that I will definitely be adding comments for each and every one of the entries explaining my opinions. I agree that the very least every contestant deserves for their efforts is good feedback from the judge. I'll be posting the reasoning behind my scores, it's just the numbers themselves I'm unsure about posting.

I suppose I'm just concerned about the whole subjective nature of grading creativity, as well as my tendency to give much lower scores than other judges. My reasoning is that since the scores aren't being used as a pass grade I might as well use the whole number spectrum, so I often give quite reasonble entries something like a four out of ten purely so I can use higher numbers for anything more outstanding.

But the more I think about it, the more I don't see any real reason not to post the whole thing, including the breakdown (even if one of the categories is purely based on the judge's "vibe" [smile]). Contestants will want to know how they did in the categories, it'll be good to review the scoring scheme itself for any future contests, and I'm seasoned enough in judging to ignore any haggling over grades. I might just relax my own restriction on the maximum in any category so the points are all just relative and no-one thinks their getting graded out of some amount, i.e. no scores like 70 out of 100, just 70; plus that way I can give out 110s if something truly amazing comes along.

As the for the next contest, I'll have to do a post-mortem on this one to see when the next will be. Given I'm planning on making a whole bunch of really small games myself over the next few months I'm keen on doing another in the not-too-distant future. The only thing holding me back is the 4E6 contest; I figure many people will be caught up in that over the next half year so I don't know if enough people will be interested.

Share this comment


Link to comment
Hey congrats on pulling all this together, I've been following the events even though I didn't have time to compete (someone mentioned he/she is having their finals right now in the UK - same here in Portugal).

I just wanted to give you my 2 cents about the whole grading mechanism. IMO it wouldn't hurt if you just ranked them from #1 to #last, and add comments for each one. However I can see how a sort of a grade would be better, since each contestant would know how far or close he is to the next or previous entry. You mentioned scaling up to 1337, that would be nice :P One idea would be to scale the grades so that the last one would always be 1337/2, and the first 1337. I think it pretty much covers all the issues that you have mentioned...

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now