GDNet V5 Concepts: User Ratings

posted in Continuous Refinement

Published December 22, 2009

One of the questions from a previous entry was what's happening to user ratings in V5. I don't have funky screenshots to show you this time, but I'll talk about what the plan is.

The present system

The present user rating system, visible under every post as a number, was created to solve a set of problems:

How do users distinguish the people that should be listened to from the people that shouldn't?

How do we identify users who are contributing to the site and community?

How do we identify users who are detracting from the site and community?

These problems were all solvable, but they required a lot of time investment and effort. We wanted to shift away from solutions that relied on users and moderators spending lots of time watching site activity. The solution was to seek to recruit the entire userbase to help solve the problem, by giving everybody a means to indicate who should and shouldn't be listened to. That, in turn, needed some kind of balancing to determine which people were good judges, which is why higher-rated users have a larger effect on the ratings of others than lower-rated users.

It's true that in general, the rating system has worked. The top-rated users are, pretty much uniformly, good contributors to the site. The lowest-rated users are generally incoherent, in(s)ane, and unwanted - though I think that exceptions exist. And users do pay some attention to the ratings of those they read, though only around 1% of registered user accounts actually filter out posts with ratings below a given threshold.

We do definitely see some undesirable behaviour. For example:

People getting upset about their rating dropping a few points and posting threads about it. This wouldn't happen if people were less sensitive, of course, but we have to face the fact that they are this sensitive. It doesn't help that there's not much one can tell those people except "be nicer."

Bandwagoning - people voting somebody down partly because they've got a low rating, and That's What This Thread Is All About Anyway. Group dynamics can be bizarre at times.

People who are great technical contributors, ending up with low(er) ratings because they got a bit ranty in the Lounge, and therefore start to be ignored in technical discussions.

Similarly, people who are really funny in Lounge threads get high ratings, and then when posting in technical threads perhaps get given more authority and credit than they're due.

People who get low ratings can have trouble recovering that rating, partly because people aren't inclined to vote low-rated users up, and if the filters are in play then their posts won't even get seen. This usually leads to the low-rated poster either creating a new account (which is a policy violation) or just leaving the site altogether. Sometimes they'll stay and just not care about their rating, but whether or not they care doesn't change the fact that we then have a user who is making positive contributions but has a low rating.

At the heart of the current rating system's design rests a few fundamental assumptions. Firstly, it assumes that if a user is good in any one way recognised by the community, then they're good in all ways - or at least are smart enough to disclaim themselves in areas where they're not good. Secondly, it assumes that users will fully consider a user and the contributions they've made to the site as a whole before rationally rating them. Thirdly, it assumes that users have good ideas about how to respond to changes in their rating - that they don't just keep doing exactly what they've been doing (albeit with an added air of bafflement and indignation) expecting a different result.

It also contributes to a bad philosophical assumption on the part of the user, and that is: that something is right because a particular person said so. Smart users won't read the ratings in this way; but some users will, when given two answers to their question, pick the answer from the higher-rated user because the user is higher-rated rather than because the answer is better.

None of these assumptions are good. They're true enough of the time that we can point to some corroborating accounts and say, "look, the system works!" but that doesn't tell us whether the system works as well as it could do.

I'm the highest-rated user on this site, so it's not something I consider lightly [grin] but in V5 I'm planning to replace the present rating system with an approach that is less susceptible - albeit not totally immune - to the above problems.

The V5 Rating Strategy

Tagging

The first problem I set out to solve was this: How do we make the rating better convey the ways or areas in which a person is good?

The solution to this one seemed fairly obvious. A mechanism by which users can express their support of a person in arbitrary, user-defined categories? Sounds like a job for tagging to me! By letting users tag users as another kind of site content, we go from having a single rating axis, to as many axes as you want - be they subject-area tags like 'Python' or 'object oriented,' or style tags like 'funny' or 'friendly.' Reconciling the different ways users tag content is already something the tagging engine has to do.

Immediately this also defeats the assumption that 'good in one area == good in all areas.' It becomes very easy to identify when a user is participating in something that matches their tags - i.e. when they're talking about what they're good at.

Thanks

How do we defeat the second assumption - that users will think long and hard before selecting tags for a user? In reality, people don't do that - they read one post, have a strong reaction to it, and then rate accordingly; they don't go "well, this post is obnoxious, but maybe the guy's just having a bad day. I'll check out his other stuff to be sure." If we embrace the strong-reaction-to-a-single-post idea instead of denying it, what we get is: Let people express that reaction with a single click, and then aggregate those reactions to get a feeling for where the user is most well received.

The way this'll be implemented will be via a 'thanks' button on every content item that a user can contribute to. It lets you express that strong reaction quickly. Then, over time, the posts that a user is 'thanked' for will start to contribute their tags to the user - if the user receives lots of 'thanks' in threads that are tagged 'Python performance pygame' for example, then they'll start to acquire those tags themselves. This also gives users more feedback on what they're doing right.

Will there be a 'No thanks!' button? I'm not sure, but I think probably not. If you don't like a contribution, just don't thank the author. If it's really necessary, you can still tag the author explicitly, or even report the post to a moderator.

Decay

How do we deal with the fact that a user's expertise will change over time? Maybe they were a game programming guru 10 years ago, but they've not kept up and their advice is out of date now. This is a fairly simple one, actually: have tags 'decay' over time. Tags that are still frequently applied to a user will 'refresh' and will decay more slowly than tags that aren't. This also solves the 'idiot' problem - how to handle people tagging each other as 'idiot' - because if the user stops being an idiot, the tag will fade away; and it mitigates the lack of a 'no thanks' button, because posting without receiving thanks will cause your tags to fade away.

Getting input

How do we get people to actually use this stuff? That's one of the bigger problems with tagging in general. Step one is to make things as easy to use as possible - single-click to 'thanks' a post, two clicks to get to adding more complex tags. Step two is to get users to at least tag their own stuff; users will be encouraged to 'self-assess' by tagging themselves, to tag their own threads and entries, and so on. Step three is to incentivize. Now, there's a limited amount we can do here - we're not about to start paying people to tag content. What you saw in my last post, though, was the 'badges' system in userboxes; what we can quite easily do is grant a badge to people who tagged 100 content items in the past month, or something like that.

Using the output

Lastly, how do we help users find the best possible content, instead of wasting their time with incoherent in(s)anity - without encouraging them to trust an answer just because it's from a highly rated user? This is a balancing act to be sure, because most of the time the best content is produced by the high-rated users.

The first trick here is to make the way that ratings are displayed be subtle; no more four-digit numbers on each post. Instead, we're considering things like changing the background colour of the post, or the thickness of the post border, to indicate when a user is strongly aligned (tagged the same way as) a thread. Making the display subtle in this way will still make the post stand out a little in the thread, without providing such a clear and definitive thing that people can get overexcited about.

What we will probably display clearly on a post is the number of times it's been thanked (perhaps only within the past X weeks). This makes the number that people latch onto be about individual posts, rather than about users, and that's a lot safer - posts are easier to talk about without people taking things personally.

The second trick is to use the information on a broader level to bias search results. When you're searching for content on a particular topic, the search can elevate threads that have good alignment, or that have lots of 'thanked' posts in. This is still sort of acting on this idea that that content will be right 'because a smart person said it,' but by elevating it to the per-thread level instead of the per-post level, lower-rated users will still have a good opportunity to point out when the higher-rated user isn't making sense.

You'll notice I've not talked about 5-star ratings at all so far. We're still deciding exactly how they'll be integrated. The advantage that 5-star ratings offer is that they are coarse; tagging a thread with particular tokens might capture what the thread is about, but maybe you just want to convey some overall impression that the thread is awesome (or terrible), without figuring out exactly which tags would express that; they might be more applicable to, say, gallery entries. They've got their fair share of problems, of course, as comments on my previous post about the rating UI pointed out. We'll have to do some more thinking about them.

Conclusion

The new system doesn't quite solve the problems that the original rating system set out to solve. Instead, it focuses on the deeper problems of how to get the best content into your hands as quickly as possible and how to describe users; they're harder problems, naturally, but I think more worthwhile.

So, what do you think? I expect that quite a lot of people might have strong feelings about this topic [smile]

Previous Entry V5 UI: Top Navigation Bar, part 1

Next Entry Wheeee

0 likes 18 comments

Comments

dmoonfire

I think not having a negative rating is a good thing. People get obsessed with the negatives and that obsession leads to the things you mentioned. Not to mention, posting about negatives is pretty much a way of getting even more negatives. Better to just focus on the positives and let the negatives fade quickly (many people learn pretty fast if they stick in this community).

I love the idea of tags for individuals. I don't know about the aged tags. Yes, there is a point where someone's skills are not as up to date as others, but there are some skills which don't change that much. There are some new techniques, but for the most part C itself hasn't changed much. Just the libraries and tools wrapped around it. I guess it would just matter how fast those tags decay.

I like the flexibility and crowd-sourcing of this system. Yeah, it probably will get a fair amount of chatter as people assign random tags (sux0rsHard!) assuming you allow arbitrary tags. I would consider either moderating tags themselves or allowing people to ignore tags under a certain usage (only used by X different people or something like that).

December 22, 2009 10:26 AM

Aardvajk

Personally, I think the more complex the system gets, the less useful it will be. The existing system has been long argued and it always appears that the vast majority of people who complain have a clear posting history that has led to their lower rating.

I'd agree with the poster above that a negative rating is not useful. We already effectively have a negative rating system as the 1000 start is well understood. I'd rather see a single metric that began at zero (or whatever), could not go below zero and stayed as simple as the current system.

This is really just my 2c and I'm very happy we'll be starting from a clean slate with whatever new system we have.

But having perused a few other sites and seen their attempts and badge systems and the like, and being a firm believer that the simplicity and self-management of the current rating system is a big part of why GDNet remains the best set of fora for programmers on the net, this is the first proposed change to V5 that I have reservations about.

It's pretty low importance compared to the other fantastic looking changes on the horizon and I think it's fair to say that the rating system doesn't motivate any of us linked above to be helpful or contribute.

But the idea that our rating needs to reflect different areas would suggest to me that we currently have highly rated D3D experts waffling nonsense in the OGL forum, or trolling the lounge - not an issue I've ever noticed on the site.

December 22, 2009 11:53 AM

rip-off

Heresy! You would destroy our Oligarchy, and for what?

On a more serious note, I like some of the ideas. The idea that tags decay sounds interesting, and mitigates some of the reservations I had when they were originally suggested. The devil, as always, is in the details. The rate of decay needs to be high enough to allow people to recover from negative tags reasonably soon, while allowing infrequent posters to keep positive one they have earned.

It sounds like this system might weaken community moderation, which I would be against. I think that having a "no thanks" button, perhaps coupled with a more aggressive decay rate for the negative ratings, could help. I like the idea that the community can take corrective action on posts and posters that are undesired, even if they are not bad enough to warrant official moderation.

Like Aardvajk, I think that most of the charges people lay against the current rating system are unfounded, or at least exaggerated. I can't think of any reasonably high rated posters who try to weigh in on topics they know little about. I think that might be the inverse of the "unskilled and unaware of it" effect.

I wonder if you might be being a bit too subtle with the tags. Is a new user to the forums going to understand that the changes in line thickness, colour or whatever of a post means that the user's tags correlate well with the posts? I would think, like Aardvajk, that the simpler this is, the better.

That said, I like how targeted the new approach is. Being tagged with what you know is good. The current system is quite poor in this regard. Is someone at 1100 to 1200 rating on the way up - or down? How did they get those ratings? Are they having a bad week in the lounge? Hopefully the new system won't be quite so unforgiving as the current one.

December 22, 2009 06:19 PM

swiftcoder

I would tend to agree with Aardvajk, that the added complexity will be disadvantageous in the long run.

In the current system, if someone appears helpful, I give them a cookie. If someone is being a jerk, I check there last few posts and rate accordingly. Given that I can do 25 points of damage to a newbie last I checked, I don't hand out negative ratings without checking their posting history - that would be plain evil.

My point is that I see the ratings as a helpful/jerk scale, plain and simple. It doesn't particularly indicate knowledge, or skill (although most helpful posters possess both in some degree).

Once you break it into subjects, it ceases to have much meaning. In general, I don't care if Sneftel or Antheus have a +5 in 'Python' - I know that they are highly competent programmers, and their advice is invaluable regardless of language or subject area.

Decay is also completely evil here. Unless the tags are so broad as to only cover a few topics (i.e. 'programmer', 'artist', 'business/law'), then no one will be able to maintain tags in all the areas they are knowledgeable - and we have a lot of jack-of-all-trades in the top 50 posters, yourself included.

Similarly, unless one of the standard tags is 'Jerk', the new ratings won't serve very well to identify the trolls and know-it-alls who plagued the system prior to the introduction of ratings. It also won't provide them the incentive to improve their behavior that the current system does.

On a related note, I am a little worried that tagging support will fragment the membership. In the current system, everybody watches the same 'recent topics' ticker, and everybody reads (and sometimes respond) to posts in all subject areas.

One of your stated goals is to make it easier to find relevant information - but I am worried that users will start to filter the forums based on their popular tags, and thus no longer read or participate in other subject areas.

This would have two highly negative effects: a) the quality of discussions will degrade as they will lack cross-pollination of ideas from experts in other subjects, and b) the users (particularly new users) will become highly specialised, and never see the bigger picture that is game development.

December 22, 2009 07:29 PM

speciesUnknown

There is one effect which you havent mentioned, that the current rating system suffers from; Sometimes, people will "gang up" and "gang rate" a user. This generally happens in the IRC channel #gamedev, and sadly, its usually initiated by highly rated members. Human intervention is pointless, mods cannot stop users from gang rating.

Now, I'm not going to mention any names, because its usually the same small number of culprits, not the vast majority; (also, I suspect my own rating will get raped if I name and shame them) but what usually happens is as follows:
1) user says something which is "wrong"
2) one or more of our higher rated members takes offense. Either because its technically inaccurate, or because its the "wrong opinion". Microsoft sympathisers or open source fanatics are usually on the front line.
3) The others in this group then all "gang rate" the user down, and encourage others to follow.

This in part explains the "rating submarine" - and no new system which allows one to link to a user's comment or user profile and say "rate this guy down!!!" will function beyond the old one.

The solution to this problem is clear; dont allow rating of individual users, instead, allow rating of their posts. Tagging posts instead of rating entire users is definatly the solution. This means that any "gang rating" requires that little bit more effort as this user now has to go around rating down all their posts.

December 23, 2009 06:49 AM

evolutional

Quote:Original post by speciesUnknown
The solution to this problem is clear; dont allow rating of individual users, instead, allow rating of their posts. Tagging posts instead of rating entire users is definatly the solution. This means that any "gang rating" requires that little bit more effort as this user now has to go around rating down all their posts.

Agreed.

I think the rating side should be a simple "Thanks/No Thanks" on posts and threads. Rating users just boosts ego and having people rocking around with huge egos isn't good on a forum that's supposed to be about helping people.

December 23, 2009 07:28 AM

Wan

Quote:Original post by speciesUnknown
There is one effect which you havent mentioned, that the current rating system suffers from; Sometimes, people will "gang up" and "gang rate" a user.

Wow. I'm starting to wonder why the IRC channel is still associated with GameDev. [headshake]

Quote:Original post by evolutional
I think the rating side should be a simple "Thanks/No Thanks" on posts and threads. Rating users just boosts ego and having people rocking around with huge egos isn't good on a forum that's supposed to be about helping people.

Agreed x2.

I'm sure the rating system helped to 'civilize' the community back in the days, but frankly I just don't see the point of it anymore. I still rate people up for being friendly or having made something cool, but it would be more effective and explicit if I'm able to show that appreciation by rating their actual post.

December 23, 2009 01:24 PM

superpig

Quote:Original post by Aardvajk
Personally, I think the more complex the system gets, the less useful it will be.

It might be complex internally, but the system as presented to users ought to be pretty simple. You can 'thanks' a forum post; you can either 5-star-rate or 'thanks' other content items, like articles or gallery entries; and if you really want, you can tag a user as being 'about' particular subjects, though you don't have to (as the system will try and deduce it from what you thank them for). Those are the only three ways you feed data in to the system. The mysterious black box of the server does some number crunching, and mysteriously you find that getting info or locating threads where you can be the most help just becomes easier and faster. That's the intention, anyway.

Quote:But having perused a few other sites and seen their attempts and badge systems and the like, and being a firm believer that the simplicity and self-management of the current rating system is a big part of why GDNet remains the best set of fora for programmers on the net, this is the first proposed change to V5 that I have reservations about.

Hmm, you think the rating system makes that big a difference? How do you think that works?

Quote:But the idea that our rating needs to reflect different areas would suggest to me that we currently have highly rated D3D experts waffling nonsense in the OGL forum, or trolling the lounge - not an issue I've ever noticed on the site.

Indeed - I don't think any totally blatant cases have ever manifested. Though I think that more subtle cases might be hard to spot.

I'll confess to an ulterior motive. Tagging users to indicate areas where they're "focused" is very useful for general-purpose matchmaking technology. You're interested in AI; they're interested in AI; they're all attending this event; maybe we should bring that event to your attention. That kind of thing.

Also, don't dismiss the incentive effect. One complaint about the current system is that a high-rated person might be very useful, or they might just be a really funny/popular lounge poster. When it's just one number, you can't tell the difference. By making it tag-based, your efforts to get involved in one particular subject area will be reflected very clearly in the tags, reducing the "popularity contest" factor of it all.

December 23, 2009 06:44 PM

superpig

Quote:Original post by rip-off
The devil, as always, is in the details. The rate of decay needs to be high enough to allow people to recover from negative tags reasonably soon, while allowing infrequent posters to keep positive one they have earned.

Indeed. Note that decay will really be based on activity rather than time, so if you're not very active, everything will decay more slowly - possibly even stopping entirely if you're away from the site for a while.

Quote:It sounds like this system might weaken community moderation, which I would be against. I think that having a "no thanks" button, perhaps coupled with a more aggressive decay rate for the negative ratings, could help. I like the idea that the community can take corrective action on posts and posters that are undesired, even if they are not bad enough to warrant official moderation.

I'm not sure that it's any weaker than it is now. I don't like the idea of just having a 'no thanks' button because ideally if you're going to criticise somebody you should leave more detailed feedback than just a [don't like this]; I'd rather people didn't have too easy a way of slagging somebody off, because in general people get defensive and upset much more often than they get irrationally exuberant.

Quote:Is a new user to the forums going to understand that the changes in line thickness, colour or whatever of a post means that the user's tags correlate well with the posts?

No, but consider: they don't need to. All the border/colour effect needs to do is draw the user's attention to the post. The idea that a well-aligned user will post the best content is an assumption that GDNet makes, not the new user.

December 23, 2009 06:53 PM

superpig

Quote:Original post by swiftcoder
In the current system, if someone appears helpful, I give them a cookie. If someone is being a jerk, I check there last few posts and rate accordingly. Given that I can do 25 points of damage to a newbie last I checked, I don't hand out negative ratings without checking their posting history - that would be plain evil.

I think, unfortunately, that you're in the minority. The majority of people who assign ratings do so based on a single post.

Quote:My point is that I see the ratings as a helpful/jerk scale, plain and simple. It doesn't particularly indicate knowledge, or skill (although most helpful posters possess both in some degree).

Once you break it into subjects, it ceases to have much meaning. In general, I don't care if Sneftel or Antheus have a +5 in 'Python' - I know that they are highly competent programmers, and their advice is invaluable regardless of language or subject area.

Sure, but if you're looking for information about particular Python language constructs, wouldn't it make sense to show you the threads with posts by the +10 Python people first? (Conversely - if you're a Python guru and you want to help people out with their Python questions, wouldn't it be helpful to be able to quickly find threads that people like Sneftel or Antheus haven't already posted in?)

Quote:Decay is also completely evil here. Unless the tags are so broad as to only cover a few topics (i.e. 'programmer', 'artist', 'business/law'), then no one will be able to maintain tags in all the areas they are knowledgeable - and we have a lot of jack-of-all-trades in the top 50 posters, yourself included.

It'll require some balancing, but one of the tricks I'm planning on employing is that the tag system will have an internal concept of 'broad' and 'narrow' tags. So, you'll be able to use tags like 'shadows' or 'shaders', and the system will gradually apply the 'graphics' tag as well.

Quote:Similarly, unless one of the standard tags is 'Jerk', the new ratings won't serve very well to identify the trolls and know-it-alls who plagued the system prior to the introduction of ratings. It also won't provide them the incentive to improve their behavior that the current system does.

One of the standard tags could be 'jerk,' sure (though I'd expect people will use 'troll' instead). You're right that the tags don't really capture the overall 'niceness' of a user; recognising how many Thanked posts is more useful there. This might be a what the 5-star-rating could be used to represent, in the end.

Quote:On a related note, I am a little worried that tagging support will fragment the membership. In the current system, everybody watches the same 'recent topics' ticker, and everybody reads (and sometimes respond) to posts in all subject areas.

Ah, but bear in mind: you'll remember the omnipresent users more easily because you see them more often. In practice, of the 33,000 users who've posted to the site (in forums other than the Lounge, CSI, or For Beginners), about 19,000 of them have only ever posted more than three posts in a single forum, and 27,000 of them have posted at least three posts in three forums or fewer.

Quote:One of your stated goals is to make it easier to find relevant information - but I am worried that users will start to filter the forums based on their popular tags, and thus no longer read or participate in other subject areas.

They're already doing this, though; either by browser bookmarks or the forum index 'favourites' feature, or simply by a usage pattern that seems them check the same few forums over and over. The users who aren't doing this are primarily using views like Active Topics to browse the site, and there will definitely be corresponding views in the new site that they can continue using ('all topics modified within the past 24 hours')

December 23, 2009 08:06 PM

swiftcoder

Quote:Original post by superpig
if you're looking for information about particular Python language constructs, wouldn't it make sense to show you the threads with posts by the +10 Python people first?

It seems to me that we don't generally encourage this sort of questions. Highly specific questions tend to either be simple queries in 'for beginners', which can be answered by someone even vaguely familiar with the subject (or better yet a google search), or a member looking for a quick fix when they don't fully understand the ramifications of their problem. Legitimate highly specialised questions mostly seem to flow from the MVPs themselves...

This is part of a broader issue, I guess. If you aren't already fairly knowledgeable in a given subject area, the chances that you will ask the right question, or even know the correct terms to ask for/tag, are pretty minimal.

Simple queries are easy enough for the OP to tag correctly, but those queries could be answered by pretty much anyone, thus negating the need for tags.

Quote:(Conversely - if you're a Python guru and you want to help people out with their Python questions, wouldn't it be helpful to be able to quickly find threads that people like Sneftel or Antheus haven't already posted in?)

I may be in the minority here as well, but i find helping to be be more of an incidental thing. I come here to learn, and discussions where the MVPs are already weighing in tend to be the most valuable.

Quote:It'll require some balancing, but one of the tricks I'm planning on employing is that the tag system will have an internal concept of 'broad' and 'narrow' tags. So, you'll be able to use tags like 'shadows' or 'shaders', and the system will gradually apply the 'graphics' tag as well.

That sounds like a very solid idea, albeit a technical nightmare to develop - better you than me [smile]

Would broad tags then slow the decay of the narrower tags they encompass?

Quote:You're right that the tags don't really capture the overall 'niceness' of a user; recognising how many Thanked posts is more useful there. This might be a what the 5-star-rating could be used to represent, in the end.

Number of thanks still doesn't provide a way to differentiate between a friendly lurker who rarely contributes, and thus is rarely thanked, and a troll, who continually posts but is rarely thanked.

It also doesn't solve the problem of being a meaningful scale - my guess is that one can pick up several hundred thanks with a single inflammatory post in the lounge, whereas it might take months to accumulate the same in general programming.

Quote:Original post by evolutional
Rating users just boosts ego and having people rocking around with huge egos isn't good on a forum that's supposed to be about helping people.

From what I have seen, ego without the knowledge to back it up tends to knock your rating pretty hard. Purely out of academic interest, are there that many over the 1200 mark whom you would consider to be overly egotistical?

I can't say I have personally paid much attention to my rating (apart from the couple of times it has dropped substantially), since I worked my way back out of the 900's.

Quote:Original post by Wan
I'm sure the rating system helped to 'civilize' the community back in the days, but frankly I just don't see the point of it anymore

The forums were pretty rough back then - I should know, I was part of the problem: at the time the ratings were introduced I was sixteen years old, arrogant as sin, and knew just enough about programming to get into trouble [smile]

The quality of information is way up since those days, as is the general tone of discussions, and I imagine the moderator workload probably reduced a bit as well. Hell, even the lounge seems civilised these days...

I am not saying it would revert overnight if ratings were removed entirely - there is probably a big enough buffer of civilised posters, but I do think that things would degrade pretty fast.

Edit: Gah - when did journal replies become threaded? Sorry I screwed up the flow of replies.

December 23, 2009 10:13 PM

Ysaneya

In my honnest opinion..

The current system may not be perfect, but you're going to replace it with a monster.

I don't have any exact point to address, it's more of an intuition. The more complex a system is, the more people will find ways to abuse it.

If I had to improve the current system, I'd simply add decay over time and maybe two categories (game development and off-topic), but that's it.

And besides the pure design aspect, won't that cause more technical problems ? Stressing the database, slowing down the server ? I'm particularly thinking of the "tags" table..

Y.

December 24, 2009 03:32 AM

Servant of the Lord

I don't see how this system is too complex. I only see two methods of player rating, one of which can be easily ignored.

Someone made a good post, I can [thank] it. Iternally, the server does its work, and checks the tags of the thread, and gives that user a (very infinitesimally small) boost on those tags.

That's pretty much it, as far as 70% of the users are concerned. For the other 30%, those who frequent the site, they might notice that this is the 7th time that user made a very good post in the OpenGL forum, and take a minute to bother tagging the poster with an "OpenGL" tag. (Which, hopefully, the internal workings of the server will convert high "OpenGL" tag counts to a "Lvl 22 OpenGL guru" badge)

Pretty simple to [thank] someone, with an optional ability to spend an extra 60 seconds to be a bit more specific in your rating.

I personally don't like [thumbs up/thumbs down] systems, primarily because on most sites it seems to be a popularity contest (in my mind), and because it feels "dumbed down", like they are treating their users like children ("okay timmy, thumbs up means 'good'"). Since GameDev deals with technical information, it could probably work well. I'd prefer it reading [helpful] instead of [thanks], though, so it'd minimize people getting [thanked] for funny posts instead of helpful posts even in technical areas of the forums.

I would like a way to negatively rate users, though. I rarely rate someone down (well, I've been gone for half a year, but before I was gone) unless they really are acting like a jerk. Would tagging someone, "Poor attitude", "Anger issues", "Cruelly sarcastic", etc... be allowed? If someone is acting like a jerk, they ought to be able to tell that everyone realizes it - maybe they don't realize it themselves.

I'd also like a easy way to tag an entire thread [helpful] or [unhelpful], to affect search results for the entire thread.

December 24, 2009 09:38 AM

Twisol

I think it's interesting to note that most of the comments here are from people with relatively high ratings. When you're already a well-established member of the community, people tend to know you better than the less active or newer members, and it's not as important to have extra information such as tags or thanks's. Likewise, if you're at the seriously low end of the rating spectrum, clearly something is wrong. But for the middle range, there's not much of a huge difference in the weight of your rating, at least in my opinion. It's like, okay, people like/dislike him, but that's all you know. At least with the far ends you have a pretty good idea of how to weigh someone's post.

I think the proposed system would go a long way towards clarifying that middle range, personally, and I think it would be quite interesting, maybe even enjoyable.

I like speciesUnknown's suggestion of 'thanking' posts rather than users, but I think that a summary of what a users' posts have received, displayed on a user's profile or something, would still be useful at a glance. I like some aspects of how StackOverflow handles things, for example.

December 24, 2009 04:39 PM

capn_midnight

I'm assuming that tagging activity will be moderated just as posting activity is. It wouldn't do well to have users tagging other users with offensive things. Will moderators be able to limit a user's tagging ability, like maybe an "on probation" user cannot create tags, only use pre-approved tags?

How successful do you think the warning-percentage/suspension/ban system has been? Will there be any changes there?

Also, I'm curious as to how one goes about testing these sorts of changes. It's one thing to test that the tools function without defect, but it's an entirely different thing to figure out if they result in desirable outcomes. Is this planned as an iterative release (we get tagging on threads one month, see how that goes, then tagging on users, etc)? Do you expect people to be able to pick this up and use it in a congenial manner, or do you expect moderator guidance being necessary?

I'm always very interested in seeing how group dynamic tools work out. At the very least, it should be fun to see what happens.

December 26, 2009 12:43 AM

MichaelT

I think this is way more complicated than it needs to be. Fading is an excellent idea though. I believe that a simple "good - evil" bar would be enough. Employ fading on that and it would achieve two things: 1. It would add a sense of humour, which is always a good thing. Then I believe the rating would be informative and less offensive. And hey, those who want to serve the darker side can finally do it. 2. The fading would be forgiving to offenders, but the darker they are the longer it would take to be forgiven.

Primarily though, by keeping it simple the rating system would be more accessible and easier to accept. I have voiced my opinion on the present system before. I am very happy to see some of those concerns finally being acknowledged. I am just sorry that it had to take such long time. I know of others (myself included) that simply stopped posting because the value of our post got overshadowed by our lack of ability to deal with people. Such as people with Aspergers etc.. The present system have been heavily penalizing to those kind of people. Some simply are rude, that doesn't invalidate the accuracy in their statements though. This change could help alleviate that, something I am very happy about. I don't mind being called words (by someone with Tourettes) I did not even know existed, as long as I get some valuable pieces of information out of it. So, even though I think the change could be more simple, I am supporting this change.

On a final OT note: Typing this on an iPhone is à Palm ( is a pain) I kept the "corrections" so you can witness the madness called "autocorrection" on that phone. Oh the horror ;)

December 26, 2009 06:39 PM

Matias Goldberg

First of all, I consider my user rating is high enough to make me biased. I worked for it, and of course, I wouldn't like to see it go away. Whether I like it or not, I am biased.

That said,
I won't deny the problems superpig talks about. It's not perfect, but it's really good.

I strongly agree with this one:

Quote:
But having perused a few other sites and seen their attempts and badge systems and the like, and being a firm believer that the simplicity and self-management of the current rating system is a big part of why GDNet remains the best set of fora for programmers on the net, this is the first proposed change to V5 that I have reservations about.

I've always believed this is one of the top reasons why GDNet is so good, and professional. The Game Industry unlike many others has the big problem that there are hordes of kids between 10-20 years old (ok, a 20 year old person isn't a kid) who think they are into game development and want to start their MMO and spam all over the forums. Bad ratings for them make them stay away; and if they persist, they're just ignored because their bad rating + the word "MMO" says it all.

I like to see the current system not as oligarchy, tyranny or democracy, but rather as meritocracy. Something very worthy in our field.

I won't deny the current system makes very hard to recover from a critically negative rating. Furthermore, users below 800 rating just keeps going down or stays frozen.

In my personal experience, negative ratings taught me a very good lesson (which doesn't just limit to forum posts in GDNet but life itself): if you don't know about something, shut up and read/learn. Also don't post just to post.
If you don't do both of those 2 things, you end up with your rating being lowered.
If you have a vague idea of something, stay quiet and read forum posts regarding the subject, once you've learned more about it, you can start writing replying to forum posts and starting threads about it.
Whenever my rating went down like 15-30 points because a staff/moderator rated me down, I was angry. But what worried me most was that I knew 99% of the time I actually deserved it. It was good to see someone landed my foot back to earth and proved I was (very) wrong.
I only missed that I couldn't take back what I said. So I learned the best way to overcome this was to not make the same mistake again.

Quote:
People who get low ratings can have trouble recovering that rating (...)
This usually leads to the low-rated poster either creating a new account (which is a policy violation) or just leaving the site altogether

Creating a new account is bad.
But leaving the site... are you sure that's a bad thing? Of course we want to appeal to a broad audience, so yes, we don't want them to leave.
But many times those who are leaving are jerks, jerks we're better without them.

Quote:
Quote:Original post by swiftcoder
In the current system, if someone appears helpful, I give them a cookie. If someone is being a jerk, I check there last few posts and rate accordingly. Given that I can do 25 points of damage to a newbie last I checked, I don't hand out negative ratings without checking their posting history - that would be plain evil.

I think, unfortunately, that you're in the minority. The majority of people who assign ratings do so based on a single post.

Well, I'm in that minority too.
But oh wait, look at swiftcoder's rating, it's really high. He has a lot of voting power.
I'm certainly not surprise that someone in the minority who makes wise, cautious choices has higher voting power. This means the current system is working.
Prudent people don't get high ratings out of thin air.

A thanks button would be cool, fading ratings over time seems good too. Perhaps instead of fading, it should be stated how old it is.
Now that I think of, I don't just see the rating, but also the date they joined. This tells me A LOT. People who has bad rating and just signed in deserves a second chance, and people who has a very high rating but is also long since here tells me he may have gotten old in expertise, but I just need to dig the forum profile if I need that detailed information.

Thumbs Up & Thanks, with statistics like in www.mercadolibre.com (spanish site, ebay probably has something similar) would be cool (in case you don't speak spanish, they show how many people Voted him total, how many positive, how many negative, and the % of each one)

Sticking tags to users is useful since that identifies their area of expertise, however that's as long as people don't start thinking "Hey, you may know a lot of Graphic programming, but you don't know nothing about AI programming, stay away and STFU"; because I definitely would like to hear the feedback from a very talented programmer no matter his area of expertise.
I find myself often writing: "OpenGL is not my strong point here (D3D guy here), but usually blah blah" or similar stuff. A tag system would help in those cases.

I'm worried the new system may be too complex so:
a) People may not understand it
b) Jerks have more ways to break it
c) Gives so much information, that it becomes useless.

Those are my 2 cents.

December 27, 2009 09:52 PM

Staffan E

Very late reply but I'll give it a shot anyway.

I really like the idea of the system proposed here. It integrates well with the broader idea of assigning tags to posts. Also the simplicity of having a Thanks button is great and it definitely is something I think people will use.

I agree with previous posters that a No Thanks button would be a bad addition, since violates the principle that the best way to deal with offensive people (this is usually why people are rated down) is to ignore them, by not pressing Thanks on their posts. If there was a No Thanks button it would immediately be subject to the Bandwagoning problem you describe.

Even more I would not have the system allow people to explicitly adding tags to users. The principle should be "rate content", not "rate people" as it is now. If the content then influenced their authors in the way described, over time the right tags would be assigned to the right users anyway. This would be the "true" situation since it would have evolved from real community activity. If it was possible to alter this explicitly then the situation would lose its credibility.

I imagine the typical user after some typical forum activity would have a big set of tags with low weights from a large number of posts on various topics. But then there would be a few tags that stand out from well appreciated posts in some areas of expertise. Heck, I post all over the board so I would be thrilled to see what expertise tags I would distill this way. Maybe I am appreciated for posts without even realizing it now.

I can also understand that people that have worked long and hard to improve their rating in the current system may be reluctant to see it disappear. But since they are generally good contributors they would have no problem to be recognized in the new system as well.

Any change to the site that would enable me and other visitors to find better information faster is welcome. In my mind, the quality of GameDev.net stands and falls with the quality of its content, not with the quality of its users.

As I said, late reply. Hope this doesn't fall under the radar.

January 17, 2010 04:30 AM

You must log in to join the conversation.

Don't have a GameDev.net account? Sign up!

superpig

Author

GDNet V5 Concepts: User Ratings

The present system