Jump to content

April 2017 »

S M T W T F S
      1
2345678
9101112131415
16171819202122
23 24 2526272829
30      
- - - - -

Upgrade Update: Reputation Feedback



4: Adsense

Reputation.

This is a tricky subject.

The site upgrade - by nature of the upgrade - is going to introduce changes to our reputation system. There is no stopping the change. The current reputation system is custom-made and is not portable to the upgrade. This doesn't mean we couldn't port it, but it's a question of ROI. Some may perceive this to be a negative, but the new software has an integrated and much more flexible reputation system for us to work with. We can carry forward with our existing reputation rules, but we don't have to do that.

The purpose of reputation is to encourage participation and interaction with members. I do believe our existing system achieves this to a degree, but it has its issues. For instance, it doesn't take into account or potentially over-weighs certain activities that aren't as important in a growing community. Also, reputation loses its impact over time, meaning once you reach a certain level of reputation it's like getting to Level 100 in World of Warcraft.

The name "reputation" itself doesn't make complete sense, either. If you logged in every day of a non-leap year you'd earn 365 reputation points, without participating in the community. If you upvoted every comment you saw you'd earn thousands of points, without providing your own content. Our reputation system right now is more of an activity meter, and just because someone has high reputation doesn't necessarily mean they have contributed valuable content and interactions in the community.

For these reasons and the fact we're doing a complete software switchover, perhaps it's time to reconsider how our reputation system operates.

Before sharing my own thoughts I want to gather yours. So let's start with a few simple questions to help frame the conversation:

1. What do you like about the current reputation system?

2. What do you dislike about the current reputation system?

3. Do you have any proposals to improve the existing system?

If the community can contribute a minimum of 25 answers to these questions (i.e. 25 comments answering these 3 questions) I'll randomly select a winner from the comments to receive 3 free months of GDNet+ (or 3 months added to your existing subscription).

I look forward to seeing your feedback. :)

Apr 20 2017 04:42 PM

I'd like to start the discussion from first(er?) principles, even: if "the purpose of reputation is to encourage participation and interaction with members," do we even need a reputation system? Aren't there other ways we could encourage and reward interaction between members without trying to distill all those complex interactions into a single global, visible number? Do we even need a silly numeric ranking system with potentially complex rules about weights and caps and costs to do this?

I'd vote for a world without it.

Apr 20 2017 05:30 PM

I like numbers and seeing how they change over time. Always have.

There have been times where knowing that a answer will likely yield some points have contributed to me "bothering" to reply or give input. This was especially true when reputation was always visible at the top of the page, without having to expand various stuff to see it. Partly, this was due to it being easier for me to track when reputation changes happened -- with multiple tabs open and swapping between them, I often miss the notifications that pop up in the lower right of the window.

So, in response to Josh, I personally do like some kind of value to keep track of, even if it's just for my own number-fixated self (and it doesn't even have to be visible to others).

In regards to the listed questions:

1) I like being able to see a sort of "progress". I like that helpful answers can be recognized. I like that I can vote people up instead of having to quote them and say "This", or similar. I don't need to clutter threads in order to show appreciation of a post. I like how you can generally spot trolls and people seemingly more interested in provoking others than asking/discussing things -- although this isn't a sure-fire way in any sense, I find it works in a lot of cases. Similarly, I like how it allows me to put specific answers to a given topic into context. If there's something I don't know a lot about, and there's 2 replies - one with +8 and the other with -8 -- chances are I should consider the first one to be more factual or at least commonly accepted.

2) I dislike several things. For one, votes are given for a variety of reasons. The main one is probably that disagreeing with someone often leads to negative votes. A well thought out post, even if I don't necessarily agree with it, does not mean that the poster is less reputable, in my eyes. There's no distinction between "you are just trolling and I think you are a silly person" and "No I hate that language/feature so I will vote you down". Similarly, there is no distinction between "insightful post, I learned something", "that's totally irrelevant, but funny" and "yeah tell that troll where they should stick their head (where the sun doth not shine)". Related, it's kinda impossible to know why a vote was given. Was I down-voted because what I said was wrong or unfunny, or was it because someone just disagrees with the approach, or because someone dislikes me due to earlier posts?

3) Like Josh, I have to wonder... if participation and interaction is the goal, why not just base it on pure participation? I.e. 1 point per post, potentially with thumbs up/down you give worth 1/10th or something.

Apr 20 2017 05:40 PM

So one, I never liked getting points for logging in or upvoting. But two, I'm quite ok with the OP getting points or good answers getting points. At least we know those people are adding to the knowledge base of GDN and the points can indicate that.

However, users should be rewarded, quite visibly, for actual contributions to the site. Blogs (arguably), articles, content for the Store, creating tutorials, hosting Yearly Events (ex: WoA). If those rewards come in the form of (major) points, icons, titles. Those are nice. But what about something a bit more tangible. Books, [x] amount of time of free membership, a ticket to a conference (ex: CppCon, E3, PAX East), Steam Keys, etc. Either heavily discounted or free. (Or if you really wanna get crazy, pull a string and get someone's game published by a AAA studio). I think that would spark some activity or at least motivation for some more, if not better, content to be added by the users.

It still would be nice if there were some place to hosted or publish games. (But I know that a different subject for a different day.)

Apr 20 2017 05:41 PM
To answer your actual questions (please exclude me from the GDNet+ running, naturally):


1. I guess the thing I like the most about the current system is that it is a mechanism for providing feedback on how well an individual post contributes to the overall thread. But to be clear I don’t think that’s a huge advantage. I’d rather see such feedback articulated with actual words, especially when that feedback is negative.


2. I dislike that it reduces the problem space to a simple, constantly-inflating, dimensionless number. I dislike that that context-free number is given such prominence in the site’s UI, that the system encourages quantity over quality, and that there’s no point to increasing it except to increase it. I feel like most people either don’t care about the number much at all and just post what they want to, or they do and just post to game the system.


3. The simplest possible thing I’d like to do that involves mostly keeping the existing system is to stop calling it “reputation,” to remove the display from every post, and to label it in a user’s profile what it is: X posts voted helpful, Y posts voted unhelpful. Also, detach the other mechanisms for ticking the value up (logging in, flagging, et cetera). Reduce it to and treat it like what it mostly is: a count of helpful/unhelpful posts. I’d still vote to axe the system entirely though.

If the point of the system is to award participation, I’d definitely like to hear from others what they’d consider a compelling reward for such participation. Is it really “points?” I find that uninteresting, but maybe that’s just me.

I tend to vote for the posts of people I find to repeatedly post interesting, helpful or educational stuff. What I’d like to see is a system whereby I can find or follow the content from these users more easily, while also hiding the content from users I find unhelpful or problematic. To that end, one reputation-like system I’d like to see in place is a way for me to give a thumbs up or down to an individual user, once, and for the software to indicate to me when threads have posts by people I like in them, highlight those posts in some fashion, and similarly de-emphasis (but not hide entirely) posts from users I’ve “thumbed-down.” Sort of like a follower system, without all the silly social network connotations.

I also see the value in a system where individual posts are rated helpful or not; I’m probably being idealistic in thinking that if we remove per-post votes, people will use their words. Realistically they’ll still just not post anything and move on. So, if such a system ends up being in place I’d like it simply stay on the posts and perhaps not show up until there is a net threshold crossed, sort of like how we implement the post badges: +5 score or more, it gets a little star or whatever. -4 score or lower, it gets de-emphasized.

I don’t think we need to reflect these values publicly. Maybe a user could see, on their profile, that X people have found them interesting and Y of their posts were helpful, or whatever, but I don’t really see the need to expose that so obviously in public and turn it into a contest.
Apr 20 2017 06:17 PM

Yeah, I agree. I never cared for getting upvoted for logging in. I'm not really participating (In the strictest sense) more so just scanning the forums to see if there is something I can contribute to, and maybe looking over new articles. Maybe instead of us having points; when someone get's an answer with +5 upvotes (Or even just +2) that question get's marked with a sticker conveying relevance.

This upvote counter would only be visible to moderators. (Similar to how it's done now, but +5 is a pretty high metric when not everyone feels compelled to upvote, but the OP is completely satisfied with the answer, and it solves their problem, and the op is the one to give the feedback.)

In the context of having something, as a user, to tally up what you've contributed, you could view your profile, and see all the posts you've made that were marked as relevant under a special heading. Maybe, after a number of relevant postings you earn a MVP sticker on your profile. But if that is tiered, than it's simply a contrived way of circling back to a point metric.

But, yeah. Even just axing the upvote feedback +1, and logging in would make the contribution value much more concise, and relevant for what it's intended to measure.

Edit:

Of course this isn't set in stone. As an inexperienced OP might give a +1 for relevance just as a way to incentivize people to keep contributing to their thread (I done seen it O_o), or might just run with an answer given despite it being incorrect, or having a VERY unwieldy side effect of using, though it solves their immediate problem

Apr 20 2017 06:48 PM

I don't see anything wrong with a numeric upvote/downvote system, if anything it is a quiet way for other experts validate a response to a thread.

An OP may not even know which of the answers (posts) is the correct (or best) until validated by other experts in the field,  not only by postings, but also by upvotes. And over some time an OP is more likely to trust responses given by someone who has accumulated many of such upvotes for giving accurate useful and helpful responses in a particular field. 

How can that be a wrong system?

Lets be honest here, we have varying degrees of expertise, skills and time to response to threads. And I think that should be reflected one way or another

Which is the more reason why the visibility of upvotes and a kind of cumulative reputation is correct. I will always like to be aware of those with the reputation of giving helpful answers and a numeric system is a good summarised way of doing this

It will also motivate and encourage people to put more care and attentions to their responses to posts, 

Also rude and trolling members/posts unfortunately also need to be recognised for who they are. Most people don't like that public stigma so negative numeric reputation could also be a deterrent  

Maybe the system needs to be perfected rather than discarded all together. 

Apr 20 2017 07:13 PM

Now suggestion on how this can be perfected? Hmm... very good question. The ideal system that I have in mind would be too tasking for mods and  staff so I will go for a more watered down second-tier  system.

In addition to keeping the current system, add a bronze, silver and gold badge expertise reputation to the current system. This means that people like me who accumulated their reputation merely by logging in and voting up others would never have an expert contributor's badge medal even if I have a high numeric reputation. On the other hand others who have accumulated most of their reputation by actually giving valuable insightful and helpful answers to threads would gradually move up the badge system

Late Edit: If visible cumulative voting up/down is scrapped, the fun and edge would be taken off this site, I for one would find it a very boring site. Most people would like to see a kind of dynamism for their participation even if it is (as Bregma calls it) fake internet points. Its a psychological necessity and its also a bit of fun. And with badge medal added on top of the current one, I can't see any weaknesses

Apr 20 2017 07:21 PM

I live for fake internet points.

I think the statement that just logging in and reading is not participating in the community is incorrect. It's passive participation, and the most comfortable form of participation for many introverted programmers.  Taking away the barest minimum of reward just because you're not a extroverted social butterfly hardly seems fair.

I seldom upvote and I don't think I've ever downvoted anything, but it's a rush when an idea you've expressed gets upvoted by someone. And yeah, it's frustrating when you get a downvote, especially when it's obviously for idiological reasons.  Then again, they're only fake internet points.

The only proposal I'd have is to just nix the downvotes.  Good comments will be indicated by more votes, plain and simple.  Give good answers, get more fake internet points.  Elegant.

Apr 21 2017 04:12 PM

1. What do you like about the current reputation system?

The current reputation system encourages me to visit the site more than just once a day.

The system is very game-like, which has be returning or even when my day is fill thinking about login to see what other's are doing.

2. What do you dislike about the current reputation system?

The current reputation system could value actions a bit better. Like you stated there are activities on the site that reward actions with low impact, which I think should remain, but fails to reward some actions.

 

3. Do you have any proposals to improve the existing system?

If there was a +3 for posting a Journal entry, sorry Blog, I would be more pushed to post one at least once a week. Maybe that could be a thing, +3 rep for blog entries up to 3 times a week. Similar things could be done for comments or forum post.

Maybe have thread creators select an "answer" to their question and that user receive additional points. 

Think if there was a way to grant points for retweeting or drawing in more people/traffic. 

If the community can contribute a minimum of 25 answers to these questions (i.e. 25 comments answering these 3 questions) I'll randomly select a winner from the comments to receive 3 free months of GDNet+ (or 3 months added to your existing subscription).

Wow! This is an awesome offer, and maybe it should be expanded. Like have a monthly reward giveaway.

 

Short Answer:

I wouldn't cut the system. When there were different options in a thread on a topic I referred to the rep value to see whom was a more trusted/active member. I think that it has worth, reddit has their Krama system and it has shown some power.

Apr 21 2017 04:16 PM

 

 

In addition to keeping the current system, add a bronze, silver and gold badge expertise reputation to the current system. This means that people like me who accumulated their reputation merely by logging and voting up others would never have an expert contributor's badge medal even if I have a high numeric reputation. On the other hand others who have accumulated most of their reputation by actually giving valuable insightful and helpful answers to threads would gradually move up the badge system

 

I can see the bronze, silver and gold badge expertise reputation. It would be in a similar fashion to the way there were shoutouts, on the Developer Journal Weekly Summary, to the people that did commenting while not posting themselves.

 

Side Note:

My Quote from selected text system is not working :(

Apr 22 2017 04:06 AM

Albeit not perfect, upvoting is widely understood and sorta working.

I would however get away with the idea of scoring as numbers: whatever you keep them serverside it's a thing but I don't think they should be presented to user.

Everything that matters is to bubble up valuable insights. What I care: understand if something/someone is valuable/reliable. In general the specific number isn't relevant to me but I know if it's in the thousands he's probably been around a bit. Perhaps I will re-read. Sometimes I also check 'member since'.

Some messages also have 'tags'. I recall the 'popular' tag on the Vulkan/API discussion. I'm pretty sure those could help. Then, how do we enable people to distribute them?

Apr 22 2017 12:20 PM

I second what Bregma thinks.

My own experience with Reputation on this site is it's kinda like a drug, I look forward to seeing my rep go up, and I'm in a tough camp because I know that I don't have too much technical expertise to offer given my limited skill set.  I rely more on the creative side of the forums for my rep.  I think a lot of other community members are employed within an industry related to computers, I am not.  

I judge a person by their rep, someone below 300 I think they may not stick around too long and I should caution how much time I invest, once someone is in the 1000's I know they're more serious and anyone in the +10000's knows their stuff and I trust their advice and answers.

Yesterday, 09:12 PM

Thank you all for the comments and input. There's a lot here, but I see a few common themes.

We'll continue with some kind of reputation system, so I hope I didn't give the impression that we're considering killing it. We also don't intend to reset everyone's reputation, but there will need to be some kind of normalization with the new system, whatever it may be.

A few issues with the current system were also brought up that I might go ahead and fix.. like no points for Blogs. :)

Few other thoughts:

  • Reputation or some form of acknowledging a member's value to the community is important to keep
  • All actions need to be reviewed to ensure valuable contributions are rewarded
  • A member's value can be judged by the a) quality of contributions, b) number of contributions, and c) membership duration

I'll make another post on the ideas under consideration, but I'll give this post and comment thread a few more days to gather feedback before sharing those thoughts.

Thanks again!

Yesterday, 10:07 PM
Looking back to when I was just a lurker on the site in 2001-2003, the reputation system has helped cut down on the problems, and with each evolution has helped reduce the number of bad postings.

Having a visible marker to say "the community approves/agrees" or "the community disapproves/disagrees" tends to strongly encourage positive and useful replies, tends to discourage negative and unhelpful replies. The switch from a per-user basis to per-post basis had an enormous boost to the quality of posts. That type of feedback should be kept even if there is a programming cost.

Upvote points are limited in a day, and upvotes are nearly always given on positive feedback. If someone's only contribution to the site are upvoting content they find useful, that's fine. They may be occasional visitors who don't feel the need to post but do find useful answers. Even though they are more consumers than contributors, by voting the are contributing something. I think that should continue if reasonably possible.

I like the small penalty for downvotes since it discourages downvote sprees. There have been times over the years where that has been a problem, people doing upvote and downvote attacks, but since upward points are limited and downward points are penalized it has become a rare event. Rogue accounts (or sockpuppet accounts) that downvote every post against a user for all time, or upvote every post for all time, have mostly vanished, and on those rare instances are quickly identified by moderators and staff.

Some of the other points are less valuable to the community, except they encourage people to participate in things the normally might not. It encourages the submission of articles, encourages reporting spam or other problems, encourages other contributions. Without those reputation boosts there is less visible benefit to users of the site and they may contribute less. That is speculation about what might motivate people, no idea if it is true.

Since points are meaningless beyond a certain level --- over a thousand or so -- perhaps it would be good to shift from numbers to words at a certain point, or at least hide the numbers on forum posts. That is probably more custom programming that you want to avoid, but it does reduce some of the bias. There is some pretty clear deference in the discussions for Hodgman (50K), me (45K), SotL(33K), the four people in the 20K range, and the 27 others who exceed 10,000 points. Sadly it is sometimes discussion killing. Having visible points replaced with words rather than numbers on forum posts could improve the site, although it would be good to still see your exact numbers in your own profile.
Today, 05:01 AM
Just a note on the reputation gain for logging in. I think it's useful for new member up to a certain point, it helps encourage people to come back to the site and gets their initial reputation up. But once you get above a certain point (arbitrarily picking 1000 points), it probably shouldn't count anymore.
Today, 06:52 AM

@Khawk - poor blogs. :D Is this to promote more articles or because of the random nature of blogs? Just curious.

@frob - It would be nice if somehow people could be given some sort of acknowledgement as a guru. Like if someone has a "C++" tag next to their name, then people know that user is knowledgeable in C++. The tag could be given by committee or number of articles (or something).

Note: GameDev.net moderates comments.