Raise the Hammer
The purpose of comment scoring is to encourage and reinforce civility, respect, and intellectual honesty in commenting.
By Ryan McGreal
Published May 11, 2009
Raise the Hammer is an online community of choice dedicated to urban revitalization and sustainable economic development in Hamilton, Ontario. One important form of participation on the site is in the comments on posted articles and blog entries.
Due to a recent escalation in the incidence and disruptive influence of "trolls" - comments posted for the express purpose of disrupting and undermining the discussion. One result has been that many RTH readers have become reluctant to: a) post comments for fear of getting trolled; or b) bother reading the comments as they increasingly constitute fruitless debates with a troll.
As a result, RTH proposed implementing a system of community-based comment moderation based on votes by registered users. The overwhelming response we received, both in the comments on that post and in direct correspondence, has been enthusiastic support for this system of user moderation.
However, we also received some comments that expressed concerns about a) suppressing free expression and b) RTH users 'gaming' the system with multiple sock-puppet accounts. The design of the comment moderating system attempts to address these valid concerns; and it will be a work in progress, adapting if and when problems emerge.
Here is how the system will work:
Registered users will be able to vote on comments - a simple vote up or down on each comment. You can always change your vote after the fact. Clicking the same arrow again cancels the vote, and clicking opposite arrow reverses the vote.
Registered users will have an option to set a comment score threshold - a score below which comments will be hidden by default. The default is to show all comments, and even hidden comments can be revealed by clicking a button on the hidden comment.
Registered users will also have an option to turn comment voting off entirely, so they only see the comments themselves, not their scores.
No registered user will be able to change the settings for what another user can see. Each user will choose their own settings in their user profile.
Anonymous users will still be able to leave anonymous comments and will see all posted comments (regardless of score) plus comment scores, but will not be able to vote. To do that, they will have to register a user account.
We are rolling this out in two stages. The first stage will introduce the comment voting functionality itself. Once that is running smoothly, the second stage will add the option to set thresholds and to turn comment scoring on and off to the user profile management page.
During stage one, the default will be for comment scoring to be enabled for all users and for all comments to be visible regardless of score.
If you run into any problems using this, please contact the site editor and provide details about the error.
The purpose of comment scoring is not to downvote unpopular opinions but to encourage and reinforce civility, respect, and intellectual honesty in commenting. Toward that end, we propose the following guidelines:
Please DO:
Post comments that contribute meaningfully to the discussion, either by: adding new information; correcting misinformation; offering a new perspective on an issue; challenging a conclusion already drawn; or proposing a resolution to an existing debate.
Upvote comments that are well-researched and reasonably argued, regardless of whether you agree with the conclusion or how you feel about the person who wrote it. If someone known for trolling posts a reasonable comment, encourage it with an upvote instead of dismissing or punishing it because of who wrote it.
Downvote comments that: use rude or insulting language, are needlessly inflammatory, seek to provoke an emotional reaction from others, are attempts to disrupt and derail the discussion, or abuse evidence and reasoning to defend an unjustifiable conclusion.
Write in defence of a comment that you feel is being downvoted unfairly just because it draws an unpopular conclusion or was written by someone controversial.
Please DO NOT:
Post comments that use rude or insulting language, are needlessly inflammatory, seek to provoke an emotional reaction from others, are attempts to disrupt and derail the discussion, or abuse evidence and reasoning to defend an unjustifiable conclusion.
Post comments just to say you agree/disagree with another comment. "Me too" posts don't add to the conversation and make threads long and awkward to navigate.
Vote on comments based on whether you agree with the author's opinion. Comments should be rated for the quality of their argument, not which side they've taken.
Create a phony "sock puppet" RTH account just to add upvotes for your comments or downvotes for someone else's comments.
It is up to all of us as members of an online community to reflect and reinforce our community standards of respect, civility and intellectual curiosity in our exchanges with each other.
We all have a responsibility to avoid using argumentative fallacies, to maintain a civil and respectful tone with each other, to make a sincere attempt to contribute meaningfully to the discussion, and to encourage others in the same responsible behaviour.
By reuben (registered) - website
Posted May 11, 2009 14:48:25
im very interested in seeing how this works out. nice work getting this up and running so quickly.
( Permalink )
By smz180 (registered)
Posted May 11, 2009 17:13:34
I don't post much on this site, but I'd like to state that I find the brief notice and sudden implementation of this functionality lacked any real user input. It was obviously decided before you asked any users their opinions.
I'd rather you went ahead and updated the site however you see fit, than falsely allude to inclusiveness.
( Permalink )
To date, there's 30 comments made (all at least 2 days old) on this RTH post asking users their opinion of the comment-voting idea: http://raisethehammer.org/blog.asp?id=13...
I don't know if that counts for you, but that's a fairly substantial effort (and response) in my eyes regarding user input.
RTH staff: Thanks for getting this in so quickly and seamlessly -- sometimes these discussions can drag on for weeks, or implementation can be messy. This looks quick and self-explanatory.
( Permalink )
By Brioski8 (registered)
Posted May 11, 2009 18:41:35
Thanks for this Ryan and RTH staff. You inspired me to register finally. As we can all see, the comment system is working already. SMZ's poorly researched and perhaps hasty comment has the negative score it deserves.
( Permalink )
By nobrainer (registered)
Posted May 11, 2009 21:50:01
Here here! RTH is nothing if not responsive to it's readers/commenters. I'm not sure how complicated the voting is to put in place but kudos for doing it so fast.
Next up: voting on articles themselves...
( Permalink )
By jason (registered)
Posted May 11, 2009 21:59:57
good job Ryan. Should be interesting to see how this works.
( Permalink )
By smz180 (registered)
Posted May 12, 2009 09:38:25
Yup, I read that post. If all opinions voiced we're so valued, I'd really like to know which additition functionality was added after that dialog and which user comments they were based on?
My opinion getting "bad ratings" based some assumptions of what I've read is half of why I'm posting this to begin with. Just because an opinion is contrary to most users doesn't mean it's less valid.
Lastly, for those who voted the dreaded "Negative". Exactly where does my comment fail in on the Comment Voting Etiquette above?
( Permalink )
By reuben (registered) - website
Posted May 12, 2009 10:08:22
i agree, smz180, the negative ratings on your comments are unwarranted. especially the negative votes on your followup comment questioning why.
this is not a good way to start with this new system, everyone! i dont think smz180 was trolling, just stating an opinion.
( Permalink )
smz180 wrote:
If all opinions voiced we're so valued, I'd really like to know which addition functionality was added after that dialog
The additional functionality will all be rolled out in stage two, but specifically:
Just because an opinion is contrary to most users doesn't mean it's less valid.
I'm just guessing, but I suspect it has to do with what sounded like an accusatory tone. You wrote, "It was obviously decided before you asked any users their opinions" and "I'd rather you went ahead and updated the site however you see fit, than falsely allude to inclusiveness." [emphasis added]
In fact, I proposed the idea in specific response to a user comment complaining about a lack of user moderation and in general response to a significant uptick in similar sentiments of late.
When I wrote the blog that formally proposed the comment scoring system, I was fully prepared to back away from the idea if a large part of the community opposed it - registered and non-registered users alike.
With the overwhelmingly positive response in both comments and direct emails, I decided to go ahead and develop it over the weekend and implement it in two stages.
As for how quickly the function arrived, it was a simple implementation. I had to alter two database tables, add a table, and write maybe 50 lines of code - and that includes optimizing the string method that generates the list of comments and fixing the login process for commenting.
Again, I'm prepared to modify or even remove the function if it turns out not to work. In the meantime, let's try it out and see how it works.
( Permalink )
By reuben (registered) - website
Posted May 12, 2009 10:15:22
i had one note on the etiquette list above:
DO NOT: Post comments just to say you agree with another a comment. Use the upvote arrow for that. DO NOT: Downvote a comment just because you disagree with it.
are these not conflicting points? if agreement earns a vote up, shouldnt that be balanced by disagreement earning a vote down? in my opinion, voting should never be used for agreement or disagreement of what is being said.
( Permalink )
By z jones (registered)
Posted May 12, 2009 10:18:29
Well, I registered an account, so thanks for making it worth my while.
@smz180 -
I downvoted your first comment which basically accused the site admins of lying. And then I upvoted your second comment which asked a fair question and pointed out a problem with the voting.
This is an important issue that the users can deal with by encoraging responsible voting on each other. I bet you'll see people vote one way, then read a reply and reverse there vote, that's what I see on other sites that have comment voting.
Also, don't feed the trolls! I've been saying this for awhile now but now instead of feeding the troll you can just downvote and move on.
( Permalink )
reuben wrote:
are these not conflicting points? if agreement earns a vote up, shouldnt that be balanced by disagreement earning a vote down? in my opinion, voting should never be used for agreement or disagreement of what is being said.
Heh, good point. :)
I guess what I was trying to suggest is that a good comment adds something to the discussion, rather than just a "me, too" sentiment. I'll try to think of better wording - and please feel free to make a suggestion if you have anything in mind.
( Permalink )
By UrbanRenaissance (registered)
Posted May 12, 2009 10:41:26
Ryan wrote: "I'll try to think of better wording - and please feel free to make a suggestion if you have anything in mind."
How about...
DO NOT: Post comments just to say you agree/disagree with another comment. "Me too" posts don't add to the conversation and make threads long and awkward to navigate. DO NOT: Vote on comments based on the author's opinion. Comments should be rated for the quality of their argument, not which side they've taken.
( Permalink )
Excellent, UrbanRenaissance. I've updated the article with your suggestion.
( Permalink )
By UrbanRenaissance (registered)
Posted May 12, 2009 11:04:01
One more small thing Ryan, in the "DO NOTs" section the change in item c now makes item d redundant.
Though I suppose you could always replace it with
DO NOT: Nitpick the Editor
:)
( Permalink )
By reuben (registered) - website
Posted May 12, 2009 11:09:14
I bet you'll see people vote one way, then read a reply and reverse there vote, that's what I see on other sites that have comment voting.
good point zjones. i think that this could happen a lot, and an example of where voting a comment down will encourage the original poster to clarify or re-frame their argument.
( Permalink )
UrbanRenaissance wrote:
in the "DO NOTs" section the change in item c now makes item d redundant.
Fixed. And by the way -
DO: Nitpick the Editor. :)
( Permalink )
By Blossomful (anonymous)
Posted May 12, 2009 11:12:52
Voting on articles and blog entries: what do people think of this? It wouldn't be that hard to implement.
( Permalink )
By jason (registered)
Posted May 12, 2009 11:40:15
sure. go for it. Then we'll see which ones of us deserve a raise!
( Permalink )
By UrbanRenaissance (registered)
Posted May 12, 2009 11:43:43
I don't think voting on the articles themselves is necessary, or even appropriate.
If the voting is being used as its intended, (to downgrade low quality posts and upgrade high quality posts) then Ryan, as editor is already acting to prevent low quality articles from being added to the site.
If we agree or disagree with an article then we have the comments section to argue all sides of the issue.
( Permalink )
By reuben (registered) - website
Posted May 12, 2009 11:54:19
i do not think voting on articles is needed. this site is an expression of the editors, the comments are the way readers can provide feedback, or by submitting articles themselves. this site is not a discussion forum where everything and everyone carries equal weight. it is a news source / blog. while there is a lot of two-way communication taking place here, it is primarily setup as a one-way communication tool.
( Permalink )
By JonC (registered)
Posted May 12, 2009 12:38:47
I don't think voting on the articles is necessary either.
What would be interesting is a ratio of total comment scores to number of votes (which would theoretically indicate articles with more interesting comments.
( Permalink )
By UrbanRenaissance (registered)
Posted May 12, 2009 12:53:04
JonC wrote: "What would be interesting is a ratio of total comment scores to number of votes (which would theoretically indicate articles with more interesting comments."
Not a bad idea, assign the rating to the discussion of the article, rather than the article itself. To that end why not use the ratio of comments that have positive vote counts to the number of comments with negative vote counts. That way you can see at a glance how derailed a thread has become, and at the same time it would (hopefully) prevent people from overusing the down vote since it would make that thread (and their own arguments) appear less relevant.
( Permalink )
By jason (registered)
Posted May 12, 2009 18:54:18
I can tell how derailed a thread has become when I log in and see that there are 75 comments in 1 day about Burlingtons NHL arena. People simply need to stop feeding the troll.
( Permalink )
Newsfeeds
By Ryan (registered) - website
Posted May 11, 2009 13:19:24
Additional note: at the same time, we tweaked the algorithm that displays comments to speed up page loading on articles with lots of comments, and we also fixed the comments form so that once you log in, you don't have to keep entering your username and password to post each comment.
( Permalink )
--Stuart Chase