In the tread DSSS, Dsource and cpan it have been suggested that dsource should implement a rating system like on the matlab site and the firefox extension site.

I would like to suggest a ranking system instead of the rating system.

In a ranking system every user have a list with rankings,
when ranking a new project the user add it into the ranking between
two other projects.

Some of the problem i see with a rating system is:

  * Different users might use different bases.
  One user might examine code quality, user interface, documentation etc.
  and give 3 points on basis on this analyses.
  Another might just give 5 point to every project he likes.
  That a ranking system compare the project with each other help make a
  common basis for reference.


  * It difficult for a user to change his base.
  Imagine a user discover a new brilliant project
  that is much better than the project he already
  have rated with max points.
  To rate the new project fairly he would have
  to go back and re-rate all his rated projects.
  With ranking he can just put it at the top.

  * Low differentiation between top projects.
  Given most users only rating the projects they like.
  As a result,an extremely good and a very good project will have almost the same average score and
  the total sum will tell more about the popularity than how good it is.


A advantages of the ranking system is that it provide much more data for data mining.

Your can extract a value and make a simple average like with a ranking.
Your can run most voting system on the data, but you also have data which can
tell you that n users think that project x is better than project y. etc.

Knud Soerensen wrote:
> In the tread DSSS, Dsource and cpan it have been suggested that dsource should implement a rating system like on the matlab site and the firefox extension site.
> 
> I would like to suggest a ranking system instead of the rating system.
> 
> In a ranking system every user have a list with rankings,
> when ranking a new project the user add it into the ranking between
> two other projects.
> 
> Some of the problem i see with a rating system is:
> 
>   * Different users might use different bases.
>   One user might examine code quality, user interface, documentation etc.
>   and give 3 points on basis on this analyses.
>   Another might just give 5 point to every project he likes.
>   That a ranking system compare the project with each other help make a
>   common basis for reference.
> 
> 
>   * It difficult for a user to change his base.
>   Imagine a user discover a new brilliant project
>   that is much better than the project he already
>   have rated with max points.
>   To rate the new project fairly he would have
>   to go back and re-rate all his rated projects.
>   With ranking he can just put it at the top.
> 
>   * Low differentiation between top projects.
>   Given most users only rating the projects they like.
>   As a result,an extremely good and a very good project will have almost the same average score and
>   the total sum will tell more about the popularity than how good it is.
> 
> 
> A advantages of the ranking system is that it provide much more data for data mining.
> 
> Your can extract a value and make a simple average like with a ranking.
> Your can run most voting system on the data, but you also have data which can
> tell you that n users think that project x is better than project y. etc.

Interesting... How does this handle the differences in projects' goals?   i.e. rating Tango higher than Derelict - because they're two different categories, or goals for end-user use?

BA

Knud Soerensen wrote:
> Some of the problem i see with a rating system is:

Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure.

But looking at it with a broad brush will be, I predict, fine. Look at sites with user or moderator ranking systems - slashdot, digg, imdb, amazon, download, youtube. By and large, they are successful at separating the ones worth a second look from the ones not.

Knud Soerensen wrote

> when ranking a new project the user add it into the ranking between two other projects.

Not quite because this implies that the first two projects cannot be ranked without a third. Ranking ideally only compares two projects.

But ranking has the same vulnerabilities as rating in respect to "bad guys".

-manfred

Walter Bright wrote:
> Knud Soerensen wrote:
> 
>> Some of the problem i see with a rating system is:
> 
> 
> Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure.
> 
> But looking at it with a broad brush will be, I predict, fine. Look at sites with user or moderator ranking systems - slashdot, digg, imdb, amazon, download, youtube. By and large, they are successful at separating the ones worth a second look from the ones not.

How about whatever system is used, when the user asks for data, they are never given actual numbers. In place of numbers the server converts the scores to percentile and then gives them all a random fuzz factor of about 5% and then orders them and spits out an order. That way if 2 things are within 5-10% they will get shuffled back and forth somewhat randomly.

On Tue, 17 Apr 2007 13:53:59 -0400, Brad Anderson wrote:
> Interesting... How does this handle the differences in projects' goals?   i.e. rating Tango higher than Derelict - because they're two different categories, or goals for end-user use?

What you can do is select the rankings base on the projects category and thereby calculate the projects ranking within the category.

On Tue, 17 Apr 2007 11:30:35 -0700, Walter Bright wrote:

> Knud Soerensen wrote:
>> Some of the problem i see with a rating system is:
> 
> Any ranking system is going to have misleading, unfair and inaccurate results. So will any ranking system using annointed moderators. Any attempt to make fine distinctions based on a ranking system is doomed to failure.
Yes, correct.

But from the users ranking in my way, you can generate many types of ranking lists and here you also have data of how the project compared to each other something which you don't have in ordinary rating.

A thing I use in eigenpolls. http://all-technology.com/eigenpolls/

> But looking at it with a broad brush will be, I predict, fine. Look at sites with user or moderator ranking systems - slashdot, digg, imdb, amazon, download, youtube. By and large, they are successful at separating the ones worth a second look from the ones not.

What I don't like with these type of moderation systems is
that a expert option very often is down moderated by what I call
school kids.  (People which blindly believe what they learned in school.)

What i like best is amazons "the users also bought this
books" system but again I have discover better books with my eigenpolls.

On Tue, 17 Apr 2007 18:52:12 +0000, Manfred Nowak wrote:

> Knud Soerensen wrote
> 
>> when ranking a new project the user add it into the ranking between two other projects.
> 
> Not quite because this implies that the first two projects cannot be ranked without a third. Ranking ideally only compares two projects.

Sorry, of cause you can also put the project at the top or at the bottom
of you list.
You can thing of the list as a favorite list with ranking added.

> But ranking has the same vulnerabilities as rating in respect to "bad guys".

I think that it might be best you compute different types of rating form the ranking lists. So, gaming might win on one type of rating but lose on another type.


> -manfred

I tend to think that both have obvious deficiencies.

I would personally implement:

[] Did you use the project
[] Did you examine the source

[1 2 3 4 5] The maintainer is my best friend/family [ comments ]
[1 2 3 4 5] Project satisfied my needs [ comments ]
[1 2 3 4 5] Project was well designed [ comments ]
[1 2 3 4 5] Project was well documented [ comments ]
[1 2 3 4 5] Project performed well [ comments ]

This sort of system could then be fed into a user-defined weighted ranking system.  If the user only cares about performance, they can ask for it.  If they insist on documentation, they can ask for that.

That way we can search by our own priorities, rather than the priorities of the people who voted.  We can also weed out people who vote without knowing shit, or vote just because they like someone or hate someone else.

Developers can also get feedback from the comments and rankings so they know what to work on rather than simply getting a "your project sucks" big fat zero.

Knud Soerensen wrote:
> What I don't like with these type of moderation systems is
> that a expert option very often is down moderated by what I call school kids.  (People which blindly believe what they learned in school.)

Sure, that's the big problem with democracy. On the other hand, every other system tried is worse.