In my summary of last month's D Language Foundation meeting, I mentioned that we discussed a system intended to reward contributors who contribute pull requests that fix Bugzilla issues. This was Razvan Nitu's baby from conception to implementation, and we all think it is a great idea.

The system has been running in the background for a few weeks now, quietly gathering data and awarding points, proving that the programming side works as intended. Our first round of point scoring kicks off on September 20th. Razvan has put together a blog post explaining how the system works.

We'll revise and adapt the system as needed as time goes by. In the meantime, happy bug fixing!

The blog:
https://dlang.org/blog/2021/09/16/bugzilla-reward-system/

Reddit:
https://www.reddit.com/r/d_language/comments/ppbp7d/bugzilla_reward_system/

On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

From the post:

In my experience, the only severity settings most people actually use when filing issues on Bugzilla are "enhancement", "normal", and "regression". And when people do use the other settings, there's no consistency to how they get applied. For example, the first two search results for priority "blocker", issues 22283 and 22148, have no indication of what (if anything) they block. Meanwhile, issues 14196 and 13983 are both enhancement requests but have their priority set to "major", and issue 22136 is listed as "critical" even though it is actually a regression!

I don't blame anyone who files reports like these. The fact is, there is no official guidance anywhere about what distinguishes a "minor" issue from a "normal" one, or a "normal" issue from a "major" one, so people just guess. But treating the output of this guessing process as though it were meaningful data is probably a mistake.

On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

Nice idea to reward contributors. Happy to see that you just try and see how it works, and adjust if needed. I think the overall positive synergy of the community is important, and this initiative should not damage it. To achieve this, I would suggest to consider giving more than 3 prizes each evaluation period. Furthermore, I would suggest to think about rewarding "rookies" as well... But let's first see how this works.

On Thursday, 16 September 2021 at 11:56:21 UTC, Mike Parker wrote:

This is a good move, I hope it will work so that D will keep contributors that are not already gone and gain new talented ones.

Another answer mentions that the lack of "issue triage" might cause problems.
I think to the opposite that this system could encourage

1. better triage
2. better reviews

But well, we'll see if this works next year. Let's not be negative ;)

On Thursday, 16 September 2021 at 14:35:08 UTC, Paul Backus wrote:

Yes of course. Rule #1:

Obviously, this isn't perfect, but it's the best we've got for the moment. I expect there will be quite a few kinks to work out as time goes by. The initial trial run will surely provide us with lessons we can apply going forward, and we will continue to refine the process as needed.

They aren't meaningful because there has been no organized process to make them meaningful. Part of Razvan's job description is to oversee the Bugzilla issues, and this initiative is a product of that. But he's one man, and the issues are legion :-) I predict we'll see the publication of reviewer guidelines down the road, and will eventually ask for a handful of people to assist Razvan in making sure Bugzilla issues have roughly accurate severity levels before a PR is submitted. Those are the most obvious steps I see to improving accuracy.

On Thursday, 16 September 2021 at 14:35:08 UTC, Paul Backus wrote:

Given that points are obtained depending on severity, my expectation is that reviewers will pay more attention to it when a PR is submitted. In addition, people that try to score as much points as possible will be interested in making sure that the competition does get the right amount of points. Therefore, I think that the rewarding system will improve the status quo with regards to labeling bugs.

On Friday, 17 September 2021 at 12:39:42 UTC, RazvanN wrote:

My "null hypothesis" expectation is that, absent guidance, reviewers will continue to do what they have always done, which is to pay no attention to the severity at all (except for regressions).

I think that this system has the potential to improve the status quo, but it will take some effort to actually get reviewers to change their habits. At minimum, we will need to

1. Write down a set of guidelines for how to use the "severity" field on Bugzilla.
2. Copy+paste and/or link to those guidelines in all of our existing instructions for contributors, including:
   • The CONTRIBUTING.md files of the dmd, druntime, and phobos repositories.
   • Wiki articles like Get involved, Starting as a Contributor, and Guidelines for maintainers.
   • The comment that dlang-bot adds automatically to all new PRs.