While trying to make a few minor contributions during the holidays, it seemed like every other PR was hitting a "random" unit test failure.

So I solved the few that I noticed [1-3] (the last is still waiting review / merge.)

However finding failures that have gone unnoticed or unreported is a manual process. So I wrote a simple script that scrapes the auto tester website, process the logs of "interesting" failures, groups identical failures and outputs them.


I'm wondering if there is interest in having the auto tester detect & track the sporadic failures to make them more prominent and easier to find & fix.

I perceive that the attitude towards "random" test failures is widespread apathy. Failures seldom get filed in the bug tracker. While bad tests are often the cause, sometimes there are significant issues:

For example, [1] was a significant implementation flaw in std.experimental.allocator that was present for ~1.25 years.

[1] Phobos PRs #4988 & #4993
[2] Phobos PR #4997
[3] Phobos PR #5004