On Monday, 23 January 2017 at 01:52:29 UTC, Chris Wright wrote:
> On Sun, 22 Jan 2017 20:18:11 +0000, Mark wrote:
>> Have you considered adding randomized tests to Phobos?
>
> Randomized testing is an interesting strategy to use alongside deterministic testing. It produces more coverage over time. However, any given test run only has a fraction of the coverage that you see over a large number of runs.
>
> In other words, if the randomized tests catch something, you don't know who dun it. This is generally considered a bad thing.
>
> Phobos does have some tests that use a PRNG. They all use a fixed seed. This is a shortcut to coming up with arbitrary test data; it's not a way to increase overall coverage.
>
> I think the right way to do it is to have a nightly randomized test build, but since I'm not willing to do the work, I don't have much say.

This. So much this.
Unit tests that loop over many randomly generated input test vectors are just a waste of everybody's CPU time.

Don't get me wrong: fuzzing is also necessary. But it relies on an arbitrary time limit, which is hardly compatible with keeping a test suite fast. Which means it should be done in a another validation process than unit tests.