After a couple of weeks of quiet on the D blog, it's about to get noisy again. The latest is is a post by Mario Kröplin of Funkwerk describing how the company now uses D's built-in tests in their codebase after several years of using third-party frameworks.

Blog:
https://dlang.org/blog/2017/10/20/unit-testing-in-action/

Reddit:
https://www.reddit.com/r/programming/comments/77m8r8/ds_builtin_unit_testing_in_action/

On Friday, 20 October 2017 at 14:04:25 UTC, Mike Parker wrote:
> After a couple of weeks of quiet on the D blog, it's about to get noisy again. The latest is is a post by Mario Kröplin of Funkwerk describing how the company now uses D's built-in tests in their codebase after several years of using third-party frameworks.
>
> Blog:
> https://dlang.org/blog/2017/10/20/unit-testing-in-action/
>
> Reddit:
> https://www.reddit.com/r/programming/comments/77m8r8/ds_builtin_unit_testing_in_action/

Thanks for this post. Personally, I have not really hit the pain points described here, so I learned something. It is a valuable comparison of different unit-testing libraries and which aspects they tackle.

I took the following items from the post:

* Phobos should provide a UnitTestError class, so we can separate expectation libraries (which throw) from execution libraries (which catch). The community is not ready to decide on best library, so we need to try things and this separation would make that easier.

* fluent-asserts is considered the best expectations library. Syntax is `(x + y).should.equal(42).because("of test reasons");` and it gives nice output with code snippets.

* unit-threaded is considered the best execution library, because it shows description strings for each test. The parallelization feature did not work out for the author.

* coverage is not sufficiently solved. The author suggests to reformat code so short-circuit evaluations become multiple lines?

* Fixtures and test parameters do not require special support because builtin features like static foreach are sufficient.

On Friday, 20 October 2017 at 21:26:35 UTC, qznc wrote:
> * coverage is not sufficiently solved. The author suggests to reformat code so short-circuit evaluations become multiple lines?

If you can use gdc or ldc, branch coverage should be supported out of the box.
Other tools support regions to be marked as unreachable, e.g GCOVR_EXCL_START/GCOVR_EXCL_STOP.
I'd also err on the side that unittests themselves should not be part of coverage, but an option in druntime and more metadata from dmd might solve this.

Filed under https://issues.dlang.org/show_bug.cgi?id=17923.

On Friday, 20 October 2017 at 21:26:35 UTC, qznc wrote:
> * fluent-asserts is considered the best expectations library. Syntax is `(x + y).should.equal(42).because("of test reasons");` and it gives nice output with code snippets.

The code snippets were the prominent feature from the announcement of fluent-asserts. But this feature was the reason why I originally dismissed the library. In my opinion, the goal is that the failure message describes the issue without the need to look at the test implementation.

The diff of lengthy strings is, what I was always looking for. Back then, I wrote a lightweight kind of diff for dunit. In writing the blog post, I rechecked code.dlang.org. To my surprise, Sönke Ludwig ported google-diff-match-patch to D in 2014. (The status is "build: error", but there is hope that it's only corner cases that don't work.) Further investigation revealed that fluent-asserts uses this port. So, it's this "hidden feature" that currently makes fluent-asserts my favorite.

On 10/21/2017 6:14 AM, Martin Nowak wrote:
> On Friday, 20 October 2017 at 21:26:35 UTC, qznc wrote:
>> * coverage is not sufficiently solved. The author suggests to reformat code so short-circuit evaluations become multiple lines?
> 
> If you can use gdc or ldc, branch coverage should be supported out of the box.
> Other tools support regions to be marked as unreachable, e.g GCOVR_EXCL_START/GCOVR_EXCL_STOP.
> I'd also err on the side that unittests themselves should not be part of coverage, but an option in druntime and more metadata from dmd might solve this.
> 
> Filed under https://issues.dlang.org/show_bug.cgi?id=17923.

Not sure what is meant by branch coverage.

Consider:

     x = 2;
     if (x == 1 || x == 2)

Coverage would give:

1|    x = 2;
2|    if (x == 1 || x == 2)

I.e. the second line gets an execution count of 2. By contrast,

1|    x = 1;
1|    if (x == 1 || x == 2)

What's happening here is each of the operands of || are considered to be separate statements as far as coverage analysis goes. It becomes clearer if it is reformatted as:

1|    x = 2;
1|    if (x == 1 ||
1|        x == 2)

or:

3|    x = 2; if (x == 1 || x == 2)

It's usually possible to trivially suss out the coverage of the clauses by looking at the preceding and succeeding line counts. Putting the clauses on separate lines also works. If there's a better way to display the various counts, please add it to the bugzilla report.

On Saturday, 21 October 2017 at 22:50:51 UTC, Walter Bright wrote:
> What's happening here is each of the operands of || are considered to be separate statements as far as coverage analysis goes. It becomes clearer if it is reformatted as:
>
> 1|    x = 2;
> 1|    if (x == 1 ||
> 1|        x == 2)
>
> or:
>
> 3|    x = 2; if (x == 1 || x == 2)

What about (adding a flag) making coverage operate at the expression level instead? Meaning that each coverage result would be associated with a column offset and length aswell as the line number.

Of course the program would run much slower...

On Friday, 20 October 2017 at 14:04:25 UTC, Mike Parker wrote:
> After a couple of weeks of quiet on the D blog, it's about to get noisy again. The latest is is a post by Mario Kröplin of Funkwerk describing how the company now uses D's built-in tests in their codebase after several years of using third-party frameworks.
>
> Blog:
> https://dlang.org/blog/2017/10/20/unit-testing-in-action/
>
> Reddit:
> https://www.reddit.com/r/programming/comments/77m8r8/ds_builtin_unit_testing_in_action/

Yay! My app - covered is in this post!

That's so cool, when somebody uses your code. Thank you, Mario.

On Saturday, 21 October 2017 at 22:50:51 UTC, Walter Bright wrote:
> Coverage would give:
>
> 1|    x = 2;
> 2|    if (x == 1 || x == 2)
>
> I.e. the second line gets an execution count of 2. By contrast,
>
> 1|    x = 1;
> 1|    if (x == 1 || x == 2)

Interesting point, but would likely fail for more complex stuff.

1| stmt;
2| if (api1 == 1 && api2 == 2 ||
       api2 == 2 && api3 == 3)

Anyhow, I think the current state is good enough and there are gdc/ldc for further coverage features.

On Friday, 20 October 2017 at 14:04:25 UTC, Mike Parker wrote:
> After a couple of weeks of quiet on the D blog, it's about to get noisy again. The latest is is a post by Mario Kröplin of Funkwerk describing how the company now uses D's built-in tests in their codebase after several years of using third-party frameworks.
>
> Blog:
> https://dlang.org/blog/2017/10/20/unit-testing-in-action/
>
> Reddit:
> https://www.reddit.com/r/programming/comments/77m8r8/ds_builtin_unit_testing_in_action/

"Only for troubleshooting should you switch to unit-threaded. You have to be careful, however, to only use compatible features."

I probably should have made it more widely known that unit-threaded now has a `unitThreadedLight` version geared towards fast compile times. It uses the default test runner you get normally with just `-unittest`, but implements all of the custom assertions as plain asserts for faster turnaround times when the tests pass.

"parallel test execution (from it’s name, the main goal of unit-threaded) was quite problematic with the first test suite we converted"

I'd love to know what the problems were, especially since it's possible to run in just one thread with a command-line option, or to use UDAs to run certain tests in a module in the same thread (sometimes one has to change global state, as much as that is usually not a good idea).

"With the new static foreach feature however, it is easy to implement parameterized tests without the support of a framework"

It is, but it's the same problem with plain asserts in terms of knowing what went wrong unless the parameterised value happens to be in the assertion. And there's also the issue of running the test only for the value/type that it failed for instead of going through the whole static foreach everytime.

Atila

On Monday, 23 October 2017 at 12:38:01 UTC, Atila Neves wrote:
> "parallel test execution (from it’s name, the main goal of unit-threaded) was quite problematic with the first test suite we converted"
>
> I'd love to know what the problems were, especially since it's possible to run in just one thread with a command-line option, or to use UDAs to run certain tests in a module in the same thread (sometimes one has to change global state, as much as that is usually not a good idea).

Delays are our business, so we use the clock and timers everywhere. Long ago, we introduced Singletons to be able to replace the implementations for unit testing. By now, lots of tests fake time and it's a problem if they do so in parallel. It's not too hard, however, to change this to thread-local replacements of the clock and timers.

Another problem was using the same port number for different test cases. We now apply "The port 0 trick" (https://www.dnorth.net/2012/03/17/the-port-0-trick/).

> "With the new static foreach feature however, it is easy to implement parameterized tests without the support of a framework"
>
> It is, but it's the same problem with plain asserts in terms of knowing what went wrong unless the parameterised value happens to be in the assertion. And there's also the issue of running the test only for the value/type that it failed for instead of going through the whole static foreach everytime.

That's why I recommend to put the `static foreach` around the `unitest`. My example shows how to instantiate test descriptions (with CTFE of `format`) so that these string attributes are used to report failures or to slectively execute a test in isolation.