I like to talk about the state of unit testing, test runner & frameworks and assert libraries, as I did some little experiments recently.

DLang, put a strong emphasis on unit testing. Language & tooling support for unit testing (unittest blocks, assert, assert helpers on std.exception and AssertError on core.exception, dub test ...)

However, the out of the box test runner embedded on the guts of DLang, it's pretty limited. So, many test runner has been write, as can bee see on dub package repository. This by itself, isn't bad. Having several test runner to choose it's good. Specially when one test runner it's specialized on some ways that other not.. being faster or being straightforward to use, make more easy to integrate with IDE, etc. Some of these test runners, are really more a test framework, as include some helper libraries for mocking & stubing, fluent asserts, bdd, etc.

And separated of the test runners, we have some auxiliary libraries, that implements mocking, fluent asserts, etc. Aiming to allow to be used on agnostic way respect the test runner, or even on the default test runner that comes with DLang.

So, I like to show what I saw trying the five most popular test runners (Unit-threaded, Silly, Trial, DUnit and D-Unit), and a little details that puzzles me about the different assert libraries.

D-Unit

D-Unit try to be a simple implementation of the xUnit Testing Framework. It complete ignores unittest blocks and requires to encapsulate the tests on a class, following the same pattern that does xUnit/jUnit (but using a mixin, instead of extending from some base class). Includes some assert helpers following the xUnit/jUnit framework, etc. Even can generate xUnit XML report files and differentiated between a failed test (ie. a test case where an assertion has failed) from an errored test (ie. a test where a uncaught exception has been raised)

However, isn't straightforward to use (dub run -c my-d-unit-config), and only understand as failed test only when it's being using his own assert helper functions. A failed assert form any other 3rd party assert library will be marked as an error.

DUnit

DUnit tries to being another full framework with mocking & assert helpers.

Sadly, it's dead. The *githup repo it's archived, and don't see any update since early 2020's. Also, isn't very straightforward to use (requires some extra config on dub.json, but at least could be run with a simple dub test). Other annoying thing, it's that the runner iot's pretty limited. Can't list the tests, run a single test, or show anything useful if the test run OK. Also, the pretty print of a error/fail on a test it's implemented on his own assert helpers. Using 3rd party assert library on it, would be the same experience that using on the DLang embed test runner.

Trial

Trial aims to be a very powerful test runner. It allow to use a config file to select the reporters (in plural), how discover tests, policies, and test runner implementation. Also, aims to be more easy to integrate with IDEs. Even the author had a Visual Code extension that auto uses Trial to discover tests on a project. Sadly this extension it's dead, as the author saw zero interest on it.

Using directly Trial, it's really weird. I couldn't find a way to allow it to be used as dub test or dub run on the project. Instead, you must build the Trial executable (dub run trial does the work), and it will find&execute the test in your project. Also, on my case, only worked using the master branch. Because Trial uses internally dub (compiles dub in it'self) and so, it's strongly coupled with dub.

It's really sad about the Visual Code extension, because gives the quick&easiest experience to run & debug tests that I saw using DLang on a IDE.

Silly

Silly tries to being a better test runner that it's simple to use, could run test on multiple threads, color output, list&filter tests.

To use, it's really the most easy & straightforward of any test runner that I try. I simply need to add a dependency to Silly, and run dub test. If it's being used on a library, it's recommended to add the dependency on a on "unittest" config or use it on a sub package or separated dub.json .

In my personal opinion, it's the best simply because it's the most easy to use. And don't give headhaches to configure or require special stuff to discover tests, or put tests inside classes, etc. Gives a quick&short summary, and gives good information when a test fails/errors.

However, there it's room for improvement seeing what Trial and D-Unit could do. Also looks that the author it's a bit absent. Plus, there is some issues on the gitlab repository...

Unit-threaded

Unit-threaded shares some goodies with Silly. Like his name pinpoints, run the tests on multiple threads. Also, aims to being a testing framework, including hos own assertions library, mocking helpers, sandboxing filesystem, integration testing facilities...

However, to discover tests, requires manually generating a file to register the tests or add a prebuildcommand to dub.json to autogenerate it. Something very weird, when the other tests runners can discover the test on the fly. Also, this make more problematic run some tests only when a dub configuration it's activated.

Another weird thing, it's that expects that 3rd party assert libraries, use/extend from his own exception (UnitTestException) to fail on a test. If not, it show not useful junk stack trace on the output.

About Asserts, assert libraries and AssertError

And this last details, makes my head to scratch, because I saw this on many assert libraries. Every assert library, throws a plain Exception (or UnitTestException if Unit-threaded it's added as dependency). This make really impossible to differentiate a failed test from an errored test (Remember, an test case where an assertion fails, its a fail test. An test case where a uncaught exception it's raised its an error.)

To me, sound logic that any assert library should use the same exception that it's throw by DLang's assert(). This is AssertError. And that tests runners should count any AssertError (or derived) inside a test case as failed, and any other exception as an error. This allows to assert libraries and assert runners work better without needing to be coupled to extend from some internal Exception implemented by a test runner and to have code on assert libraries to handle every test runner on the wild.

So, I like to ask WHY no body does this. Perhaps i don't see the inconvenient. It's a thing that puzzles me, and I think that should be fixed. It's one of many rought edged that have DLang and that it's pretty to fix.

I have my little experiment fork/branch of Silly doing exactly this, and Pijamas develop branch using AssertError. The result, it's that Silly counts correctly failed and errored tests and Silly & Pijamas not have any coupling between.

PD : Sorry for my poor english. Also, I have a strong background with Java and JavaScript, so I have tendency to use jUnit, Spock, jShould, etc... as some kind of gold standard for how unit testing should be.

On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

Just some comments regarding D-Unit:

You can run it just using dub test by using a dub configuration unittest.

If I am not completely wrong it does not ignore unittest blocks. It shows the result of the unittest blocks in the output.

You use asserts in your productive logic for program errors, therefore it make sense that D-Unit Mark's them as errors. (For resource errors on the other hand you use Exceptions).

Kind regards
Andre

On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

I see your point here, but I don't think it's as straightforward.

Imagine a custom array type sort of like this, being tested:

    struct MyArray
    {
        int length;
        string get(int index)
        {
            assert(index < length, "Index out of bounds.");
            return "Hello!";
        }
    }

    unittest
    {
        MyArray array;
        array.length = 2;
        assert(array.get(1) == "Hello!"); // Success
        assert(array.get(2) == "Hello!"); // Assert thrown by MyArray.get
    }

My point is, would the assert thrown by MyArray.get be considered a test failure, or a programmer error? The test runner wouldn't know since it can't distinguish between the two cases with just AssertError.

While the Error classes are for unrecoverable errors (programmer error/bugs), the Exception classes are for known bad-states that can be recovered from. E.g. JSON parsing failure is an Exception, not an Error.

So in general, it just feels more natural to catch an Exception instead of an Error, since catching an Error is bad practice anyway.

If you derive from AssertError, you still need to make your own assert functions/throw it manually, like exceptions, so it isn't really too much different from just handling exceptions instead.

Just an aside, I use Silly since it's easy to use; unittests can still run without it since it only needs a pure-string UDA @("Test name"), and honestly it makes me wonder why D's builtin ability to run unittests is so limited.

On Monday, 10 May 2021 at 17:45:03 UTC, Andre Pany wrote:

It just ignores the unit test blocks, You can try here : https://github.com/Zardoz89/pijamas/tree/develop

Launch with dub run --root=tests/d-unit
Also, you can launch with dub run --root=tests/d-unit -c fail-tests to enable enforce some failing/errors on the test cases.

Also, I try again with dub test . It just executes the dlang default unit test runner. Perhaps I have something wrong on dub.json

On Wednesday, 12 May 2021 at 18:52:13 UTC, Zardoz wrote:

You need to add to dub.json a configuration with name "unittest". Within this configuration you set "mainSourceFile" to "apptest.d" and you need to exclude you module containing the productive main function.

In apptest.d you add a main function and call the dunit_main function.

Dub test will cause the right compiler flags to have unittests included into your executable. By just using dub run, your executable likely does not have the unittests.

Kind regards
Andre

On Wednesday, 12 May 2021 at 21:23:08 UTC, Andre Pany wrote:

Thanks! I manage to get it working.

On Monday, 10 May 2021 at 19:52:28 UTC, SealabJaster wrote:

    struct MyArray
    {
        int length;
        string get(int index)
        {
            assert(index < length, "Index out of bounds.");
            return "Hello!";
        }
    }

    unittest
    {
        MyArray array;
        array.length = 2;
        assert(array.get(1) == "Hello!"); // Success
        assert(array.get(2) == "Hello!"); // Assert thrown by MyArray.get
    }

I see. I will do that Pijamas throw a custom Exception that extends from the appropriated exception from the test runner.