This post is mostly directed to DMD contributors.

Currently most of the tests for DMD are end-to-end like tests. The test invokes the compiler as a new process and the test asserts the exit code and/or error messages outputted by the compiler. This means that basically for each test file a new process is created which runs the compiler. There are two problems with this: a lot of processes need to be started and executed and since the compiler runs in a separate process from the test, it's not possible to access the internals of the compiler.

DMD also contains a few more of traditional unit tests (using the `unittest` blocks) which tests individual functions. Mostly utility functions.

I would like to announce that a new test runner has recently been added for testing DMD. It's like a mix of both of the existing tests types. It uses the `unittest` blocks and uses the compiler as a library. It runs the test and the compiler in the same process. It allows to compile code without starting a new process, then the test is free to assert whatever it likes. For example, the structure of internal data that are only available in the same process as the compiler.

The test runner compiles all files of this test type into a single executable. This should hopefully speed up running the tests because only a single executable is compiled and run, instead of hundreds.

This type of tests lives in the `test/unit` directory [1]. They're using the `unittest` block to declare a test. It supports attaching a UDA to filter tests. It also supports before and after hooks that are executed before each `unittest` block within the same module and after each `unittest` block (regardless of the test failed or passed). The test runner will print a report at the end of which tests that failed.

The tests are executed by default when `make -C test` or `./test/run.d` is executed. The runner supports a more finer grained control of running the tests:

* To run all the unit test (without running the other tests), run: `./test/run.d -u`

* To only run the unit tests in one or more specific files:
`./test/run.d -u test/unit/self_test.d`

* To only run a subset of the unit tests in a single file:
./test/run.d -u test/unit/self_test.d --filter "self test"

In the above example, the `--filter` flag will filter to only run the tests with a UDA matching the given value, in this case `self test`.

An example can look like this [2]:

module self_test;

import support : afterEach, beforeEach, defaultImportPaths;

@beforeEach initializeFrontend()
{
    import dmd.frontend : initDMD;
    initDMD();
}

@afterEach deinitializeFrontend()
{
    import dmd.frontend : deinitializeDMD;
    deinitializeDMD();
}

@("self test")
unittest
{
    import std.algorithm : each;
    import dmd.frontend;

    defaultImportPaths.each!addImport;

    auto t = parseModule("test.d", q{
        int a = 3;
    });

    assert(!t.diagnostics.hasErrors);
    assert(!t.diagnostics.hasWarnings);
}

You're free to test whatever you like inside the `unittest` block, the `parseModule` function is not required to use. Here's another example that does not use it [3].

I highly recommend splitting up the `unittest` blocks to only test one thing per `unittest` block.

I encourage all DMD contributors to give this a try. Using this kind of test will hopefully speed up the DMD test suite and make it possible to test things that are not tested today.

[1] https://github.com/dlang/dmd/tree/master/test/unit
[2] https://github.com/dlang/dmd/blob/master/test/unit/self_test.d
[3] https://github.com/dlang/dmd/blob/master/test/unit/deinitialization.d

-- 
/Jacob Carlborg

Thanks, I think it's a nice tool to improve quality via more thorough/faster testing.