I've been thinking lately about unit tests in D. The built-in support is a bit lacking. There's been many threads with this topic, even an attempt to get unit-threaded (or parts of it) into druntime/Phobos.

I was thinking, instead of trying to come up with a unit test framework that will satisfy everyone's wishes, we could create a low level unit test library in druntime. There are many styles of unit test frameworks available that look very different, but (as far as I know) most of them have very similar functionality and behavior. Although, not all of them may have all of the functionality.

The unit test library would provide functionality for registering test, before and after hooks, indicating test failures, running the tests and so on. This low level library can then be used to build different kind of unit test frameworks on top of. Be it something simple as what we have now in D, JUnit style or something like RSpec. The library would not provide functionality for finding/collecting the unit tests, that part is very framework dependent.

If this is added do druntime, we would update the existing runner to use this new library and (at least as a start) configure it to have the exact same behavior as the existing runner

To give a better idea of what I'm talking about, at the bottom is an example of how the API of the library could look like. Please don't focus on any details in the API, it only acts like any illustration to give a better understanding of the above descriptions.

Thoughts? Is this something we would like to have in druntime?

class UnitTester
{
    bool parallel;
    bool stopOnFirstFaliure;
    Formatter formatter;

    void beginTestGroup(string title = null, string file = null,
        size_t line = 0);
    void endTestGroup();

    void registerTest(void delegate() test, string title = null,
        string file = null, size_t line = 0);

    void registerTest(Context)(void delegate(Context) test, Context context,
        string title = null, string file = null, size_t line = 0);

    void registerBeforeSuiteCallback(void delegate() callback);
    void registerBeforeAllCallback(void delegate() callback);
    void registerBeforeEachCallback(void delegate() callback);

    void registerAfterSuiteCallback(void delegate() callback);
    void registerAfterAllCallback(void delegate() callback);
    void registerAfterEachCallback(void delegate() callback);

    void testFailed(Expected, Actual)(Expected expected, Actual actual);
    void testPassed();
    void testSkipped(string reason = null);
    void testPending(string reason = null);

    void runAll();
    void runSpecfic(string path, size_t line = 0);
    void runSpecfic(Regex regex, string path = null);
}

-- 
/Jacob Carlborg

On Friday, 26 August 2016 at 17:13:23 UTC, Jacob Carlborg wrote:
> I've been thinking lately about unit tests in D. The built-in support is a bit lacking. There's been many threads with this topic, even an attempt to get unit-threaded (or parts of it) into druntime/Phobos.
>
> [...]

I'm obviously very biased but having written a unit testing library that I'm happy with and proud of I'd just use that instead. I wouldn't be interested in this.

I'd much rather have `assert` be magical or have AST macros to make the syntax for writing tests better than what it is now.

Atila

Definitely not in druntime. Optional extra test runners in Phobos (or libraries for higher level test suites) are reasonable but I don't want any changes to fundamentals of how _unit_ tests are defined and used.

On 08/30/2016 10:44 AM, Atila Neves wrote:
> I'd much rather have `assert` be magical or have AST macros to make the
> syntax for writing tests better than what it is now.

Same here. BTW I'd like unittests that "must not compile" and unittests that "must fail dynamically".

For the former case, the compiler should cooperate:

@incompilable unittest { ... }

fails if it passes compilation. So the compiler must know about that attribute.

For the latter case, no change in language is necessary, only in druntime:

@mustfail unittest { ... }

Would love these two.


Andrei

On 2016-08-30 16:44, Atila Neves wrote:

> I'm obviously very biased but having written a unit testing library that
> I'm happy with and proud of I'd just use that instead. I wouldn't be
> interested in this.

The point of this low level library is that a high level unit test library, like your unit-threaded, can use the low level library in druntime. If this was to be implemented I imagine we could move some code from unit-threaded to this low level library. No need for unit test frameworks to reimplement the low level stuff that would be very similar.

> I'd much rather have `assert` be magical or have AST macros to make the
> syntax for writing tests better than what it is now.

That would be nice to have.

-- 
/Jacob Carlborg

On 2016-08-30 16:54, Dicebot wrote:
> Definitely not in druntime. Optional extra test runners in Phobos (or
> libraries for higher level test suites) are reasonable but I don't want
> any changes to fundamentals of how _unit_ tests are defined and used.

You both are kind of missing the point. This would not be a runner in the same sense as the existing runners. It would be a low level library that a runner can use. If this would to be implemented the existing unit test runner in druntime would be updated to use this library. It would configure the library to behave exactly like the current runner work.

Since this library does collect any unit tests it's up to the code that uses the library to collect the tests. The runner in druntime would collect the tests exactly as it does now and those how tests are defined and used would not be changed at all.

The reason to put in the druntime is because that's where the existing runner is located.

The advantage of this low level library is that:

* Third party unit test library don't need to reinvent the wheel

* All third party libraries using this low level library would be compatible and can be combined in the same project

* If we would like to, it would be easy to extend the existing runner, like not stopping on the first failure. _Not_ saying that we should

-- 
/Jacob Carlborg

On Tuesday, 30 August 2016 at 16:06:21 UTC, Jacob Carlborg wrote:
>> I'd much rather have `assert` be magical or have AST macros to make the
>> syntax for writing tests better than what it is now.
>
> That would be nice to have.

There is a bit of work on a closely related topic on the wiki:

https://wiki.dlang.org/DIP83

On 2016-08-30 18:17, Seb wrote:

> There is a bit of work on a closely related topic on the wiki:
>
> https://wiki.dlang.org/DIP83

Or https://wiki.dlang.org/DIP50 :)

-- 
/Jacob Carlborg

On 08/30/2016 07:17 PM, Jacob Carlborg wrote:
> The reason to put in the druntime is because that's where the existing runner is located.
> 
> The advantage of this low level library is that:
> 
> * Third party unit test library don't need to reinvent the wheel
> 
> * All third party libraries using this low level library would be compatible and can be combined in the same project
> 
> * If we would like to, it would be easy to extend the existing runner, like not stopping on the first failure. _Not_ saying that we should

I definitely wouldn't want to use API like you proposed if I was to write my own test runner. Minimal common ground which would be cool to have is getting range/array of all unittest blocks together with their annotations. Anything more than that is optional luxury that specific test systems may define.

And any usage of classes in what is supposed to be a base ground API is immediate "no" for me :)

On Tuesday, 30 August 2016 at 16:18:30 UTC, Jacob Carlborg wrote:
> On 2016-08-30 18:17, Seb wrote:
>
>> There is a bit of work on a closely related topic on the wiki:
>>
>> https://wiki.dlang.org/DIP83
>
> Or https://wiki.dlang.org/DIP50 :)

Or that unit-threaded's @ShouldFail seems very similar to Andrei's @MustFail