August 25, 2021

On 8/25/21 3:50 AM, Mathias LANG wrote:

>

Personally, I'd like my unittest binary to generate something that has a much better UX than what's currently in druntime. You know, timing statistics, colors, ability to run a single test, the whole thing.

You can get almost all that via registering a unittest handler function. The only thing that isn't provided is running individual tests, but that is a compiler issue (the compiler combines all unittests in a module into one callable).

>

I'm aware that there are frameworks out there, but it needs to be built-in.

Disagree. I'm fine with the simple unittest experience, and avoiding putting all this kind of bloat into the runtime.

-Steve

August 25, 2021

On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven Schveighoffer wrote:

>

The only thing that isn't provided is running individual tests, but that is a compiler issue (the compiler combines all unittests in a module into one callable).

You can run individual tests:

module tester1;

unittest { assert(true); }
unittest { assert(!!true); }
unittest { assert(1 != 1); }
unittest { assert(1 > 0); }

version (unittest) {
    bool tester() {
        import std.meta : AliasSeq;
        import std.stdio : writef, writeln;

        alias tests = AliasSeq!(__traits(getUnitTests, tester1));
        static foreach (i; 0 .. tests.length) {
            writef!"Test %d/%d ..."(i + 1, tests.length);
            try {
                tests[i]();
                writeln("ok");
            } catch (Throwable t) {
                writeln("failed");
            }
        }
        return false;
    }

    shared static this() {
        import core.runtime : Runtime;

        Runtime.moduleUnitTester = &tester;
    }
}

void main() {
    assert(false); // this doesn't get run
}

usage:

$ dmd -unittest -run tester1.d
Test 1/4 ...ok
Test 2/4 ...ok
Test 3/4 ...failed
Test 4/4 ...ok

I found this out in https://forum.dlang.org/post/bukhjtbxouadyunqwdih@forum.dlang.org , which has a tester that calls tests differently depending on their having a UDA

August 25, 2021

On 8/25/21 7:46 AM, jfondren wrote:

>

On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven Schveighoffer wrote:

>

The only thing that isn't provided is running individual tests, but that is a compiler issue (the compiler combines all unittests in a module into one callable).

You can run individual tests:

Well, yes. You can do everything yourself (you don't need to use the unittest system for this BTW). What I meant to say is, if you want to run all unit tests linked into the program, regardless of compile-time introspection, you need to rely on the compiler/linker to gather those for you (they are all provided as part of the ModuleInfo items), the compiler does not store references to individual tests.

The things you could do are:

  1. Print out messages for each module run, even if they pass
  2. Colorize the output of those messages
  3. Run unittests of specified modules only.
>

I found this out in https://forum.dlang.org/post/bukhjtbxouadyunqwdih@forum.dlang.org , which has a tester that calls tests differently depending on their having a UDA

UDAs are also not provided to the ModuleInfo, so there isn't a way to affect the standard unittest framework that way.

-Steve

August 25, 2021
On Wednesday, 25 August 2021 at 11:46:18 UTC, jfondren wrote:
> On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven Schveighoffer wrote:
>> The only thing that isn't provided is running individual tests, but that is a compiler issue (the compiler combines all unittests in a module into one callable).
>
> You can run individual tests:
>
> ```d
> module tester1;
>
> unittest { assert(true); }
> unittest { assert(!!true); }
> unittest { assert(1 != 1); }
> unittest { assert(1 > 0); }
>
> version (unittest) {
>     bool tester() {
>         import std.meta : AliasSeq;
>         import std.stdio : writef, writeln;
>
>         alias tests = AliasSeq!(__traits(getUnitTests, tester1));
>         static foreach (i; 0 .. tests.length) {
>             writef!"Test %d/%d ..."(i + 1, tests.length);
>             try {
>                 tests[i]();
>                 writeln("ok");
>             } catch (Throwable t) {
>                 writeln("failed");
>             }
>         }
>         return false;
>     }
>
>     shared static this() {
>         import core.runtime : Runtime;
>
>         Runtime.moduleUnitTester = &tester;
>     }
> }
>
> void main() {
>     assert(false); // this doesn't get run
> }
> ```
>
> usage:
>
> ```
> $ dmd -unittest -run tester1.d
> Test 1/4 ...ok
> Test 2/4 ...ok
> Test 3/4 ...failed
> Test 4/4 ...ok
> ```
>
> I found this out in https://forum.dlang.org/post/bukhjtbxouadyunqwdih@forum.dlang.org , which has a tester that calls tests differently depending on their having a UDA

But due to the fact that all unittests are compiled into one callable, i.e. it's all in the same process, doesn't that mean that after catching a throwable that the program is in UB ? And isn't that the reason why the runtime unittester aborts on the 1st failure ?
So shouldn't it be like every module has its own callable which can then run in separate processes which then wouldn't affect others if one of them failed ? Also, that way all the modules could be tested in parallel.
August 25, 2021
On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>
> But due to the fact that all unittests are compiled into one callable, i.e. it's all in the same process, doesn't that mean that after catching a throwable that the program is in UB ? And isn't that the reason why the runtime unittester aborts on the 1st failure ?
> So shouldn't it be like every module has its own callable which can then run in separate processes which then wouldn't affect others if one of them failed ? Also, that way all the modules could be tested in parallel.

Errors thrown by assertions in unit tests are a special case:

> Individual tests are specified in the unit test using AssertExpressions. Unlike AssertExpressions used elsewhere, the assert is not assumed to hold, and upon assert failure the program is still in a defined state.

Source: https://dlang.org/spec/unittest.html
August 25, 2021
On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
> On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>> [...]
>
> Errors thrown by assertions in unit tests are a special case:
>
>> [...]
>
> Source: https://dlang.org/spec/unittest.html

That's good to know but I was specifically referring to the catch(Throwable) part in the above example.
August 25, 2021
On Wednesday, 25 August 2021 at 13:28:40 UTC, wjoe wrote:
> On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
>> On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>>> [...]
>>
>> Errors thrown by assertions in unit tests are a special case:
>>
>>> [...]
>>
>> Source: https://dlang.org/spec/unittest.html
>
> That's good to know but I was specifically referring to the catch(Throwable) part in the above example.

Good point. It should probably be catch(AssertError), although even that is technically too broad.
August 25, 2021
On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
> On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>> [...]
>
> Errors thrown by assertions in unit tests are a special case:
>
>> [...]
>
> Source: https://dlang.org/spec/unittest.html

Also, does that hold for something like assertThrown!SomeError ?
August 25, 2021
On Wednesday, 25 August 2021 at 13:36:27 UTC, wjoe wrote:
> On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
>> On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>>> [...]
>>
>> Errors thrown by assertions in unit tests are a special case:
>>
>>> [...]
>>
>> Source: https://dlang.org/spec/unittest.html
>
> Also, does that hold for something like assertThrown!SomeError ?

Officially, according to what's written in the spec? No. In practice? Yes. And enough people rely on custom assertions like assertThrown and the ones in unit-threaded [1] that this is unlikely to change.

[1] https://unit-threaded.dpldocs.info/unit_threaded.assertions.html
August 25, 2021
On Wednesday, 25 August 2021 at 13:39:32 UTC, Paul Backus wrote:
> On Wednesday, 25 August 2021 at 13:36:27 UTC, wjoe wrote:
>> On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
>>> On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
>>>> [...]
>>>
>>> Errors thrown by assertions in unit tests are a special case:
>>>
>>>> [...]
>>>
>>> Source: https://dlang.org/spec/unittest.html
>>
>> Also, does that hold for something like assertThrown!SomeError ?
>
> Officially, according to what's written in the spec? No. In practice? Yes. And enough people rely on custom assertions like assertThrown and the ones in unit-threaded [1] that this is unlikely to change.
>
> [1] https://unit-threaded.dpldocs.info/unit_threaded.assertions.html

Thanks. I wondered about that for some time.