My very first D project is now feature-complete as far as I can see (alpha) and I'd like to share it with the community:

https://github.com/atilaneves/unit-threaded

There are more details on github but here are the highlights:

1. Automatic registration of unit tests via compile-time reflection
2. Unit tests can be classes or functions, the latter for minimal boilerplate
3. Support for D's unittest blocks
4. Runs in multiple threads by default, possible to run in one thread
5. Manual selection of tests to run at the command-line (runs all tests by default)

I've used it myself to test my other D projects (2 other for now) to make sure it did what I wanted it to. I basically wrote this for myself knowing what I'd want from a unit testing library, but I'm sure it can be useful for the general D public.

Feedback is of course more than welcome!

Atila

P.S. I totally expect this to break for large real-world projects in its current state. It can test some modules in phobos but had problems with others, for instance. My own other D projects are tiny and even if I had a large D codebase the tests I'd write would reflect the biases that go into this library.

On Tuesday, 27 August 2013 at 13:07:02 UTC, Atila Neves wrote:
I cannot tell you how handy this will be. I've got a couple of projects that badly need this as is, and I can see myself using it for all future projects of any notable size.

"But doesn't D have built-in unittest blocks"? Yes, and they're massively useful. Even short scripts can benefit from them with 0 effort and setup. In fact, I use them to test this library. However, for larger projects it lacks some functionality:

    If all tests pass, great. If one fails, it's hard to know why.
    The only tool is assert, and you have to write your own assert messages (no assertEqual, assertNull, etc.)
    No possibility to run just one particular test
    Only runs in one thread.


-------------

That's exactly the reason I did something like that, too. Especially the first and third reason.

However I use a different approach, which I find easier to handle: I'm using the build-in unittest blocks to register delegates as tests in a central table. These than get executed by a test runner in main().

This way you can register multiple tests using i.e. a loop over sample data or something similar. And it's very easy to implement and has almost no impact on compile time.

On 8/27/13, Tobias Pankrath <tobias@pankrath.net> wrote:
> If all tests pass, great. If one fails, it's hard to know why.

Quoting you but responding to OP:

There was a pull I made to make assert print out some more info on failure, but it never passed the test-suite.

Pull: https://github.com/D-Programming-Language/dmd/pull/1426

>      The only tool is assert, and you have to write your own
> assert messages (no assertEqual, assertNull, etc.)

Currently we have assertThrown/assertNotThrown, but assertEqual and friends are easy to implement and we thought about including them. However it needs more community support before it's included in Phobos (i.e. more people need to want these).

I have some of these functions here: https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/test.d

>      No possibility to run just one particular test

E.g. test only one unittest block rather than the entire module at once? I'd imagine this will be possible with the new getUnitTests trait in 2.064.

>      Only runs in one thread.

getUnitTests should help here, although it's still based on compile-time introspection just like your library. I guess if nothing else, this trait will make your library implementation simpler. :)

I haven't had a problem with compile times. Ideally the test code should be small and well-separated into modules so I'm not too worried about that (the compile-time scanning only happens on the test modules).

As for the registration, it was one of my goals to not have to do any. You could still conceivably loop at compile-time with a foreach on a tuple and generate the required test functions / classes. In my experience there's usually no need for that kind of thing, but YMMV.


> However I use a different approach, which I find easier to handle: I'm using the build-in unittest blocks to register delegates as tests in a central table. These than get executed by a test runner in main().
>
> This way you can register multiple tests using i.e. a loop over sample data or something similar. And it's very easy to implement and has almost no impact on compile time.

> Currently we have assertThrown/assertNotThrown, but assertEqual and
> friends are easy to implement and we thought about including them.
> However it needs more community support before it's included in Phobos
> (i.e. more people need to want these).

   They're easy to implement and I did to test the library itself. But it's either reimplement them everytime for each project or format assert messages manually. Neither appealed to me.

>>      No possibility to run just one particular test
>
> E.g. test only one unittest block rather than the entire module at
> once?

Yes.

>I'd imagine this will be possible with the new getUnitTests trait in 2.064.

Hadn't heard of that, I'm going to have to take a look.

> getUnitTests should help here, although it's still based on
> compile-time introspection just like your library. I guess if nothing
> else, this trait will make your library implementation simpler. :)

Yay? :)

On Tuesday, 27 August 2013 at 15:42:28 UTC, Andrej Mitrovic wrote:
> getUnitTests should help here, although it's still based on
> compile-time introspection just like your library. I guess if nothing
> else, this trait will make your library implementation simpler. :)

By the, way, can we currently in 2.064 attach UDA's to unittest blocks? Together with getUnitTest that will allow to have same test code base that acts in both "old-school" mode and gets used by some fancy introspection library.

On 8/27/13, Dicebot <public@dicebot.lv> wrote:
> By the way, can we currently in 2.064 attach UDA's to unittest blocks?

Yes. E.g.:

-----
alias Seq(T...) = T;

struct RunTest { }
struct SkipTest { }

@RunTest unittest { }
@SkipTest unittest { }

void main()
{
    alias Tests = Seq!(__traits(getUnitTests, test));

    // http://d.puremagic.com/issues/show_bug.cgi?id=7804 workaround
    alias Attrib1 = Seq!(__traits(getAttributes, Tests[0]))[0];

    // ditto
    alias Attrib2 = Seq!(__traits(getAttributes, Tests[1]))[0];

    static assert(is(Attrib1 == RunTest));
    static assert(is(Attrib2 == SkipTest));
}
-----

The Seq() nonsense is to avoid running into Issue 7804. Working with attributes is generally very quirky, I had to define a whole set of __traits(compiles) versions of templates just in order to be able to extract various attributes from symbols (an attribute can be a type, an instance, a template declaration, a template instance, a literal, etc...). Here's some of the work I've had to do to work with attributes:

Helper functions: https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/attributes.d

And I use the attributes to enable things like (nevermind the ironic
name "simple" :p):
https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/parser/simple_ini/tests.d

On 2013-08-27 18:59, Dicebot wrote:

> By the, way, can we currently in 2.064 attach UDA's to unittest blocks?
> Together with getUnitTest that will allow to have same test code base
> that acts in both "old-school" mode and gets used by some fancy
> introspection library.

Yes, and unit tests for CTFE:

http://forum.dlang.org/thread/ks1brj$1l6c$1@digitalmars.com

-- 
/Jacob Carlborg

On Tue, 27 Aug 2013 17:42:19 +0200
Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:
> 
> >      The only tool is assert, and you have to write your own
> > assert messages (no assertEqual, assertNull, etc.)
> 
> Currently we have assertThrown/assertNotThrown, but assertEqual and friends are easy to implement and we thought about including them. However it needs more community support before it's included in Phobos (i.e. more people need to want these).
> 

Someone actually crated all of those and made a pull request to Phobos, but it was rejected because people decided it was better to improve the regular assert to be able to display such information automatically. Predictably, these hypothetical improvements to assert never actually got made (it's been probably around a couple years now), so that's why we've still got nothing like that in the stock DMD/Phobos.

I do wish we weren't in the habit of letting "perfect" get in the way of progress.