Hello,


A coworker mentioned the idea that unittests could be run in parallel (using e.g. a thread pool). I've rigged things to run in parallel unittests across modules, and that works well. However, this is too coarse-grained - it would be great if each unittest could be pooled across the thread pool. That's more difficult to implement.

This brings up the issue of naming unittests. It's becoming increasingly obvious that anonymous unittests don't quite scale - coworkers are increasingly talking about "the unittest at line 2035 is failing" and such. With unittests executing in multiple threads and issuing e.g. logging output, this is only likely to become more exacerbated. We've resisted named unittests but I think there's enough evidence to make the change.

Last but not least, virtually nobody I know runs unittests and then main. This is quickly becoming an idiom:

version(unittest) void main() {}
else void main()
{
   ...
}

I think it's time to change that. We could do it the non-backward-compatible way by redefining -unittest to instruct the compiler to not run main. Or we could define another flag such as -unittest-only and then deprecate the existing one.

Thoughts? Would anyone want to work on such stuff?


Andrei

Andrei Alexandrescu:

> A coworker mentioned the idea that unittests could be run in parallel

In D we have strong purity to make more safe to run code in parallel:

pure unittest {}


> We've resisted named unittests but I think there's enough
> evidence to make the change.

Yes, the optional name for unittests is an improvement:

unittest {}
unittest foo {}

I am very glad your coworker find such usability problems :-)


> We could do it the non-backward-compatible way by
> redefining -unittest to instruct the compiler to not run main.

Good.

I'd also like some built-in way (or partially built-in) to use a module only as "main module" (to run its demos) or as module to be imported. This problem is solved in Python with the "if __name__ == "__main__":" idiom.

Bye,
bearophile

Am Wed, 30 Apr 2014 08:43:31 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> However, this is too coarse-grained - it would be great if each unittest could be pooled across the thread pool. That's more difficult to implement.

I filed a pull request which allowed running unit tests individually (and in different threads*) two years ago, but didn't pursue this further:

https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308

To summarize: It provides a function pointer for every  unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either.


* Some time ago there was a discussion whether unit tests can rely on
  other tests being executed first / execution order. AFAIK some phobos
  tests require this. That of course won't work if you run the tests in
  different threads.

On 4/30/14, 8:54 AM, bearophile wrote:
> Andrei Alexandrescu:
>
>> A coworker mentioned the idea that unittests could be run in parallel
>
> In D we have strong purity to make more safe to run code in parallel:
>
> pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

On 4/30/14, 8:54 AM, Johannes Pfau wrote:
> Am Wed, 30 Apr 2014 08:43:31 -0700
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
>
>> However, this is too
>> coarse-grained - it would be great if each unittest could be pooled
>> across the thread pool. That's more difficult to implement.
>
> I filed a pull request which allowed running unit tests individually
> (and in different threads*) two years ago, but didn't pursue this
> further:
>
> https://github.com/D-Programming-Language/dmd/pull/1131
> https://github.com/D-Programming-Language/druntime/pull/308
>
> To summarize: It provides a function pointer for every  unit test to
> druntime or user code. This is actually easy to do. Naming tests
> requires changes in the parser, but I guess that shouldn't be difficult
> either.

That's fantastic, would you be willing to reconsider that work?

> * Some time ago there was a discussion whether unit tests can rely on
>    other tests being executed first / execution order. AFAIK some phobos
>    tests require this. That of course won't work if you run the tests in
>    different threads.

I think indeed a small number of unittests rely on order of execution. Those will be still runnable with a fork factor of 1. We'd need a way to specify that - either a flag or:

static shared this() { Runtime.unittestThreads = 1; }


Andrei

On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

> 
> I think indeed a small number of unittests rely on order of execution. Those will be still runnable with a fork factor of 1. We'd need a way to specify that - either a flag or:
> 
> static shared this() { Runtime.unittestThreads = 1; }
> 
> 
> Andrei

Named tested seems like a no brainier to me.

Maybe nested unittests?

unittest OrderTests {
  // setup for all child tests?

  unittest a {

  }

  unittest b {

  }

}

I also wonder if its just better to extend/expose the unittest API for more advanced things like order of execution, test reporting, and parallel execution. And we can just support an external unittesting library to do all the advanced testing options.

On Wednesday, 30 April 2014 at 15:43:35 UTC, Andrei Alexandrescu wrote:
> Hello,
>
>
> A coworker mentioned the idea that unittests could be run in parallel (using e.g. a thread pool). I've rigged things to run in parallel unittests across modules, and that works well. However, this is too coarse-grained - it would be great if each unittest could be pooled across the thread pool. That's more difficult to implement.
>
> This brings up the issue of naming unittests. It's becoming increasingly obvious that anonymous unittests don't quite scale - coworkers are increasingly talking about "the unittest at line 2035 is failing" and such. With unittests executing in multiple threads and issuing e.g. logging output, this is only likely to become more exacerbated. We've resisted named unittests but I think there's enough evidence to make the change.
>
> Last but not least, virtually nobody I know runs unittests and then main. This is quickly becoming an idiom:
>
> version(unittest) void main() {}
> else void main()
> {
>    ...
> }
>
> I think it's time to change that. We could do it the non-backward-compatible way by redefining -unittest to instruct the compiler to not run main. Or we could define another flag such as -unittest-only and then deprecate the existing one.
>
> Thoughts? Would anyone want to work on such stuff?
>
>
> Andrei

An existing library implementation:
https://github.com/atilaneves/unit-threaded

On 4/30/14, 9:19 AM, Byron wrote:
> On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:
>
>>
>> I think indeed a small number of unittests rely on order of execution.
>> Those will be still runnable with a fork factor of 1. We'd need a way to
>> specify that - either a flag or:
>>
>> static shared this() { Runtime.unittestThreads = 1; }
>>
>>
>> Andrei
>
> Named tested seems like a no brainier to me.
>
> Maybe nested unittests?
>
> unittest OrderTests {
>    // setup for all child tests?
>
>    unittest a {
>
>    }
>
>    unittest b {
>
>    }
>
> }

I wouldn't want to get too excited about stuff without there being a need for it. We risk overcomplicating things (i.e what happens inside loops etc).

> I also wonder if its just better to extend/expose the unittest API for
> more advanced things like order of execution, test reporting, and parallel
> execution. And we can just support an external unittesting library to do
> all the advanced testing options.

That would be pretty rad.


Andrei

On 4/30/14, 9:24 AM, QAston wrote:
>
> An existing library implementation:
> https://github.com/atilaneves/unit-threaded

Nice! The "Warning: With dmd 2.064.2 and the gold linker on Linux 64-bit this code crashes." is hardly motivating though :o).

I think this project is a confluence of a couple others, such as logging and a collection of specialized assertions. But it's hard to tell without documentation, and the linked output https://github.com/atilaneves/unit-threaded/blob/master/unit_threaded/io.d does not exist.


Andrei

Am Wed, 30 Apr 2014 09:28:18 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> > I also wonder if its just better to extend/expose the unittest API for more advanced things like order of execution, test reporting, and parallel execution. And we can just support an external unittesting library to do all the advanced testing options.
> 
> That would be pretty rad.

We can kinda do that.
I guess the main problem for a simple approach is that unittests are
functions but advanced frameworks often have unittest classes/objects.
We can't really emulate that on top of functions.

What we can easily do is parse UDAs on unittests and provide access to these UDAs. For example:

------------
module my.testlib;

struct Author
{
    string _name;
    string serialize() {return _name;} //Must be evaluated in CTFE
}
------------
module test;

import my.testlib;
@Author("The Author") unittest
{
    //Code goes here
}
------------

Then with the mentioned pull request we just add another field to the runtime unittest information struct: An associative array with string keys matching the qualified name of the UDA and as values the strings returned by serialize() (evaluated by CTFE).

Then we have for the test runner:
------------
foreach( m; ModuleInfo )
{
    foreach(test; m.unitTests)
    {
        if("my.testlib.Author" in test.uda)
            writefln("Author: %s", test.uda["my.testlib.Author"]);
    }
}
------------

This is some more work to implement though, but it's additive so we can first implement the basic mechanism in pull #1131 then add this uda stuff later.