On Thu, 01 May 2014 10:42:54 -0400
Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> On Thu, 01 May 2014 00:49:53 -0400, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> 
> > On Wed, 30 Apr 2014 20:33:06 -0400
> > Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com>
> > wrote:

> I do think there should be a way to mark a unit test as "don't parallelize this".

Regardless what our exact solution is, a key thing is that we need to be able have both tests which are run in parallel and tests which are run in serial. Switching to parallel by default will break code, but that may be acceptable. And I'm somewhat concerned about automatically parallelizing unit tests which aren't pure just because it's still trivial to write unittest blocks that aren't safely parallelizable (even if most such examples typically aren't good practice) whereas they'd work just fine now. But ultimately, my main concern is that we not enforce that all unit tests be parallelized, because that precludes certain types of tests.

> A function may be impure, but run in a pure way.

True. The idea behind using purity is that it guarantees that the unittest blocks would be safely parallelizable. But even if we were to go with purity, that doesn't preclude having some way to mark a unittest as parallelizable in spite of its lack of purity. It just wouldn't be automatic.

> Anything that requires using the local time zone should be done in a single unit test. Most everything in std.datetime should use a defined time zone instead of local time.

Because LocalTime is the default timezone, most of the tests use it. In general, I think that that's fine and desirable, because LocalTime is what most everyone is going to be using. Where I think that it actually ends up being a problem (and will eventually necessitate that I rewrite a number of the tests - possibly most of them) is when tests end up making assumptions that can break in certain time zones. So, in the long run, I expect that far fewer tests will use LocalTime than is currently the case, but I don't think that I agree that it should be avoided on quite the level that you seem to. It is on my todo list though to go over std.datetime's unit tests and make it stop using LocalTime where that will result in the tests failing in some time zones.

> Take for example, std.datetime. The constructor for SysTime has this line in it:
> 
> _timezone = tz is null ? LocalTime() : tz;
> 
> All unit tests that pass in a specific tz (such as UTC) could be pure calls. But because of that line, they can't be!

Pretty much nothing involving SysTime is pure, because adjTime can't be pure, because LocalTime's conversion functions can't be pure, because it calls the system's functions to do the conversions. So, very few of SysTime's unit tests could be parallelized based on purity. The constructor is just one of many places where SysTime can't be pure. So, it's an example of the types of tests that would have to be marked as explicitly parallelizable if we used purity as a means of determining automatic parallelizabitily.

- Jonathan M Davis

On 2014-05-01 17:15, Andrei Alexandrescu wrote:

> That's all nice, but I feel we're going gung ho with overengineering
> already. If we give unittests names and then offer people a button
> "parallelize unittests" to push (don't even specify the number of
> threads! let the system figure it out depending on cores), that's a good
> step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in.

-- 
/Jacob Carlborg

Am Thu, 01 May 2014 08:07:51 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 5/1/14, 4:31 AM, Johannes Pfau wrote:
> > @Andrei do you think having to explicitly import modules to be tested is an issue?
> 
> Well it kinda is. All that's written on the package is "unittest", we should add no fine print to it. -- Andrei

It'd be possible to make it work transparently, but that's much
more work. We might need to do that at some point, as it's also
necessary for std.benchmark and similar code but for now that's probably
over-engineering.

Here's the revived pull request: https://github.com/D-Programming-Language/dmd/pull/3518 https://github.com/D-Programming-Language/druntime/pull/782

On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
> On 2014-05-01 17:15, Andrei Alexandrescu wrote:
>
>> That's all nice, but I feel we're going gung ho with overengineering
>> already. If we give unittests names and then offer people a button
>> "parallelize unittests" to push (don't even specify the number of
>> threads! let the system figure it out depending on cores), that's a good
>> step to a better world.
>
> Sure. But on the other hand, why should D not have a great unit testing
> framework built-in.

It should. My focus is to get (a) unittest names and (b) parallel testing into the language ASAP.

Andrei

On 5/1/14, 12:05 PM, Johannes Pfau wrote:
> Am Thu, 01 May 2014 08:07:51 -0700
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
>
>> On 5/1/14, 4:31 AM, Johannes Pfau wrote:
>>> @Andrei do you think having to explicitly import modules to be
>>> tested is an issue?
>>
>> Well it kinda is. All that's written on the package is "unittest", we
>> should add no fine print to it. -- Andrei
>
> It'd be possible to make it work transparently, but that's much
> more work. We might need to do that at some point, as it's also
> necessary for std.benchmark and similar code but for now that's probably
> over-engineering.
>
> Here's the revived pull request:
> https://github.com/D-Programming-Language/dmd/pull/3518
> https://github.com/D-Programming-Language/druntime/pull/782

I'm unclear what this work does even after having read the description. What does it require in addition to just sprinkling unittests around? -- Andrei

>
> Last but not least, virtually nobody I know runs unittests and then
>> main. This is quickly becoming an idiom:
>>
>> version(unittest) void main() {}
>> else void main()
>> {
>>     ...
>> }
>>
>> I think it's time to change that.
>>
>

The current system of running unit tests prior to main is, in my opinion, fundamentally broken.

Logically, the unit tests are a build step - something you do after compile to ensure things are good.  Tying them to running main means I cannot have a build that passes unit tests that is also a production build.

Granted, it is (as far as I know) impossible to actually compile a production version of code separately from the unittest code, and be able to run the one on the other.  But it would be nice to move to something more in line with unittest-as-build-step, rather than -as-different-build.


On named tests, I heartily support this.  Especially if it comes with the ability to selectively run one test - such is incredibly useful for large projects, to quickly iterate on broken bits.

Am Thu, 01 May 2014 12:26:07 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 5/1/14, 12:05 PM, Johannes Pfau wrote:
> > Here's the revived pull request: https://github.com/D-Programming-Language/dmd/pull/3518 https://github.com/D-Programming-Language/druntime/pull/782
> 
> I'm unclear what this work does even after having read the description. What does it require in addition to just sprinkling unittests around? -- Andrei

Nothing. This just changes the way druntime internally handles the unit tests, but nothing changes for the user:

Right now we have one function per module, which calls all unit tests in these modules, but the test runner does not have access to the individual unittest functions.

Instead of exposing one function per module, this now exposes every single unittest function. So you can now run every test individually or pass the function pointer to a different thread and run it there.

Additionally this provides information about the source location of every unit test. I thought the example is quite clear?
-------------
bool tester()
{
    import std.stdio;
    //iterate all modules
    foreach(info; ModuleInfo)
    {
        //iterate unit test in modules
        foreach(test; info.unitTests)
        {
            //access unittest information
            writefln("ver=%s file=%s:%s disabled=%s func=%s", test.ver,
test.file, test.line, test.disabled, test.func); //execute unittest
            test.func()();
        }
    }
    return true;
}

shared static this()
{
    Runtime.moduleUnitTester = &tester;
}
-------------


You customize the test runner just like you did before, by setting Runtime.moduleUnitTester in a module constructor.

Now you still have to implement a custom test runner to run unittest in parallel, but that's trivial. We can of course add an implementation this to druntime, but we can't use std.parallelism there.



What's a little more complicated is the versioning scheme, but that's an implementation detail. It ensures that we can change the exposed information (for example if we want to add a name field, or remove some field) without breaking anything.

On 5/1/14, 12:47 PM, Johannes Pfau wrote:
> Am Thu, 01 May 2014 12:26:07 -0700
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
>
>> On 5/1/14, 12:05 PM, Johannes Pfau wrote:
>>> Here's the revived pull request:
>>> https://github.com/D-Programming-Language/dmd/pull/3518
>>> https://github.com/D-Programming-Language/druntime/pull/782
>>
>> I'm unclear what this work does even after having read the
>> description. What does it require in addition to just sprinkling
>> unittests around? -- Andrei
>
> Nothing. This just changes the way druntime internally handles the
> unit tests, but nothing changes for the user:

Great, thanks. Just making sure there's no unstated assumptions somewhere. Help with reviewing from the compiler and druntime folks would be appreciated! -- Andrei

Am Thu, 01 May 2014 13:00:44 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> >
> > Nothing. This just changes the way druntime internally handles the unit tests, but nothing changes for the user:
> 
> Great, thanks. Just making sure there's no unstated assumptions somewhere. Help with reviewing from the compiler and druntime folks would be appreciated! -- Andrei
> 
> 

I added an parallel test runner example: http://dpaste.dzfl.pl/69baabd83e68

Output:
--------
Test 2, Thread 7FE6F0EA4D00
Test 4, Thread 7FE6F0EA4E00
Test 7, Thread 7FE6F0EA4E00
Test 8, Thread 7FE6F0EA4E00
Test 9, Thread 7FE6F0EA4E00
Test 10, Thread 7FE6F0EA4E00
Test 1, Thread 7FE6F0EA4F00
Test 5, Thread 7FE6F0EA4B00
Test 3, Thread 7FE6F0EA4C00
Test 6, Thread 7FE6F0EA4D00

On Thursday, 1 May 2014 at 17:57:05 UTC, Andrei Alexandrescu wrote:
> Well how complicated can we make it all? -- Andrei

As simple as possible, but no simpler :)

I've seen you favor this or that feature because it would make unit testing easier and more accessible, and eschew features that would cause folks to not bother.  In truth, we could leave it how it is.  But I surmise you started this thread to improve the feature and encourage more use of unit test.  So we're looking for the sweet spot.

I don't think it's important to support the sharing of state between unit tests.  But I do see value in being able to influence the order of test execution, largely for debugging reasons.  It's important for a module's tests to be able to depend on other modules--otherwise, unittest is not very enticing.  If it does and there's a failure, it's hugely helpful to know the failure is caused by the unit-under-test, and not the dependency(s).  The common way to do that is to run the tests in reverse order of dependency--i.e. levelize the design and test from the bottom up.  See "Large Scale C++ SW Design", Lakos, Chp. 3-4.

I imagine there are other niche reasons for order, but for me, this is the driving reason.  So it seems a nice middle ground.

If order is important, it might be a workable approach to run unittest in the reverse module dependency order by default.  A careful programmer could arrange those classes/functions in modules to take advantage of that order if it were important.  Seems like we'd have the dependency information--building and traversing a tree shouldn't be that tough....  To preserve it, you'd only be able to parallelize the UTs within a module at a time (unless there's a different flag or something.)

But it seems the key question is whether order can EVER be important for any reason.  I for one would be willing to give up parallelization to get levelized tests.  What are you seeing on your project?  How do you allow tests to have dependencies and avoid order issues?  Why is parallelization more important than  that?