On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote:
> This post marks the start of the two week review process of std.experimental.testing.
>
> PR: https://github.com/D-Programming-Language/phobos/pull/3207
> Dub: http://code.dlang.org/packages/unit-threaded
> Doc: See CyberShadow/DAutoTest for up-to-date documentation build
>
> Previous Thread: http://forum.dlang.org/post/uzocokshugchescbawlj@forum.dlang.org

I've updated the PR with a proposal on the functionality to auto-generate the file containing the main function to run the unit tests.

Atila

On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote:
>> 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs
>
> There's `@singleThreaded` for that: all tests in a module with that UDA run in series (other modules are still run in parallel). I didn't think one was needed for random ordering.
>
> Atila

I had inverse thing in mind - all tests within a module / block run in parallel, but blocks run sequentially. It is not a good feature for unit tests but quite important one to higher level ones which deal with nasty environment issues.

On Tuesday, 15 September 2015 at 08:27:29 UTC, Dicebot wrote:
> On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote:
>>> 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs
>>
>> There's `@singleThreaded` for that: all tests in a module with that UDA run in series (other modules are still run in parallel). I didn't think one was needed for random ordering.
>>
>> Atila
>
> I had inverse thing in mind - all tests within a module / block run in parallel, but blocks run sequentially. It is not a good feature for unit tests but quite important one to higher level ones which deal with nasty environment issues.

I'm not sure we're understanding each other. With the current
implementation and a module like this:

@singleThreaded
@name("serial1") unittest { ... }

@singleThreaded
@name("serial2") unittest { ... }

@name("parallel1") unittest {... }
@name("parallel2") unittest { ...}


3 tasks would get scheduled in parallel: parallel1, parallel2, and a composite task that does serial1 then serial2. If there are any other modules, all of the other tests run in parallel with these 3 tasks.

I'm proposing to extend the same behaviour to randomised running of tests, but if that's the case the name would change.

Atila

On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:
> 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests)

I'm leaning towards not including this now and concentrating on getting it
approved - a PR to change std.experimental.testing later will be much
easier to deal with than doing it now.

> 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs

I did the check for random runs and renamed @singleThreaded to @serial in the PR.

Atila

Sure, it isn't really important and does not impact my opinion anyway. Was simply sharing experience of writing similar purpose library.

Review of std.experimental.testing formal review

the two weeks of the formal review phase are over.
The review thread was very shallow. Dicebot again expressed this disaffection with the assert function names "should(BeTrue|BeFalse|...)" No agreement could be found. Personally, I'm not sure if an agreement can be found, as this is more a personal style question rather then technical question.

Some minor comments still need to be addressed:
std.experimental.testing.reflection.TestData fields have no DDoc

I will start the voting thread a week from now.

On Monday, 28 September 2015 at 10:03:14 UTC, Robert burner Schadek wrote:
> Review of std.experimental.testing formal review
>
> the two weeks of the formal review phase are over.
> The review thread was very shallow. Dicebot again expressed this disaffection with the assert function names "should(BeTrue|BeFalse|...)" No agreement could be found. Personally, I'm not sure if an agreement can be found, as this is more a personal style question rather then technical question.
>
> Some minor comments still need to be addressed:
> std.experimental.testing.reflection.TestData fields have no DDoc
>
> I will start the voting thread a week from now.

Added the DDoc.

Atila

On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote:
> This post marks the start of the two week review process of std.experimental.testing.

Will `runTests` automatically assert that all pure unittests by default are parallellized and all non-pure are serialized? If so why is the @serial UDA needed?

Will it make use of D's builtin threadpool or will every unittest run in its own thread? IMHO, we should strive for threadpool usage here.

On Tuesday, 29 September 2015 at 10:45:23 UTC, Per Nordlöw wrote:
> On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote:
>> This post marks the start of the two week review process of std.experimental.testing.
>
> Will `runTests` automatically assert that all pure unittests by default are parallellized and all non-pure are serialized? If so why is the @serial UDA needed?

runTests with no optional arguments will run the tests in threads.
There's nothing about purity enforcement there. In fact, I tried
using pure unit tests yesterday with std.experimental.testing
and couldn't. The compiler inferred the functions to not be pure.
I tried adding pure to them and descended into a madness of adding
pure all over phobos until I got fed up. It seems to be something
to do with `format` not being pure, I have no idea why. It really
should be.

@serial is needed to _not_ run tests in threads, which you'd need
if those tests call functions that mutate global state. Even
"thread-global" is bad here since those tests could run in the same
or different threads depending on the run.

> Will it make use of D's builtin threadpool or will every unittest run in its own thread? IMHO, we should strive for threadpool usage here.

It uses the thread pool.

Atila

On Tuesday, 29 September 2015 at 14:34:42 UTC, Atila Neves wrote:
> runTests with no optional arguments will run the tests in threads.
> There's nothing about purity enforcement there. In fact, I tried
> using pure unit tests yesterday with std.experimental.testing
> and couldn't. The compiler inferred the functions to not be pure.
> I tried adding pure to them and descended into a madness of adding pure all over phobos until I got fed up. It seems to be something to do with `format` not being pure, I have no idea why. It really should be.

There are various unit tests in Phobos that are annotated with @pure, @nogc, nothrow, etc. to test that those attributes will be inferred by the compiler. In light of that, this could pose a problem.