On Friday, 11 September 2015 at 15:54:57 UTC, Jacob Carlborg wrote:
> On 2015-09-11 13:27, Atila Neves wrote:
>
>> Unit tests should run in a fraction of a second... no, there's no such
>> functionality.
>
> Why would I not use this for other kinds of tests?

I guess you're right. In any case, getting what's in there to get accepted is work enough as it is.

Atila

On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote:
> This post marks the start of the two week review process of std.experimental.testing.
>
> PR: https://github.com/D-Programming-Language/phobos/pull/3207
> Dub: http://code.dlang.org/packages/unit-threaded
> Doc: See CyberShadow/DAutoTest for up-to-date documentation build
>
> Previous Thread: http://forum.dlang.org/post/uzocokshugchescbawlj@forum.dlang.org

My opinion hasn't changed since last review - it is a very solid and useful library but I don't want provided API to become standard (== part of Phobos).

I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency.

At the same time actual test running facilities (parallel, randomized, etc) look generally applicable and uncontroversial.

On 2015-09-12 11:52, Atila Neves wrote:

> I guess you're right. In any case, getting what's in there to get
> accepted is work enough as it is.

Yes, absolutely. I wasn't arguing for the feature, just the reason not to add it :)

-- 
/Jacob Carlborg

On 2015-09-12 15:34, Dicebot wrote:

> I also don't like mixing unittest and higher level functional tests
> (with setup and cleanup phases) into the same buckets - this doesn't fit
> nice with D module system. Latter should be placed in a separate
> modules/package to avoid being picked up by rdmd & Co when compiled as
> dependency.

Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory?

-- 
/Jacob Carlborg

On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote:
> Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory?

Nope but the fact that they are treated the same (no separate category, no output separation, no documentation warning) encourages to do so - while in practice it is almost always a bad idea.

On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote:
> On 2015-09-12 15:34, Dicebot wrote:
>
>> I also don't like mixing unittest and higher level functional tests
>> (with setup and cleanup phases) into the same buckets - this doesn't fit
>> nice with D module system. Latter should be placed in a separate
>> modules/package to avoid being picked up by rdmd & Co when compiled as
>> dependency.
>
> Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory?

On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose:

1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests)
2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs

On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:
> On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote:
>> On 2015-09-12 15:34, Dicebot wrote:
>>
>>> I also don't like mixing unittest and higher level functional tests
>>> (with setup and cleanup phases) into the same buckets - this doesn't fit
>>> nice with D module system. Latter should be placed in a separate
>>> modules/package to avoid being picked up by rdmd & Co when compiled as
>>> dependency.
>>
>> Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory?
>
> On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose:
>
> 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests)

I've never heard of functionality like that, but should be easy to implement.

> 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs

There's `@singleThreaded` for that: all tests in a module with that UDA run in series (other modules are still run in parallel). I didn't think one was needed for random ordering.

Atila
Atila

On 09/11/2015 01:27 PM, Atila Neves wrote:
>> How about Fuzz-tests, randomize input for test on each run?
> 
> Like QuickCheck? Robert has something for that.

There is also https://github.com/MartinNowak/qcheck for that.

On 2015-09-13 12:44, Atila Neves wrote:

> I've never heard of functionality like that, but should be easy to
> implement.

We're using that at work, but on a different level. We have two separate jobs in Jenkins, one depends on the other one. If the first one fails, the second one is not run.

-- 
/Jacob Carlborg

On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:
> On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote:
>> On 2015-09-12 15:34, Dicebot wrote:
>>
>>> I also don't like mixing unittest and higher level functional tests
>>> (with setup and cleanup phases) into the same buckets - this doesn't fit
>>> nice with D module system. Latter should be placed in a separate
>>> modules/package to avoid being picked up by rdmd & Co when compiled as
>>> dependency.
>>
>> Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory?
>
> On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose:
>
> 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests)

I'll see if I can add this before the end of the review phase.

> 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs

How about I change the name from @singleThreaded to @serial and make sure it works with randomised runs as well (it probably already does, I just have to check)?

Atila