On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote:
> On 3/31/15 4:38 PM, Martin Nowak wrote:
>> On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
>>> The reasonable course is to see how far we can get with a library-only
>>> solution. Amaury, want to work on that? -- Andrei
>>
>> In any case you should talk to Atila Neves who wrote a really good
>> unittest library.
>>
>> http://code.dlang.org/packages/unit-threaded
>> http://code.dlang.org/packages/dtest
>>
>> It's also used by quite a lot of people already.
>> http://code.dlang.org/api/packages/unit-threaded/stats
>
> Atila, is there interest in moving your library to std.experimental? -- Andrei

Sure, I can have the PR ready by next week. As mentioned earlier, a UDA to attach a name to a unittest block and then run with that name in the output is 30min work. Making the whole thing std.experimental ready will take a few days.

TBH, if I'd known it had a chance of making it in there, I'd've done it ages ago!

Atila

On Wednesday, 1 April 2015 at 04:42:33 UTC, Shammah Chancellor wrote:
> I can possibly help with a DIP, but it seems like a very straightforward request?
>
> -Shammah

Considering the heated debate going on here, I'd say it's not as straightforward as it seems...

I have PR https://github.com/D-Programming-Language/phobos/pull/2995 open since October 2014.

it includes:
* extensible haskell like quickcheck
* benchmarking with "names" and record keeping to see progress
* offline tool to plot benchmark results with gnuplot
* most std.string functions already have benchmarks
* makefiles got a new BUILD=benchmark

IMO continuous benchmarking of phobos is a must-have feature and a nice marketing tool ( benchmark graphs on dlang.org). Something similar to quickcheck is equally important, to get the test input you didn't expect.

I got one and a half comments so far.

On 2015-03-31 23:08, Idan Arye wrote:

> Limiting unittest names to legal identifiers will save a lot of headache
> when we set our tools to actually use these names. Matching a legal
> identifier in a text stream is much easier than matching an arbitrary
> unicode string, even if that string is escaped.

I don't see any problem with arbitrary strings. It works perfectly fine in RSpec, which has much better tooling than D.

-- 
/Jacob Carlborg

On 2015-03-31 11:08, Johannes Pfau wrote:

> But here's the problem:
>
> 1) The compile time approach requires some kind
>     of explicit registration of the unittests. At least one mixin per
>     module.
> 2) This mixin will usually provide a module constructor. But
>     using module constructors will cause issues with cycle detection.

When we get RTInfo for modules [1] there shouldn't be any problems. A template will be instantiated once for each module the compiler sees. With that module __traits(getUnitTests) and __traits(allMembers) can be used to access the unit tests, benchmarks, or whatever. Not need for any registration or module constructors.

[1] https://github.com/D-Programming-Language/dmd/pull/2271

-- 
/Jacob Carlborg

On 2015-03-31 23:14, Idan Arye wrote:

> Building by unittest name! Imagine - instead of placing temporary code
> in `main` to develop a new feature or fix a bug, you put in a named
> unittest and tell your IDE/build-system to only build that unittest(and
> whatever code needed for it to run). You `writeln` stuff to the console,
> and when you get some changes to output what you want you change the
> `writeln`s to `assert`s and proceed to work on the next step. When you
> are done, all you have to do is tidy it up a bit and BAM - without any
> special effort you get a unittest that tests that feature/bug you just
> worked on.

You just reinvented test driven development ;). It's perfectly possible to do this with a UDA an a text string as well. UDA's also allows you to tag the tests. Basically a short name you put on multiple tests, then tell the test runner to run only those tests, or ignore those.

@tag("foo") @name("this is my test name") unittest {}

$ run-tests -t foo

-- 
/Jacob Carlborg

On 2015-04-01 00:02, Idan Arye wrote:

> I think you and I work under different assumptions of the goals for this
> feature. If we only want unittest names to be something that can be
> printed when the unittest runner runs the unittests, than a UDA with a
> string is indeed preferable. If we want something that tools can
> actually use to refer to a specific unittest, we need a proper
> identifier(yes, even though it can be implemented in library code
> because D is Turing-complete...)

The tools should, mostly, use the file and line information.

-- 
/Jacob Carlborg

On 2015-03-31 15:31, Dicebot wrote:

> Most powerful solution would be to simply put attributes for unittest
> blocks in runtime information for tests (using RTTI it should be
> possible to define such variadic structure in similar manner as D-style
> variadic function arguments).

I think the most powerful and most generic solution would be to:

1. Remove the unittest keyword
2. Make code executable at module scope
3. Add support for trailing delegate syntax

module foo;

unittest("foobar")
{
}

Would be lowered to

unittest("foobar", {

});

Works with benchmark and other things as well.

-- 
/Jacob Carlborg

On 4/1/15 12:46 AM, Atila Neves wrote:
> On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote:
>> On 3/31/15 4:38 PM, Martin Nowak wrote:
>>> On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
>>>> The reasonable course is to see how far we can get with a library-only
>>>> solution. Amaury, want to work on that? -- Andrei
>>>
>>> In any case you should talk to Atila Neves who wrote a really good
>>> unittest library.
>>>
>>> http://code.dlang.org/packages/unit-threaded
>>> http://code.dlang.org/packages/dtest
>>>
>>> It's also used by quite a lot of people already.
>>> http://code.dlang.org/api/packages/unit-threaded/stats
>>
>> Atila, is there interest in moving your library to std.experimental?
>> -- Andrei
>
> Sure, I can have the PR ready by next week. As mentioned earlier, a UDA
> to attach a name to a unittest block and then run with that name in the
> output is 30min work. Making the whole thing std.experimental ready will
> take a few days.
>
> TBH, if I'd known it had a chance of making it in there, I'd've done it
> ages ago!
>
> Atila

Sounds like a plan. Thanks in advance! -- Andrei

On Wednesday, 1 April 2015 at 14:05:46 UTC, Jacob Carlborg wrote:
> On 2015-03-31 23:14, Idan Arye wrote:
>
>> Building by unittest name! Imagine - instead of placing temporary code
>> in `main` to develop a new feature or fix a bug, you put in a named
>> unittest and tell your IDE/build-system to only build that unittest(and
>> whatever code needed for it to run). You `writeln` stuff to the console,
>> and when you get some changes to output what you want you change the
>> `writeln`s to `assert`s and proceed to work on the next step. When you
>> are done, all you have to do is tidy it up a bit and BAM - without any
>> special effort you get a unittest that tests that feature/bug you just
>> worked on.
>
> You just reinvented test driven development ;). It's perfectly possible to do this with a UDA an a text string as well. UDA's also allows you to tag the tests. Basically a short name you put on multiple tests, then tell the test runner to run only those tests, or ignore those.
>
> @tag("foo") @name("this is my test name") unittest {}
>
> $ run-tests -t foo

The problem is not with running the tests, it's with building them. In heavily templated libraries(like, for example Phobos), building without unittests takes seconds and building with unitetests takes minutes - mainly because the tests need to to many template instantiations. If we could tell the compiler to only build a single, specific test the development cycle can become orders of magnitude faster.