On Tuesday, 31 March 2015 at 12:10:15 UTC, w0rp wrote:
> On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
>> On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu wrote:
>>> On 3/30/15 2:55 PM, Panke wrote:
>>>> I've implemented this in a library and I'm sure others have as well. Are
>>>> you sure, you want a language solution?
>>>
>>> With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
>>
>> The "unittest" is a language thing. Why would name support be put into library I don't get it.
>>
>> unittest{}  => Unnamed
>>
>> unittest!"Testing new classes" {}  => Named (Explained)
>
> Becaused adding language features takes longer than using a library, and every single feature, no matter how seemingly simple, will increase the number of langauge bugs and lead to more odd things happening.

I can understand this, and I agree with you. But this is no different then ordinary businesses those do not care about quality of their products or customer service as long as they make profit. And as you know, we mostly swear at them due to this.

On 3/30/15 5:58 PM, Dicebot wrote:
> I'd prefer putting alternative test runner into Phobos instead which
> will support `@name("Something") unittest { }`

Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests.

We still don't have module RTInfo.

And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime.

-Steve

On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
> On 3/30/15 5:58 PM, Dicebot wrote:
>> I'd prefer putting alternative test runner into Phobos instead which
>> will support `@name("Something") unittest { }`
>
> Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests.
>
> We still don't have module RTInfo.
>
> And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime.
>
> -Steve

ModuleInfo does actually exist, but it's not documented. I'm not sure if it's usable for this purpose though. Maybe?

On 3/31/15 9:05 AM, w0rp wrote:
> On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
>> On 3/30/15 5:58 PM, Dicebot wrote:
>>> I'd prefer putting alternative test runner into Phobos instead which
>>> will support `@name("Something") unittest { }`
>>
>> Yes, this is one of the benefits I touted 2 years ago when I asked for
>> module RTInfo -- we can use this information in the runtime to
>> instrument how we run unit tests.
>>
>> We still don't have module RTInfo.
>>
>> And yes, then it can be a library solution. unittests are a language
>> feature, but only in how they are compiled and linked. The runtime is
>> fully responsible for how they are run. All we need is a way to tell
>> the compiler how to describe them to the runtime.
>
> ModuleInfo does actually exist, but it's not documented. I'm not sure if
> it's usable for this purpose though. Maybe?

No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271

Essentially, you have user-defined generation of runtime info stored inside the ModuleInfo. When this is working, we can do whatever we want for unit tests via attributes.

-Steve

On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
> Am Mon, 30 Mar 2015 14:52:36 -0700
> schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
>
>> We're having a strong need for named unittests at Facebook for
>> multiple reasons.

> Right now the default implementation works by putting pointers to a
> test function into ModuleInfo. We could instead add arrays of some
> 'unittest information' struct to ModuleInfo to support names etc. But
> we can't make this as extensible and powerful as it should be: In order
> to support arbitrary UDAs we'd always need some kind of UDA=>runtime
> serialization.

Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).

> The other option is getting a list of unittests at compile time.
> (__traits allMEmbers, etc). AFAIK all unittest frameworks
> supporting UDA use this approach. This is much more powerful and
> extensible. It might make sense to switch the default implementation.
>
> But here's the problem:
>
> 1) The compile time approach requires some kind
>    of explicit registration of the unittests. At least one mixin per
>    module.
> 2) This mixin will usually provide a module constructor. But
>    using module constructors will cause issues with cycle detection.

This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable.

And module constructors are not needed at all for this, there is http://dlang.org/traits.html#getUnitTests

On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
> I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant.
>
> In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations.

It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.

On 3/31/15 9:28 AM, Steven Schveighoffer wrote:
> On 3/31/15 9:05 AM, w0rp wrote:
>> On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
>>> On 3/30/15 5:58 PM, Dicebot wrote:
>>>> I'd prefer putting alternative test runner into Phobos instead which
>>>> will support `@name("Something") unittest { }`
>>>
>>> Yes, this is one of the benefits I touted 2 years ago when I asked for
>>> module RTInfo -- we can use this information in the runtime to
>>> instrument how we run unit tests.
>>>
>>> We still don't have module RTInfo.
>>>
>>> And yes, then it can be a library solution. unittests are a language
>>> feature, but only in how they are compiled and linked. The runtime is
>>> fully responsible for how they are run. All we need is a way to tell
>>> the compiler how to describe them to the runtime.
>>
>> ModuleInfo does actually exist, but it's not documented. I'm not sure if
>> it's usable for this purpose though. Maybe?
>
> No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271
>
> Essentially, you have user-defined generation of runtime info stored
> inside the ModuleInfo. When this is working, we can do whatever we want
> for unit tests via attributes.

See here too:

https://issues.dlang.org/show_bug.cgi?id=10023

-Steve

Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public@dicebot.lv>:

> > But here's the problem:
> >
> > 1) The compile time approach requires some kind
> >    of explicit registration of the unittests. At least one
> > mixin per
> >    module.
> > 2) This mixin will usually provide a module constructor. But
> >    using module constructors will cause issues with cycle
> > detection.
> 
> This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable.

But then you still have to explicitly import (or at least name) all modules that should be tested. This is a drawback compared to the current builtin-unittests where you do not have to explicitly import to-be-tested modules.

I was thinking of a fully-automated way where you only have to import a module (could even be object-d => no explicit import required) and have all tests run. This can be done by mixing in a module constructor in every module. From that constructor you'd call std.unittest.registerTest(...) and in the main function you could then call std.unittest.runTests(). This way you never need to name or know the tested modules.

IIRC std.benchmark used that approach, with the drawback of manual mixins and module constructor cycle issues.

> 
> And module constructors are not needed at all for this, there is http://dlang.org/traits.html#getUnitTests

Sure, but I was thinking about a runtime registration scheme as explained above.

Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public@dicebot.lv>:

> On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
> > Am Mon, 30 Mar 2015 14:52:36 -0700
> > schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
> >
> >> We're having a strong need for named unittests at Facebook for multiple reasons.
> 
> > Right now the default implementation works by putting pointers
> > to a
> > test function into ModuleInfo. We could instead add arrays of
> > some
> > 'unittest information' struct to ModuleInfo to support names
> > etc. But
> > we can't make this as extensible and powerful as it should be:
> > In order
> > to support arbitrary UDAs we'd always need some kind of
> > UDA=>runtime
> > serialization.
> 
> Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).

Yes, one array of TestAttribute[]
struct TestAttribute
{
    TypeInfo ti;
    void* value;
}
per unittest and an array of unittests in ModuleInfo would likely work.

But this works only for unittests and it's still restricted*. I'd prefer a more general solution which also works for benchmarks and similar usecases.

*For example you can extract file/line
information (useful for IDEs) but you'll have to add explicit @fileline
UDAs to the tests. With compile-time reflection you can gather this
information without additional UDAs.

On Tuesday, 31 March 2015 at 14:27:45 UTC, Johannes Pfau wrote:
> Am Tue, 31 Mar 2015 13:31:58 +0000
> schrieb "Dicebot" <public@dicebot.lv>:
>
>> On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
>> > Am Mon, 30 Mar 2015 14:52:36 -0700
>> > schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:
>> >
>> >> We're having a strong need for named unittests at Facebook for
>> >> multiple reasons.
>> 
>> > Right now the default implementation works by putting pointers to a
>> > test function into ModuleInfo. We could instead add arrays of some
>> > 'unittest information' struct to ModuleInfo to support names etc. But
>> > we can't make this as extensible and powerful as it should be: In order
>> > to support arbitrary UDAs we'd always need some kind of UDA=>runtime
>> > serialization.
>> 
>> Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).
>
> Yes, one array of TestAttribute[]
> struct TestAttribute
> {
>     TypeInfo ti;
>     void* value;
> }
> per unittest and an array of unittests in ModuleInfo would likely work.
>
> But this works only for unittests and it's still restricted*. I'd prefer
> a more general solution which also works for benchmarks and similar
> usecases.
>
> *For example you can extract file/line
> information (useful for IDEs) but you'll have to add explicit @fileline
> UDAs to the tests. With compile-time reflection you can gather this
> information without additional UDAs.

Problem with more generic solution is that it is also note demanding and potentially more intrusive - right now UDA don't affect actual types at all. I'd prefer something simple and practical that can work for texts right now - those are special enough case to justify dedicated approach (even if generic rtinfo gets added later)