March 31, 2015
On 04/01/2015 12:58 AM, deadalnix wrote:
> So now we are going to change the language for this ?
> 
> There is a natural name for unitests, the name of the module. We have way to break module into pieces in a backward compatible manner now, so it's all good.

Are you saying one should split off unittests into submodules?

Or are you saying https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d is enough for your needs?

I think having multiple tests per module makes sense, e.g. because it's better to parallelize the tests.

> We may want to add various annotation to a test, and we have UDA for that.

Sure use a UDA, I don't care. It seems important though, that the
compiler recognizes it and emits it as metadata to the ModuleInfo.
And if you need some compiler magic anyhow, a UDA just looks redundant.

@unittest("my test") unittest

unittest("my test")

> So the only things that is really needed is a way to customize the test runner from client code to output whatever everybody needs.

You mean this? http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester
March 31, 2015
On 03/31/2015 11:01 PM, Atila Neves wrote:
> This is true, but importing modules by name can be code that is generated. Which is exactly what I do with dtest [1] and unit-threaded [2]. It's not that big of a deal when it's part of the build system.

Dub does it as well (dub test), because if you don't import a module, it
might not get linked too.
March 31, 2015
On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
> The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei

In any case you should talk to Atila Neves who wrote a really good unittest library.

http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest

It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
March 31, 2015
On 3/31/15 4:38 PM, Martin Nowak wrote:
> On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
>> The reasonable course is to see how far we can get with a library-only
>> solution. Amaury, want to work on that? -- Andrei
>
> In any case you should talk to Atila Neves who wrote a really good
> unittest library.
>
> http://code.dlang.org/packages/unit-threaded
> http://code.dlang.org/packages/dtest
>
> It's also used by quite a lot of people already.
> http://code.dlang.org/api/packages/unit-threaded/stats

Atila, is there interest in moving your library to std.experimental? -- Andrei
April 01, 2015
On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote:
> The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei

If nobody picks it up, I can, but I have very little bandwidth right now with the work involved on SDC.
April 01, 2015
On 3/31/15 5:58 PM, deadalnix wrote:
> On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote:
>> The reasonable course is to see how far we can get with a library-only
>> solution. Amaury, want to work on that? -- Andrei
>
> If nobody picks it up, I can, but I have very little bandwidth right now
> with the work involved on SDC.

I'll quote that when you post a long rant here :o). -- Andrei
April 01, 2015
On Tuesday, 31 March 2015 at 23:29:28 UTC, Martin Nowak wrote:
> On 04/01/2015 12:58 AM, deadalnix wrote:
>> So now we are going to change the language for this ?
>> 
>> There is a natural name for unitests, the name of the module. We have
>> way to break module into pieces in a backward compatible manner now, so
>> it's all good.
>
> Are you saying one should split off unittests into submodules?
>

No necessarily.

Typically, in most test framework, you have a class, and various method, each of them testing various aspect of what you want to test.

I see the class here as the module, and each of the method as a unitest block.

> Or are you saying
> https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d
> is enough for your needs?
>
> I think having multiple tests per module makes sense, e.g. because it's
> better to parallelize the tests.
>

See my previous comment, that is not needed. In fact i also think having several unitest blocks is useful.

>> We may want to add various annotation to a test, and we have UDA for that.
>
> Sure use a UDA, I don't care. It seems important though, that the
> compiler recognizes it and emits it as metadata to the ModuleInfo.
> And if you need some compiler magic anyhow, a UDA just looks redundant.
>

How I see it, it should be possible for a user to specify a test runner template. This test runner template can do compile time reflection and format its output the way to wants.

We must not bake all required attribute we need to associate a test in the language. What if we want to associate the test with a team name, or give it a name, or give an id that my tools uses to check for tests, or whatever.

Every user will want to add various metadata, and these will vary. Every users will want various output, and these will vary.

It should be possible to have a test runner that use template and compile time reflection to make sense of the unitest block. Yes, there is a bit of compiler magic involved, but nothing terribly fancy.

> You mean this?
> http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester

We need template to use compile time reflection.
April 01, 2015
On 2015-03-30 21:52:36 +0000, Andrei Alexandrescu said:

> We're having a strong need for named unittests at Facebook for multiple reasons.
> 
> 1. We have sophisticated tooling that verifies whether unittests are flaky. The automated monitor (for e.g. C++) figures whether a given unittest fails several times across several commits. Unittests are identified by name; relying on file/line is impossible because the line of a failure is not stable across changes.
> 
> 2. Again for efficient automated testing and flakiness detection, one should be able to run only a subset of unittests by mentioning them by line in the command line. Note that this implies there's no interdependency between distinct unittests, which is fine because the new ability is opt-on; I'd say is pure style anyway.
> 
> 3. Mentioning unittest names in failure messages helps human communication (e.g. "AddPeer is failing after your change"). This is impossible with file and line numbers.
> 
> I'd like to make a DIP for named unittests. Who can help me with that?
> 
> 
> Andrei

This is a fantastic idea -- and something I was concerned about lately.  +1

I can possibly help with a DIP, but it seems like a very straightforward request?

-Shammah

April 01, 2015
>
> We need template to use compile time reflection.

Please don't make it all compile time. It is useful to register tests at runtime as well. Like one tests per SAT instance in a given SAT benchmark.

Practically only thing we need is a standard way to register a test function as a unittest and associate a key-value store with it.
April 01, 2015
On Tuesday, March 31, 2015 14:45:49 Idan Arye via Digitalmars-d wrote:
> But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.

Exactly what I was thinking. If we just made

unittest myTest
{
}

or

unittest(myTest)
{
}

legal, then the unit test function would then be named myTest instead of whatever it's already named right now. It's a lot less ugly than using attirbutes, and it fits really well into what we already have. Certainly, it seems like a very straightforward and simple solution to the problem.

- Jonathan M Davis