On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
>
> I'd like to make a DIP for named unittests. Who can help me with that?
>
>
> Andrei

I agree that using library-defined annotations would be a better approach than language changes. Currently things like tested use the form
@name("AddPeer") unittest { /* ... */ }
Which is nice, because then you can extend it as desired, such as
@parallel @name("AddPeer") unittest

The main issue is that at this point, practically every single person has defined:
struct Name {
    string val;
}
string name(string val) { return Name(val); }

Adding common attributes such as this to Phobos or druntime and potentially have the default unittest runner include them would be good.

We do have an `@name` as UDA in Vibe.d, so that'll be a breaking change
(But `@NamedUnittest("name")` will do).
I also think it should be a library solution.

2015-03-31 0:21 GMT+02:00 Kapps via Digitalmars-d < digitalmars-d@puremagic.com>:

> On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
>
>>
>> I'd like to make a DIP for named unittests. Who can help me with that?
>>
>>
>> Andrei
>>
>
> I agree that using library-defined annotations would be a better approach
> than language changes. Currently things like tested use the form
> @name("AddPeer") unittest { /* ... */ }
> Which is nice, because then you can extend it as desired, such as
> @parallel @name("AddPeer") unittest
>
> The main issue is that at this point, practically every single person has
> defined:
> struct Name {
>     string val;
> }
> string name(string val) { return Name(val); }
>
> Adding common attributes such as this to Phobos or druntime and potentially have the default unittest runner include them would be good.
>

On Monday, 30 March 2015 at 23:02:51 UTC, Mathias Lang wrote:
> We do have an `@name` as UDA in Vibe.d, so that'll be a breaking change
> (But `@NamedUnittest("name")` will do).
> I also think it should be a library solution.

I can't afford to care about used names for attributes, same as we still add new symbols to Phobos (though those do break user code for same reason). They can be disambugated using module system.

(and yes, this is HUGE drawback of D module system)

On 31/03/2015 10:52 a.m., Andrei Alexandrescu wrote:
> We're having a strong need for named unittests at Facebook for multiple
> reasons.
>
> 1. We have sophisticated tooling that verifies whether unittests are
> flaky. The automated monitor (for e.g. C++) figures whether a given
> unittest fails several times across several commits. Unittests are
> identified by name; relying on file/line is impossible because the line
> of a failure is not stable across changes.
>
> 2. Again for efficient automated testing and flakiness detection, one
> should be able to run only a subset of unittests by mentioning them by
> line in the command line. Note that this implies there's no
> interdependency between distinct unittests, which is fine because the
> new ability is opt-on; I'd say is pure style anyway.
>
> 3. Mentioning unittest names in failure messages helps human
> communication (e.g. "AddPeer is failing after your change"). This is
> impossible with file and line numbers.
>
> I'd like to make a DIP for named unittests. Who can help me with that?
>
>
> Andrei

/**
 * Does something funny
 *
 * Uses writeln, to tell the user off.
 * And perhaps something else too.
 */
unittest {
	import std.stdio;
	import std.ddoc;
	writeln("BAD USER, now say \"NO\"");

	assert(readln() == "NO");

	assert(ddocParse(__DDOC__).summary == "Does something funny");
}


Only one addition required. __DDOC__ to the compiler. The rest is all library solution. Oh and assert should probably be aware of it.

void assert(bool value, string text="", string mod = __MODULE__, uint line = __LINE__, string ddoc = __DDOC__);

Well along those lines for a prototype.

On 2015-03-30 23:52, Andrei Alexandrescu wrote:
> We're having a strong need for named unittests at Facebook for multiple
> reasons.
>
> 1. We have sophisticated tooling that verifies whether unittests are
> flaky. The automated monitor (for e.g. C++) figures whether a given
> unittest fails several times across several commits. Unittests are
> identified by name; relying on file/line is impossible because the line
> of a failure is not stable across changes.
>
> 2. Again for efficient automated testing and flakiness detection, one
> should be able to run only a subset of unittests by mentioning them by
> line in the command line. Note that this implies there's no
> interdependency between distinct unittests, which is fine because the
> new ability is opt-on; I'd say is pure style anyway.
>
> 3. Mentioning unittest names in failure messages helps human
> communication (e.g. "AddPeer is failing after your change"). This is
> impossible with file and line numbers.
>
> I'd like to make a DIP for named unittests. Who can help me with that?

I completely agree. I always thought that the built-in unit test support wasn't sufficient. All of the above should be possible to implement in library code without needing to change the language.

Now, it depends on how far you want to go. We could do something simple as adding a UDA, which has already been suggested by others:

@name("this is a test") unittest
{
    assert(false);
}

Personally I would like a complete testing framework like RSpec. I have a very simple implementation [1] of this:

unittest
{
  describe("std.uni", {
    describe("toUpper", {
      it("converts a string to uppercase", {
        "foo".toUpper.should.eq("FOO")
      });
    });

    describe("toLower", {
      it("converts a string to lowercase", {
        "Foo".toLower.should.eq("foo")
      });
    });
  });
}

With RSpec, which supports multiple formatters, it can look like this:

---
Randomized with seed 28149

std.uni
  toUpper
    converts a string to uppercase
  toLower
    converts a string to lowercase

Finished in 0.00078 seconds (files took 0.0931 seconds to load)
2 examples, 0 failures

Randomized with seed 28149
---

This shows a failing test:

---
Randomized with seed 57730

std.uni
  toLower
    converts a string to lowercase
  toUpper
    converts a string to uppercase (FAILED - 1)

Failures:

  1) std.uni toUpper converts a string to uppercase
     Failure/Error: 'foo'.should == 'FOO'
       expected: "FOO"
            got: "foo" (using ==)
     # ./spec/foo_spec.rb:6:in `block (3 levels) in <top (required)>'

Finished in 0.00298 seconds (files took 0.08553 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/foo_spec.rb:5 # std.uni toUpper converts a string to uppercase

Randomized with seed 57730
---

Or with the TextMate formatter [2]. It shows both passing and failing test. When a test fails to get a link pointing back to the editor and a syntax highlighted snippet of the failing source code.

With RSpec it's also possible to specify the --line-number flag which will only run the tests matching a given line number. Or the --example flag which will run all examples (tests) matching the given string.

Is any of this interesting to have in Phobos? Otherwise I'll continue working on my own framework.

[1] https://github.com/jacob-carlborg/dspec
[2] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html

-- 
/Jacob Carlborg

On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
> We're having a strong need for named unittests at Facebook for multiple reasons.
>
> 1. We have sophisticated tooling that verifies whether unittests are flaky. The automated monitor (for e.g. C++) figures whether a given unittest fails several times across several commits. Unittests are identified by name; relying on file/line is impossible because the line of a failure is not stable across changes.
>
> 2. Again for efficient automated testing and flakiness detection, one should be able to run only a subset of unittests by mentioning them by line in the command line. Note that this implies there's no interdependency between distinct unittests, which is fine because the new ability is opt-on; I'd say is pure style anyway.
>
> 3. Mentioning unittest names in failure messages helps human communication (e.g. "AddPeer is failing after your change"). This is impossible with file and line numbers.
>
> I'd like to make a DIP for named unittests. Who can help me with that?
>
>
> Andrei


Would it be doable to make something like

unittest("Say my name") {
 // tests
}

and still be backward compatible ?

In the other hand, other proposed solutions seem to be good imho.

Am Mon, 30 Mar 2015 14:52:36 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> We're having a strong need for named unittests at Facebook for multiple reasons.
> 

As there have already been suggestions to use UDAs I think we should discuss the fundamental difficulty with unittests:

Right now the default implementation works by putting pointers to a test function into ModuleInfo. We could instead add arrays of some 'unittest information' struct to ModuleInfo to support names etc. But we can't make this as extensible and powerful as it should be: In order to support arbitrary UDAs we'd always need some kind of UDA=>runtime serialization.

The other option is getting a list of unittests at compile time.
(__traits allMEmbers, etc). AFAIK all unittest frameworks
supporting UDA use this approach. This is much more powerful and
extensible. It might make sense to switch the default implementation.

But here's the problem:

1) The compile time approach requires some kind
   of explicit registration of the unittests. At least one mixin per
   module.
2) This mixin will usually provide a module constructor. But
   using module constructors will cause issues with cycle detection.

There are some similar usecases with exactly the same issues (benchmark functions, ...). So we'd need a general solution for this.


One possible solution for problem 1 would be to have mixins that automatically get mixed in once at module scope in imported modules (a little bit like RTInfo):

-----------------------
module benchmark;

@automodule mixin template Foo()
{
    shared static this() @nocycle
    {
        //traits allmembers, ...
    }
}
-----------------------
module test;
import benchmark; //=> automatically inserts mixin Foo;
-----------------------

To solve problem problem 2 a nocycle UDA could be used to ignore
cycle checking (or @depends("benchmark") to only list explicit
dependencies).

I understand the preference to librarize as much as possible, but I don't think the desire to sacrifice every possible bit of convenience to avoid the tiniest changes to the language is always beneficial. I don't say that implementing everything inside the compiler is good either though, but in many cases some slight changes to the language can make the library solution so much more simple and elegant.

In this case, allowing to name a unittest should be a very simple language change that'll make any library implementation of the rest of the feature more elegant to use, simpler to implement, and more consistent with alternative library implementations.


Another argument in favor of language solution - sometimes the unittests take the better part of the compilation process, especially when the project is a heavily-templated library(Phobos is a prime example). When working on a bug discovered by a unittest, that means you either have to either build all unittests every time you want to check your changes, or copy the unittest code a `main` function and work on it there. If unittest names are part of the language, it could be possible to instruct the compiler to only build a single unittest, which will make the iterations much faster.

On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu wrote:
> On 3/30/15 2:55 PM, Panke wrote:
>> I've implemented this in a library and I'm sure others have as well. Are
>> you sure, you want a language solution?
>
> With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei

The "unittest" is a language thing. Why would name support be put into library I don't get it.

unittest{}  => Unnamed

unittest!"Testing new classes" {}  => Named (Explained)

On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
> On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu wrote:
>> On 3/30/15 2:55 PM, Panke wrote:
>>> I've implemented this in a library and I'm sure others have as well. Are
>>> you sure, you want a language solution?
>>
>> With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
>
> The "unittest" is a language thing. Why would name support be put into library I don't get it.
>
> unittest{}  => Unnamed
>
> unittest!"Testing new classes" {}  => Named (Explained)

Becaused adding language features takes longer than using a library, and every single feature, no matter how seemingly simple, will increase the number of langauge bugs and lead to more odd things happening.