Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:

> 
> Wait, if you build a program with -unittest will it run some/all of phobos' unittests? That would be indeed undesirable!
> 
> I wonder how often people compile external libraries together with the application within the same command line.

It runs some of the phobos unittests, those which are in templates instantiated in user code:
-------------------------------------------------
module phobos;
struct Foo(T)
{
    unittest {import std.stdio; writeln("Test");}
}

module user;
import phobos;
void main(){Foo!int a;}
-------------------------------------------------
dmd -unittest user.d

If you remove the Foo template parameter, the test will no longer run. The reason for this behavior is simple: We can only run the test for template instances, not for declarations.

The compiler could easily check whether the template declaration of a unittest in a template instance is part of the root modules (modules present on DMD command line) and skip other tests. However, this will break the test patterns where people intentionally instantiate templates in other modules. Tests in the same module do not cover visibility issues and especially if we ever get proper DLL support for windows, where we would have to mark functions explicitly as export, you really do want to test in another module, likely even in another library.




I think the biggest problem with D's unittest implementation is that it isn't built on small, orthogonal and composable features. Instead we collect all unittests in a module, generate a function which calls them one by one and the add a pointer to that function to ModuleInfo. This implementation is highly specific and very inflexible: With the current framework, we simply cannot continue running tests after one failed, as the compiler essentially concatenated all of them into one function.

I posted a DMD PR which allows running tests individually seven/five
years ago:
https://github.com/dlang/dmd/pull/1131 https://github.com/dlang/dmd/pull/
3518
Back then this was shot down on details whether we should allow to
disable unittests and whether we should add file/line information. With
the current implementation (storing information in ModuleInfo), this is
something we as compiler developers have to decide on. But all these
issues are detail decisions, which should really be left to unittest
library/runner authors. For the core language/compiler we should rather
provide reusable building blocks which are flexible enough to allow
different library implementations instead of a highly specific,
inflexible implementation.



So why do we have this specific compiler implementation? What does it do that library code can't do? Basically it does two things:

1) Test discovery: Decide which tests should be run and collect all of them. The selection criteria is essentially this: Unittests in files passed to dmd like this: 'dmd -unittest file.d' are selected. Also some tests in other modules because of the template problem.

2) Test registration: For each discovered tests, make it somehow possible for the runtime to run it.


Now we do have library test runners*. They use compile time reflection to
find tests, build on UDAs and other well-known language features and I
really think this is the way to go. In order to switch druntime to such
an implementation, we would need to reach feature parity with the current
implementation though. The main problem here is in 1), as reflection
based runners require you to somehow list all modules which should be
tested.
I do not think we want to mess with ModuleInfo here, as this always
involves serializing compile-time type information to runtime
information. We want a completely compile time solution here. So what we
need is some way for a library to reflect on every other application
module automatically. I'd propose to leverage template mixins for this
and extend them in one crucial point. Template mixins almost do what we
want:


-------------------------------------------------
module test;

void registerTest(string name)
{
    import std.stdio;
    writeln(name);
}

mixin template registerMixin(string name)
{
    pragma(crt_constructor)
    extern(C) void registerUnittest()
    //FIXME: Need a unique mangle for this function
    {
        mixin("alias members = __traits(allMembers, " ~ name ~ ");");
        foreach(member; members)
            registerTest(member.stringof);
    }
}
-------------------------------------------------
module user;
import test;

mixin registerMixin!"user";

void main() {}
-------------------------------------------------
dmd main.d test.d


The only problem here is that we have to manually invoke `mixin registerMixin!"user";` in every module we want to test. Now what if we add `import mixins`? Simply change the `registerMixin` signature to `import mixin template registerMixin(string name)` and whenever the module is imported, the compiler automatically inserts the mixin line into the importing module. Essentially an import mixin template is a template automatically mixed in into importing modules and able to reflect on these modules. This small addition now allows us to use all the compile time, template based reflection stuff and have automatically registered tests. As this feature is generic we could also use it to auto-register @benchmark(UDA) functions. Or automatically generate serialization stuff. Or generate runtime typeinfo. Or .... And every library author can do the same.

I realize this feature is a bit controversial, as it essentially allows library authors to silently hook code into user modules. OTOH I think this is a very orthogonal feature enabling many new idioms to be explored.



2) is then easily solved, we already have crt_constructor. You could also place pointers into a named section for low level targets and there may be other options. Registration is basically a solved problem.



For the default druntime tester, we could just add such an import mixin template to object.d, so it's included in every file. Then wrap the code in the mixin template in version(unittest) and we're mostly done. Not including a main function / running the main application logic would likely be the users responsibility then, as a D library cannot really prevent the runtime from calling main. The remaining problem are tests in templates (see *), as we can't automatically register these without additional compiler help. For now, the compiler could explicity emit registerTest calls for these and we could add a deprecation for this, advising to explicitly test using RegisterTests!(T).


* Note that CTFE reflection based test runners do not run tests in
templated aggregates, so they are not affected by the template issue.
Actually, this seems to be a very good thing: You could easily create a
template which runs nested tests on demand using reflection on the type:
RegisterTests!(Foo!int);
You can easily place this into different libraries, modules, files, ...
and it will always work as expected.

-- 
Johannes

On 5/19/19 10:25 AM, Johannes Pfau wrote:
> Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:
> 
>>
>> Wait, if you build a program with -unittest will it run some/all of
>> phobos' unittests? That would be indeed undesirable!
>>
>> I wonder how often people compile external libraries together with the
>> application within the same command line.
> 
> It runs some of the phobos unittests, those which are in templates
> instantiated in user code:

That is correct, and seems desirable. Instantiations of library templated types with user types should be tested during unittesting.

On 5/18/2019 10:52 AM, Andrei Alexandrescu wrote:
> On 5/18/19 4:02 PM, Nicholas Wilson wrote:
>> On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:
>>> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?
>>
>> You almost always want to run _your_ tests not your dependencies tests, presumably they have already been tested and therefore do not need to be run.
> 
> Wouldn't the dependencies be built separately (i.e. with a different compiler invocation)?

That's right. Also, running the unittests shouldn't be a time consuming operation, so there shouldn't be an issue even if the dependency unittests were run.

On 5/19/2019 6:13 AM, Andrei Alexandrescu wrote:
> On 5/19/19 10:25 AM, Johannes Pfau wrote:
>> It runs some of the phobos unittests, those which are in templates
>> instantiated in user code:
> 
> That is correct, and seems desirable. Instantiations of library templated types with user types should be tested during unittesting.

Which suggests a "best practices" of:

1. Unittesting the logic of the template in a unittest outside the scope of the template.

2. Unittests in the scope of the template should be testing the parts of the user types that the template needs.

On 5/18/2019 8:21 AM, H. S. Teoh wrote:
> This should be pretty easy once we implement --DRT-run-unittests to only
> run unittests and skip over main().

Actually, I'd just change the behavior so if any unittests are in the executable, they are run and main() is not. main() is run only if there are no unittests.

In my experience, I've never wanted to run both in the same executable.

On Sunday, 19 May 2019 at 16:17:05 UTC, Walter Bright wrote:
> On 5/18/2019 10:52 AM, Andrei Alexandrescu wrote:
>> Wouldn't the dependencies be built separately (i.e. with a different compiler invocation)?
>
> That's right. Also, running the unittests shouldn't be a time consuming operation, so there shouldn't be an issue even if the dependency unittests were run.

These aren't necessarily true in practice. Many D builds include third party code in the main invocation (dub calls this a "sourceLibrary" target type) and sometimes unittests are awfully slow.

But that said, I think we can run with it as a pilot program and see how it works, then review if the real world data confirms or debunks our ideas. Might have to reshuffle some unit tests with templates but that might be a good idea anyway to get more guarantees out of our code.

On 5/19/19 5:27 PM, Walter Bright wrote:
> On 5/18/2019 8:21 AM, H. S. Teoh wrote:
>> This should be pretty easy once we implement --DRT-run-unittests to only
>> run unittests and skip over main().
> 
> Actually, I'd just change the behavior so if any unittests are in the executable, they are run and main() is not. main() is run only if there are no unittests.

I'd find that confusing, and difficult to even describe. "-unittest does not run main if there are any unittests, but if there aren't, main will run". Randomly running main or not in a -unittest build seems just odd. A -unittest build with no unittests does nothing. Just like the plain English sentence says. KISS and all that. -unittest means build the unit test executable, period.

> In my experience, I've never wanted to run both in the same executable.

Let's do it!!!

On 2019-05-19 11:25, Johannes Pfau wrote:

> Now we do have library test runners*. They use compile time reflection to
> find tests, build on UDAs and other well-known language features and I
> really think this is the way to go. In order to switch druntime to such
> an implementation, we would need to reach feature parity with the current
> implementation though. The main problem here is in 1), as reflection
> based runners require you to somehow list all modules which should be
> tested.
> I do not think we want to mess with ModuleInfo here, as this always
> involves serializing compile-time type information to runtime
> information. We want a completely compile time solution here. So what we
> need is some way for a library to reflect on every other application
> module automatically. I'd propose to leverage template mixins for this
> and extend them in one crucial point. Template mixins almost do what we
> want:

Have a look at my reply in a new thread [1].

[1] https://forum.dlang.org/thread/qbs8t1$jo0$1@digitalmars.com

-- 
/Jacob Carlborg

On 2019-05-18 16:56, Andrei Alexandrescu wrote:

> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? 

It's useful when you're just focusing on a single test. No point in running the other tests then. The test suite can take quite a long time to finish. The DMD test suite takes, I don't know, 10-20 minutes or something (a contributor mentioned it took 30 minutes). I would also use the "unittest" blocks for non-unit testing as well, like functional, integration and end to end testing. Those kind of tests can take hours.

> Going with the typechecking metaphor - do we want to check some modules but not others?

Yes, definitely. Having the compiler type check the file I'm currently editing in my editor as I type would be great. If it can do that without checking the other files, sure why not? It would be faster.

> If including/excluding certain unittests is needed e.g. for time reasons, "version" seems the right tool for the job. Or compiling some modules with -unittest and others without. But that should be the special case, not something supported at the command line level.

No, not good enough. It should not required to edit the code when selecting which tests to run.

-- 
/Jacob Carlborg

On Monday, 20 May 2019 at 09:32:38 UTC, Jacob Carlborg wrote:
>
> No, not good enough. It should not required to edit the code when selecting which tests to run.

+1
This is one of the things I needed many times...
What I did in the past was to extract the function and unittest or unittests I currently work on in a separate module, and once I am done I move them to the right place.