On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:
> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?

You almost always want to run _your_ tests not your dependencies tests, presumably they have already been tested and therefore do not need to be run.

On Sat, May 18, 2019 at 10:58:36AM -0400, Andrei Alexandrescu via Digitalmars-d wrote:
> On 5/18/19 6:49 AM, H. S. Teoh wrote:
> > One useful pattern that we could consider, that I've developed over time, is to have the compiler compile*two*  executables for the same code, one with unittests (with no main()) and the other without (and with main()). My build script runs both in parallel, and automatically executes the unittest executable as part of the build. If a unittest fails, the build aborts with an error. Otherwise, it deletes the unittest executable, leaving the "real" one ready to run.
> 
> This is very nice, and very close do being doable today with scripting without modifying the codebase being built. What's needed is that main is not run after unittesting.

This should be pretty easy once we implement --DRT-run-unittests to only run unittests and skip over main().


T

-- 
If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell

On Sat, May 18, 2019 at 10:56:16AM -0400, Andrei Alexandrescu via Digitalmars-d wrote:
> On 5/18/19 2:20 PM, Adam D. Ruppe wrote:
[...]
> > Actually, right after I went to bed, I realized the solution to everyone's problem.
> > 
> > We all want `-unittest=package,list` to control which tests are run.
> 
> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?

This is actually one of the big warts in D's otherwise very nice built-in unittests: when you compile with -unittest, it compiles ALL unittests, including those of 3rd party libraries that you really don't care about.  You *really* want to be able to say "hey, compile *my* unittests, which are in modules X, Y, Z, and don't bother with everything else, 'cos they are external code that have already been unittested to death".  I'm really *not* interested in running Phobos unittests for the 100th time which the CI's have already run 1000 times over, every single darned time I compile my own code!


> If including/excluding certain unittests is needed e.g. for time reasons, "version" seems the right tool for the job. Or compiling some modules with -unittest and others without. But that should be the special case, not something supported at the command line level.
[...]

Wrong. Compiling with -unittest currently compiles unittests for ALL imports.  Meaning, (practically) all Phobos unittests get compiled and run as soon as you have the audacity to `import std.xyz` (Phobos being the tangled hairball of dependencies that it is right now).  Using "version" is really not practical since I'd have to modify Phobos code everytime I compile my own code and/or have knowledge of Phobos internal details such as version identifiers to turn unittests on/off.

This isn't just Phobos, of course, but any 3rd party library the code may depend on. I shouldn't have to know secret magic version identifiers for every single 3rd party library I use just so I can turn off their unittests!

Far from being the special case, this should be the *default* behaviour: nobody wants to recompile and rerun 3rd party unittests for the 100th time which the upstream authors should already have done; they want to unittest their *own* code.


T

-- 
Too many people have open minds but closed eyes.

On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:
> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few?

The others already mentioned that this is a common pain point among D users with third party code being included, but even with your own code, there might be times when you'll want to take shortcuts for a quicker test iteration cycle. Unittests are supposed to be independent, after all, so no point running hundreds of them when you are in the middle of working on fixing just one.

This specific -unittest=pattern syntax has been requested a couple times before, and there was at least one PR for it a couple years ago. Here it is: https://github.com/dlang/dmd/pull/6375

One of the suggestions back then was to make -unittest only work on modules explicitly compiled in (open question: how does this interact with the new -i switch added in 2018, after that 2017 PR?), but Atila said that breaks the utility __traits(getUnitTests). You (Andrei) at the time said that is a good trade off, and a few agreed.

I lean toward that being a good solution too, just Atila's unit-threaded is an important project that we should give careful consideration to when changing anything about D's unittest facilities.

Another possibility is to compile in all the code, and use a runtime switch to the program (instead of the compiler) to determine which ones actually run. But I'd prefer doing it on the compiler side because fast build times are one of D's key advantages. Again, very frustrating to have to wait several seconds each edit/build/debug iteration when you are working on fixing one unit test at a time.

On 5/18/19 4:02 PM, Nicholas Wilson wrote:
> On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:
>> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?
> 
> You almost always want to run _your_ tests not your dependencies tests, presumably they have already been tested and therefore do not need to be run.

Wouldn't the dependencies be built separately (i.e. with a different compiler invocation)?

On 5/18/19 4:32 PM, H. S. Teoh wrote:
> This is actually one of the big warts in D's otherwise very nice
> built-in unittests: when you compile with -unittest, it compiles ALL
> unittests, including those of 3rd party libraries that you really don't
> care about.  You*really*  want to be able to say "hey, compile*my*
> unittests, which are in modules X, Y, Z, and don't bother with
> everything else, 'cos they are external code that have already been
> unittested to death".  I'm really*not*  interested in running Phobos
> unittests for the 100th time which the CI's have already run 1000 times
> over, every single darned time I compile my own code!

Wait, if you build a program with -unittest will it run some/all of phobos' unittests? That would be indeed undesirable!

I wonder how often people compile external libraries together with the application within the same command line.

On Saturday, 18 May 2019 at 17:52:40 UTC, Andrei Alexandrescu wrote:
> On 5/18/19 4:02 PM, Nicholas Wilson wrote:
>> On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:
>>> Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?
>> 
>> You almost always want to run _your_ tests not your dependencies tests, presumably they have already been tested and therefore do not need to be run.
>
> Wouldn't the dependencies be built separately (i.e. with a different compiler invocation)?

If the dependencies are templated, like large portions of Phobos, then no, they and their unit tests will be compiled together with your code.

When you import a module, the compiler adds its unit tests to the list of all unit tests that need to be run. At the moment the compiler can't know if you're importing your own module (that you want to test) or a third-party one (that you don't care about its unit tests).

Am 19.05.19 um 07:59 schrieb Petar Kirov [ZombineDev]: [...]
> 
> If the dependencies are templated, like large portions of Phobos, then no, they and their unit tests will be compiled together with your code.
> 
> When you import a module, the compiler adds its unit tests to the list of all unit tests that need to be run. At the moment the compiler can't know if you're importing your own module (that you want to test) or a third-party one (that you don't care about its unit tests).

This is not entirely true. Only unittest blocks that are included in the modules that are actually compiled are run. There is one nasty corner case however. Consider the following excample:

--- main.d

import foo;
import std.stdio;

void main()
{
    writeln(add(1, 1));
}

unittest
{
    writeln("unittest in main");
    assert(add(1, 2) == 3);
}

--- foo.d

template add(T)
{
    T add(T a, T b)
    {
        return a + b;
    }

    unittest
    {
        import std.stdio : writeln;

        writeln("unittest in foo.add");
        assert(add(1, 2) == 3);
    }

}

unittest
{
    import std.stdio : writeln;

    writeln("unittest in foo");
    assert(add(1, 2) == 3);
}

When compiling and running with `dmd -unittest -run main.d`, this prints

unittest in main
unittest in foo.add
2

As you can see, the unittest block located in the template is executed, while the one at module level in foo is not. This is because templates are expanded at call site, so from the compilers perspective it looks like the unittest was inside main.d, which is being compiled, so the unittest is also compiled (for this particular instantiation of add).

There have been ideas to fix this by moving the unittests outside the template before expanding it by modifying the AST or something like that (I believe I hear Jonathan talk about that during DConf) but nobody has gotten around tom implementing that yet and it also sound a bit hackish to me...

Am 18.05.19 um 01:31 schrieb Andrei Alexandrescu:
> https://github.com/dlang/druntime/pull/2611

This is a great improvement. Thanks Andrei!

Am 18.05.19 um 19:49 schrieb Adam D. Ruppe:
> [...]
> One of the suggestions back then was to make -unittest only work on
> modules explicitly compiled in (open question: how does this interact
> with the new -i switch added in 2018, after that 2017 PR?), but Atila
> said that breaks the utility __traits(getUnitTests). You (Andrei) at the
> time said that is a good trade off, and a few agreed.
> 
> [...]
This is already implemented and the default behavior. See my other recent post in this thread. The interaction with the -i switch is that all unittests from the dependencies are compiled. This is consistent with the description of the -i switch: "include imported modules in the compilation".