On 08/30/2016 01:12 PM, jmh530 wrote:
> On Tuesday, 30 August 2016 at 16:18:30 UTC, Jacob Carlborg wrote:
>> On 2016-08-30 18:17, Seb wrote:
>>
>>> There is a bit of work on a closely related topic on the wiki:
>>>
>>> https://wiki.dlang.org/DIP83
>>
>> Or https://wiki.dlang.org/DIP50 :)
>
> Or that unit-threaded's @ShouldFail seems very similar to Andrei's
> @MustFail

Noice! -- Andrei

On 2016-08-30 18:31, Dicebot wrote:

> I definitely wouldn't want to use API like you proposed if I was to
> write my own test runner. Minimal common ground which would be cool to
> have is getting range/array of all unittest blocks together with their
> annotations. Anything more than that is optional luxury that specific
> test systems may define.

Basically the only thing that would be different in different frameworks :(

> And any usage of classes in what is supposed to be a base ground API is
> immediate "no" for me :)

Yeah that would be crazy, like threads, fibers, sockets and exceptions ;)

-- 
/Jacob Carlborg

On Tuesday, 30 August 2016 at 16:31:56 UTC, Dicebot wrote:
> On 08/30/2016 07:17 PM, Jacob Carlborg wrote:
>> The reason to put in the druntime is because that's where the existing runner is located.
>> 
>> The advantage of this low level library is that:
>> 
>> * Third party unit test library don't need to reinvent the wheel
>> 
>> * All third party libraries using this low level library would be compatible and can be combined in the same project
>> 
>> * If we would like to, it would be easy to extend the existing runner, like not stopping on the first failure. _Not_ saying that we should
>
> I definitely wouldn't want to use API like you proposed if I was to write my own test runner. Minimal common ground which would be cool to have is getting range/array of all unittest blocks together with their annotations. Anything more than that is optional luxury that specific test systems may define.
>
> And any usage of classes in what is supposed to be a base ground API is immediate "no" for me :)

And never mind that any such low level library would suffer from the same problem unit-threaded did until dub fixed it: D can't reflect on packages so a program must be written that explicitly lists all modules that need to be looked at.

The current situation in druntime with the default runner doesn't work either because all unit tests end up being a single function pointer from the module.

There's a disconnect between what's possible at runtime with ModuleInfo and what's possible at compile-time with __traits(getUnittests). Fortunately for me, running unit-threaded itself as an executable with dub fixed the problem, but in the general case this proposal would suffer from the same problems. Unless we force everyone to use dub.

Atila

On Tuesday, 30 August 2016 at 15:45:26 UTC, Andrei Alexandrescu wrote:
> On 08/30/2016 10:44 AM, Atila Neves wrote:
>> I'd much rather have `assert` be magical or have AST macros to make the
>> syntax for writing tests better than what it is now.
>
> Same here. BTW I'd like unittests that "must not compile" and unittests that "must fail dynamically".
>
> For the former case, the compiler should cooperate:
>
> @incompilable unittest { ... }
> fails if it passes compilation. So the compiler must know about that attribute.

Right now I think you're right and the compiler needs to know. But let me see what I can do about it with the language we have now.

> For the latter case, no change in language is necessary, only in druntime:
>
> @mustfail unittest { ... }

As previously mentioned, unit-threaded has this.

Atila

On Tuesday, 30 August 2016 at 15:45:26 UTC, Andrei Alexandrescu wrote:
> On 08/30/2016 10:44 AM, Atila Neves wrote:
>> I'd much rather have `assert` be magical or have AST macros to make the
>> syntax for writing tests better than what it is now.
>
> Same here. BTW I'd like unittests that "must not compile" and unittests that "must fail dynamically".
>
> For the former case, the compiler should cooperate:
>
> @incompilable unittest { ... }
>
> fails if it passes compilation. So the compiler must know about that attribute.
>

There should be a way to specify the error message (or match it against a regex); otherwise the test could fail accidentally for totally unrelated reasons, and nobody would notice...

On 2016-08-31 12:06, Atila Neves wrote:

> Right now I think you're right and the compiler needs to know. But let
> me see what I can do about it with the language we have now.

Or using AST macros :)

-- 
/Jacob Carlborg

On 08/30/2016 10:40 PM, Jacob Carlborg wrote:
> On 2016-08-30 18:31, Dicebot wrote:
> 
>> I definitely wouldn't want to use API like you proposed if I was to write my own test runner. Minimal common ground which would be cool to have is getting range/array of all unittest blocks together with their annotations. Anything more than that is optional luxury that specific test systems may define.
> 
> Basically the only thing that would be different in different frameworks :(

What would be different? List of unit test blocks available in the program?

>> And any usage of classes in what is supposed to be a base ground API is immediate "no" for me :)
> 
> Yeah that would be crazy, like threads, fibers, sockets and exceptions ;)

_forcing_ usage of threads, fibers, sockets and exceptions in runtime outside of their respective modules would be rather outrageous too. druntime should reasonably simple to port to systems that may not have any of those - and still degrade gracefully.

On 08/31/2016 01:01 PM, Atila Neves wrote:
> And never mind that any such low level library would suffer from the same problem unit-threaded did until dub fixed it: D can't reflect on packages so a program must be written that explicitly lists all modules that need to be looked at.

I don't even think fixing package reflection would truly help here because there exist legit D projects that don't transitively import all modules from `main` and recursive compile-time visiting of all symbols would miss them.

> There's a disconnect between what's possible at runtime with ModuleInfo and what's possible at compile-time with __traits(getUnittests). Fortunately for me, running unit-threaded itself as an executable with dub fixed the problem, but in the general case this proposal would suffer from the same problems. Unless we force everyone to use dub.

To me it feels like mandatory per-requisite for any test runner changes in druntime right now is to:

1) make compiler emit runtime info on test block basis instead of per module 2) store TypeInfo of unittest block attributes as part of that runtime info

On Thursday, 1 September 2016 at 12:06:21 UTC, Dicebot wrote:
> On 08/31/2016 01:01 PM, Atila Neves wrote:
>> And never mind that any such low level library would suffer from the same problem unit-threaded did until dub fixed it: D can't reflect on packages so a program must be written that explicitly lists all modules that need to be looked at.
>
> I don't even think fixing package reflection would truly help here because there exist legit D projects that don't transitively import all modules from `main` and recursive compile-time visiting of all symbols would miss them.
>
>> [snip]

Not a problem, since you can do things like this:

// Prints:
// tuple("CSVException", "IncompleteCellException",
// "HeaderMismatchException", "Malformed", "JSONFloatLiteral",
// "JSONOptions", "JSON_TYPE", "JSONValue", "JSONException")
pragma (msg, ModuleTypes!("std.csv", "std.json"));


/**
 * Params:
 *    Takes a sequence of module name strings and returns a sequence of all types names
 *    defined in those modules
 */
template ModuleTypes(ModuleNames...)
{
    import std.meta : staticMap, aliasSeqOf;

    alias ModuleTypes = aliasSeqOf!(getTypes());

    string[] getTypes()
    {
        import std.meta : AliasSeq;

        string[] result;

        foreach (fqModuleName; ModuleNames)
        {
            mixin ("import " ~ fqModuleName ~ ";");

            foreach (member; __traits(allMembers, mixin(fqModuleName)))
            {
                static if (mixin("is(" ~ member ~ ")"))
                {
                    result ~= member;
                }
            }
        }

        return result;
    }
}

On 09/01/2016 07:17 PM, ZombineDev wrote:
> On Thursday, 1 September 2016 at 12:06:21 UTC, Dicebot wrote:
>> On 08/31/2016 01:01 PM, Atila Neves wrote:
>>> And never mind that any such low level library would suffer from the same problem unit-threaded did until dub fixed it: D can't reflect on packages so a program must be written that explicitly lists all modules that need to be looked at.
>>
>> I don't even think fixing package reflection would truly help here because there exist legit D projects that don't transitively import all modules from `main` and recursive compile-time visiting of all symbols would miss them.
>>
>>> [snip]
> 
> Not a problem, since you can do things like this:

It is exactly _THE_ problem. You can't have the imaginary test runner to reliably find all tests automatically, at least all compiled modules have to be listed explicitly. This is not good, thus I am inclined to call extending RTTI an only viable long-term solution.