Can you not just change to going through each module running the hosts manually? Then you can do what you like before and after each.

If your lib had  a host that started a thread which received messages about a unittest registering itself in other user modules you could run the unittests as  you feel like and you could stick with the -unittest dmd arg. Reason for the thread is you can't control order of module initialization. On 23 Sep 2013 21:30, "Gary Willoughby" <dev@nomad.so> wrote:

> On Monday, 23 September 2013 at 16:40:56 UTC, jostly wrote:
>
>> Let's not end up in that situation, but rather work out a common API to run unit tests, and the D unit test community can be the envy of every other unit tester. :)
>>
>
> You've raised some nice ideas and got me thinking. However, i do think we are missing some way of knowing when unit tests start and stop. I like the built in unittest blocks but it would be nice to have something like:
>
>     beforetests
>     {
>         ...
>     }
>
>     aftertests
>     {
>         ...
>     }
>
> To apply code before and after the unit tests have run. These could be used to setup and the execute the reporting environment? I can't think of a way to do this automatically without these constructs.
>

Sorry in my last mail host should be unittest (swype auto correct errors
:))

On Monday, 23 September 2013 at 16:40:56 UTC, jostly wrote:
> I think it's great to see the D unit testing ecosystem growing. Since it's still relatively small, I think we have a good chance here to create interoperability between the different frameworks.
>
> As I see it, we have:
>
> 1. Running unit tests
>
> This is where D shines with the builting facility for unit tests. However, it suffers a bit from the fact that, if we use assert, it will stop on the first assertion failure, and there is (as far as I've been able to tell) no reliable way to run specific code before or after all the unit tests. If I'm wrong on that assumption, please correct me, that would simplify the spec running for specd.
>
> In specd, the actual code inside the unittest { } sections only collect results, and the reporting is called from a main() supplied by compiling with version "specrunner" set. I haven't checked to see if your dunit do something similar.

In my understanding, D's built-in support for unittests is best suited for test cases that can be expressed as one-liners. When it gets more complicated, we usually use classes. That's what JUnit and TestNG do and that's what dunit does (this one: https://github.com/linkrope/dunit).

For our software, we even separate the 'src' tree from the 'unittest' tree to not distort the coverage results.

> 2. Asserting results
>
> Varies from the builtin assert() to xUnit-like assertEquals() to the more verbose x.must.equal(y) used in specd.
>
> This could easily be standardized by letting all custom asserts throw an AssertError, though I would prefer to use another exception that encapsulates the expected and actual result, to help with bridging to reporting.

It's too easy to use 'assertEquals' wrong. JUnit defines 'assertEquals(expected, actual)' while DUnit defines it the other way around. For JUnit, I've seen too many wrong uses: 'assertEquals(answer, 42)' giving misleading messages "expected ... but got 42".

Even with UFCS, why shouldn't you write '42.assertEqual(actual)'? That's where the "more verbose" 'must' matchers shine: '42.must.equal(actual)' is obviously the wrong way around.

When you have violated contracts, you get 'AssertError' exceptions from deep within your code under test. To fix these errors you may wish for a stack trace. On the other hand, the pretty messages you get for failed test assertions should be enough to fix these failures. In this case, the stack trace would only show the test runner calling the test case.

So: 'must' matchers are better than 'assert...'; and 'AssertError' should not be thrown for failures!

> 3. Reporting results
>
> If we have moved beyond basic assert() and use some kind of unit test runner, then we have the ability to report a summary of run tests, and which (and how many) failed.
>
> This is one area where IDE integration would be very nice, and I would very much prefer it if the different unit test frameworks agreed on one standard unit test runner interface, so that the IDE integration problem becomes one of adapting each IDE to one runner interface, instead of adapting each framework to each IDE.
>
> In my experience from the Java and Scala world, the last point is the biggest. Users expect to be able to run unit tests and see the report in whatever standard way their IDE has. In practice this most often means that various libraries pretend to be JUnit when it comes to running tests, because JUnit is supported by all IDEs.

A few days ago, I added such a reporting to dunit. An XML test report is now available that uses the JUnitReport format. We use Jenkins (formerly known as Hudson) for continuous integration so that we can browse our test results and track failures. Nice!

> Let's not end up in that situation, but rather work out a common API to run unit tests, and the D unit test community can be the envy of every other unit tester. :)

Agreed!

Same hint as for specd: have a look at 'assertOp'!
http://d.puremagic.com/issues/show_bug.cgi?id=4653

alias assertOp!">" assertGreaterThan;
alias assertOp!">=" assertGreaterThanOrEqual;
alias assertOp!"<" assertLessThan;
alias assertOp!"<=" assertLessThanOrEqual;

avoids duplicate code.

Maybe, you can do the same for 'assertStartsWith' and 'assertEndsWith'?

On 2013-09-23 18:40, jostly wrote:

> I think it's great to see the D unit testing ecosystem growing. Since
> it's still relatively small, I think we have a good chance here to
> create interoperability between the different frameworks.
>
> As I see it, we have:
>
> 1. Running unit tests
>
> This is where D shines with the builting facility for unit tests.
> However, it suffers a bit from the fact that, if we use assert, it will
> stop on the first assertion failure, and there is (as far as I've been
> able to tell) no reliable way to run specific code before or after all
> the unit tests. If I'm wrong on that assumption, please correct me, that
> would simplify the spec running for specd.
>
> In specd, the actual code inside the unittest { } sections only collect
> results, and the reporting is called from a main() supplied by compiling
> with version "specrunner" set. I haven't checked to see if your dunit do
> something similar.

Agree. I only see the unittest blocks as a place to but the asserts, since it's not possible to put them at module level.

It's possible to implement you're own unit test handler. See:

https://github.com/D-Programming-Language/druntime/blob/master/src/core/runtime.d#L290

It's also possibly to set the assert handler:

https://github.com/D-Programming-Language/druntime/blob/master/src/core/exception.d#L368

But you most likely want to throw some kind of exception anyway. Because if an assertion is triggered in a unit test block you most likely want to end that unit test block, immediately.

-- 
/Jacob Carlborg

On Wednesday, 25 September 2013 at 06:45:12 UTC, Jacob Carlborg wrote:
> It's possible to implement you're own unit test handler. See:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/core/runtime.d#L290

This changes everything! Thanks for that pointer, it really was the missing link between what I want to do and what I thought was possible.

> But you most likely want to throw some kind of exception anyway. Because if an assertion is triggered in a unit test block you most likely want to end that unit test block, immediately.

Maybe... in my mind it depends on whether there are more than one test per unit test block, because I really like each failing test to be enumerated as well. But I can see the benefit - that would allow me to have tests directly in the unit test block, instead of wrapped inside delegates like it's done in specd. Ideally, a generic runner would support both immediate and deferred tests.

On Wednesday, 25 September 2013 at 19:50:47 UTC, jostly wrote:
> Maybe... in my mind it depends on whether there are more than one test per unit test block, because I really like each failing test to be enumerated as well. But I can see the benefit - that would allow me to have tests directly in the unit test block, instead of wrapped inside delegates like it's done in specd. Ideally, a generic runner would support both immediate and deferred tests.

UDAs + recent trait to get all unit-tests during compile-time really favors instead having lot of small independent annotated unit-test blocks.

On Wednesday, 25 September 2013 at 06:45:12 UTC, Jacob Carlborg wrote:
> It's possible to implement you're own unit test handler. See:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/core/runtime.d#L290
>
> It's also possibly to set the assert handler:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/core/exception.d#L368
>
> But you most likely want to throw some kind of exception anyway. Because if an assertion is triggered in a unit test block you most likely want to end that unit test block, immediately.

That looks interesting but the unittester handler seems to run instead of the unittest blocks and the assert handler property is private.

On Wednesday, 25 September 2013 at 20:18:57 UTC, Gary Willoughby wrote:

> That looks interesting but the unittester handler seems to run instead of the unittest blocks

I'm not exactly sure what you mean. But this is how it works. DMD turns each unit test block into a function. Then DMD creates a single function for each module, which will call all these unit test functions. If you access the "unitTest" [1] property of a ModuleInfo you will get the unit test runner and not the individual unit test functions.

To access the individual unit test functions you can use the getUnitTests trait, available in git HEAD:

https://github.com/D-Programming-Language/dlang.org/pull/366

> and the assert handler property is private.

There are property functions at line 374 and below to set the assert handler.

[1] https://github.com/D-Programming-Language/druntime/blob/master/src/object.di#L284

--
/Jacob Carlborg

On 2013-09-25 21:55, Dicebot wrote:

> UDAs + recent trait to get all unit-tests during compile-time really
> favors instead having lot of small independent annotated unit-test blocks.

If you have more than one test per unit test block you always need to run them together, and in the declared order. Example:

unittest
{
    @test("foo")
    {
        assert(1 == 1);
    }

    @test("bar")
    {
        assert(1 == 2);
    }
}

You cannot run "foo" separated from "bar". They all will always run together. You also cannot run "bar" before running "foo". Running tests like this makes it very easy to introduce order dependencies between the tests.

-- 
/Jacob Carlborg