Hi,

Is there any way to count the amount of passed/failed unit tests in a module or a program as a whole? From what I can see, druntime simply invokes a function on each module which triggers all actual tests. If this is all the information that is available, I guess it's impossible, but I'm _very_ open to suggestions/ideas here...

Thanks!

- Alex

core.runtime allows a user-supplied unittest runner. Never used it, but check out http://www.digitalmars.com/d/2.0/phobos/core_runtime.html#moduleUnitTester

Justin


Alex Rønne Petersen wrote:

> Hi,
> 
> Is there any way to count the amount of passed/failed unit tests in a module or a program as a whole? From what I can see, druntime simply invokes a function on each module which triggers all actual tests. If this is all the information that is available, I guess it's impossible, but I'm _very_ open to suggestions/ideas here...
> 
> Thanks!
> 
> - Alex

I'm not sure why it just stops after the first failing unittest though. What is the point of that 'failed' counter?

Ok I can see why, ModuleInfo.unitTest just returns a pointer to a function that calls all unittests on its own. That's not very flexible.

On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> I'm not sure why it just stops after the first failing unittest though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no more within that module are run (though fortunately, a while back it was fixed so that other modules' unit tests will still run). As I recall, there had to be a change to the compiler to fix it, but I don't known/remember the details. Certainly, the issue still stands.

- Jonathan M Davis

On Mon, 24 Oct 2011 22:08:50 +0200, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>> I'm not sure why it just stops after the first failing unittest
>> though. What is the point of that 'failed' counter?
>
> It's a long standing issue that when one unit test fails within a module, no
> more within that module are run (though fortunately, a while back it was fixed
> so that other modules' unit tests will still run). As I recall, there had to
> be a change to the compiler to fix it, but I don't known/remember the details.
> Certainly, the issue still stands.
>
> - Jonathan M Davis

It's also annoying that the runtime quits with a non-zero exit code
when you return false from your unit tester.
Somehow the boolean return became a flag for fail/success rather than for
program continuation.

martin

On 24-10-2011 20:27, Andrej Mitrovic wrote:
> Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
> function that calls all unittests on its own. That's not very
> flexible.

Exactly. I was hoping there would be some obscure workaround, but it doesn't seem like luck is on my side here. :(

- Alex

On 2011-10-24 22:08, Jonathan M Davis wrote:
> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>> I'm not sure why it just stops after the first failing unittest
>> though. What is the point of that 'failed' counter?
>
> It's a long standing issue that when one unit test fails within a module, no
> more within that module are run (though fortunately, a while back it was fixed
> so that other modules' unit tests will still run). As I recall, there had to
> be a change to the compiler to fix it, but I don't known/remember the details.
> Certainly, the issue still stands.
>
> - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d

Example of usage:

https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

-- 
/Jacob Carlborg

On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> On 2011-10-24 22:08, Jonathan M Davis wrote:
>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>>> I'm not sure why it just stops after the first failing unittest
>>> though. What is the point of that 'failed' counter?
>>
>> It's a long standing issue that when one unit test fails within a
>> module, no
>> more within that module are run (though fortunately, a while back it
>> was fixed
>> so that other modules' unit tests will still run). As I recall, there
>> had to
>> be a change to the compiler to fix it, but I don't known/remember the
>> details.
>> Certainly, the issue still stands.
>>
>> - Jonathan M Davis
>
> A workaround is to catch AssertErrors, hook it up with some library code
> and you get a minimal unit test framework:
> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d
>
>
> Example of usage:
>
> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method.  His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods.  This argument is pretty convincing to me.

On Wednesday, October 26, 2011 08:40 David Gileadi wrote:
> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> > On 2011-10-24 22:08, Jonathan M Davis wrote:
> >> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> >>> I'm not sure why it just stops after the first failing unittest though. What is the point of that 'failed' counter?
> >> 
> >> It's a long standing issue that when one unit test fails within a
> >> module, no
> >> more within that module are run (though fortunately, a while back it
> >> was fixed
> >> so that other modules' unit tests will still run). As I recall, there
> >> had to
> >> be a change to the compiler to fix it, but I don't known/remember the
> >> details.
> >> Certainly, the issue still stands.
> >> 
> >> - Jonathan M Davis
> > 
> > A workaround is to catch AssertErrors, hook it up with some library code and you get a minimal unit test framework: https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTest er.d
> > 
> > 
> > Example of usage:
> > 
> > https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
> 
> As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.

The value of that would depend on what exactly the tests are doing - particularly if tests require some setup - but I could see how it would be valuable to essentially list all of the failed assertions rather than just failed unittest blocks. However, I don't think that I'd ever do it that way simply because the code clutter would be far too great. Even if your unit tests consist simply of a bunch of one-liners which are just assertions without any setup, having a unittest block per assertion is going to create a lot of extra code, which will seriously clutter the file. Now, you have the choice to do it either way, which is great, but I don't think that I'd want to be leaning towards one assertion per unittest block. I'd generally favor one unittest block per function, or at least group of related tests.

- Jonathan M Davis