View mode: basic / threaded / horizontal-split · Log in · Help
October 24, 2011
Counting passed/failed unit tests
Hi,

Is there any way to count the amount of passed/failed unit tests in a 
module or a program as a whole? From what I can see, druntime simply 
invokes a function on each module which triggers all actual tests. If 
this is all the information that is available, I guess it's impossible, 
but I'm _very_ open to suggestions/ideas here...

Thanks!

- Alex
October 24, 2011
Re: Counting passed/failed unit tests
core.runtime allows a user-supplied unittest runner. Never used it, but 
check out 
http://www.digitalmars.com/d/2.0/phobos/core_runtime.html#moduleUnitTester

Justin


Alex Rønne Petersen wrote:

> Hi,
> 
> Is there any way to count the amount of passed/failed unit tests in a
> module or a program as a whole? From what I can see, druntime simply
> invokes a function on each module which triggers all actual tests. If
> this is all the information that is available, I guess it's impossible,
> but I'm _very_ open to suggestions/ideas here...
> 
> Thanks!
> 
> - Alex
October 24, 2011
Re: Counting passed/failed unit tests
I'm not sure why it just stops after the first failing unittest
though. What is the point of that 'failed' counter?
October 24, 2011
Re: Counting passed/failed unit tests
Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
function that calls all unittests on its own. That's not very
flexible.
October 24, 2011
Re: Counting passed/failed unit tests
On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> I'm not sure why it just stops after the first failing unittest
> though. What is the point of that 'failed' counter?

It's a long standing issue that when one unit test fails within a module, no 
more within that module are run (though fortunately, a while back it was fixed 
so that other modules' unit tests will still run). As I recall, there had to 
be a change to the compiler to fix it, but I don't known/remember the details. 
Certainly, the issue still stands.

- Jonathan M Davis
October 25, 2011
Re: Counting passed/failed unit tests
On Mon, 24 Oct 2011 22:08:50 +0200, Jonathan M Davis <jmdavisProg@gmx.com>  
wrote:

> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>> I'm not sure why it just stops after the first failing unittest
>> though. What is the point of that 'failed' counter?
>
> It's a long standing issue that when one unit test fails within a  
> module, no
> more within that module are run (though fortunately, a while back it was  
> fixed
> so that other modules' unit tests will still run). As I recall, there  
> had to
> be a change to the compiler to fix it, but I don't known/remember the  
> details.
> Certainly, the issue still stands.
>
> - Jonathan M Davis

It's also annoying that the runtime quits with a non-zero exit code
when you return false from your unit tester.
Somehow the boolean return became a flag for fail/success rather than for
program continuation.

martin
October 25, 2011
Re: Counting passed/failed unit tests
On 24-10-2011 20:27, Andrej Mitrovic wrote:
> Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
> function that calls all unittests on its own. That's not very
> flexible.

Exactly. I was hoping there would be some obscure workaround, but it 
doesn't seem like luck is on my side here. :(

- Alex
October 25, 2011
Re: Counting passed/failed unit tests
On 2011-10-24 22:08, Jonathan M Davis wrote:
> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>> I'm not sure why it just stops after the first failing unittest
>> though. What is the point of that 'failed' counter?
>
> It's a long standing issue that when one unit test fails within a module, no
> more within that module are run (though fortunately, a while back it was fixed
> so that other modules' unit tests will still run). As I recall, there had to
> be a change to the compiler to fix it, but I don't known/remember the details.
> Certainly, the issue still stands.
>
> - Jonathan M Davis

A workaround is to catch AssertErrors, hook it up with some library code 
and you get a minimal unit test framework: 
https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d

Example of usage:

https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

-- 
/Jacob Carlborg
October 26, 2011
Re: Counting passed/failed unit tests
On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> On 2011-10-24 22:08, Jonathan M Davis wrote:
>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>>> I'm not sure why it just stops after the first failing unittest
>>> though. What is the point of that 'failed' counter?
>>
>> It's a long standing issue that when one unit test fails within a
>> module, no
>> more within that module are run (though fortunately, a while back it
>> was fixed
>> so that other modules' unit tests will still run). As I recall, there
>> had to
>> be a change to the compiler to fix it, but I don't known/remember the
>> details.
>> Certainly, the issue still stands.
>>
>> - Jonathan M Davis
>
> A workaround is to catch AssertErrors, hook it up with some library code
> and you get a minimal unit test framework:
> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d
>
>
> Example of usage:
>
> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

As an argument for continuing to run tests after one fails, I'm taking a 
TDD class and the instructor asserted that for unit tests you should 
generally only have one or two assertions per test method.  His 
reasoning is that when something breaks you immediately know the extent 
of your breakage by counting the number of failed methods.  This 
argument is pretty convincing to me.
October 26, 2011
Re: Counting passed/failed unit tests
On Wednesday, October 26, 2011 08:40 David Gileadi wrote:
> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> > On 2011-10-24 22:08, Jonathan M Davis wrote:
> >> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> >>> I'm not sure why it just stops after the first failing unittest
> >>> though. What is the point of that 'failed' counter?
> >> 
> >> It's a long standing issue that when one unit test fails within a
> >> module, no
> >> more within that module are run (though fortunately, a while back it
> >> was fixed
> >> so that other modules' unit tests will still run). As I recall, there
> >> had to
> >> be a change to the compiler to fix it, but I don't known/remember the
> >> details.
> >> Certainly, the issue still stands.
> >> 
> >> - Jonathan M Davis
> > 
> > A workaround is to catch AssertErrors, hook it up with some library code
> > and you get a minimal unit test framework:
> > https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTest
> > er.d
> > 
> > 
> > Example of usage:
> > 
> > https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
> 
> As an argument for continuing to run tests after one fails, I'm taking a
> TDD class and the instructor asserted that for unit tests you should
> generally only have one or two assertions per test method. His
> reasoning is that when something breaks you immediately know the extent
> of your breakage by counting the number of failed methods. This
> argument is pretty convincing to me.

The value of that would depend on what exactly the tests are doing - 
particularly if tests require some setup - but I could see how it would be 
valuable to essentially list all of the failed assertions rather than just 
failed unittest blocks. However, I don't think that I'd ever do it that way 
simply because the code clutter would be far too great. Even if your unit 
tests consist simply of a bunch of one-liners which are just assertions 
without any setup, having a unittest block per assertion is going to create a 
lot of extra code, which will seriously clutter the file. Now, you have the 
choice to do it either way, which is great, but I don't think that I'd want to 
be leaning towards one assertion per unittest block. I'd generally favor one 
unittest block per function, or at least group of related tests.

- Jonathan M Davis
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home