On 2011-10-26 17:40, David Gileadi wrote:
> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
>> On 2011-10-24 22:08, Jonathan M Davis wrote:
>>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>>>> I'm not sure why it just stops after the first failing unittest
>>>> though. What is the point of that 'failed' counter?
>>>
>>> It's a long standing issue that when one unit test fails within a
>>> module, no
>>> more within that module are run (though fortunately, a while back it
>>> was fixed
>>> so that other modules' unit tests will still run). As I recall, there
>>> had to
>>> be a change to the compiler to fix it, but I don't known/remember the
>>> details.
>>> Certainly, the issue still stands.
>>>
>>> - Jonathan M Davis
>>
>> A workaround is to catch AssertErrors, hook it up with some library code
>> and you get a minimal unit test framework:
>> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d
>>
>>
>>
>> Example of usage:
>>
>> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
>
> As an argument for continuing to run tests after one fails, I'm taking a
> TDD class and the instructor asserted that for unit tests you should
> generally only have one or two assertions per test method. His reasoning
> is that when something breaks you immediately know the extent of your
> breakage by counting the number of failed methods. This argument is
> pretty convincing to me.

Well, in my library, if an assert error is thrown in a block (passed to the "it" method), the whole block is canceled and it will continue with the next block. So it's up to the user how the asserts should be laid out.

-- 
/Jacob Carlborg

On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
> On 2011-10-26 17:40, David Gileadi wrote:
> > On 10/25/11 4:04 AM, Jacob Carlborg wrote:
> >> On 2011-10-24 22:08, Jonathan M Davis wrote:
> >>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
> >>>> I'm not sure why it just stops after the first failing unittest though. What is the point of that 'failed' counter?
> >>> 
> >>> It's a long standing issue that when one unit test fails within a
> >>> module, no
> >>> more within that module are run (though fortunately, a while back it
> >>> was fixed
> >>> so that other modules' unit tests will still run). As I recall,
> >>> there
> >>> had to
> >>> be a change to the compiler to fix it, but I don't known/remember
> >>> the
> >>> details.
> >>> Certainly, the issue still stands.
> >>> 
> >>> - Jonathan M Davis
> >> 
> >> A workaround is to catch AssertErrors, hook it up with some library
> >> code
> >> and you get a minimal unit test framework:
> >> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
> >> ester.d
> >> 
> >> 
> >> 
> >> Example of usage:
> >> 
> >> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
> > 
> > As an argument for continuing to run tests after one fails, I'm taking a TDD class and the instructor asserted that for unit tests you should generally only have one or two assertions per test method. His reasoning is that when something breaks you immediately know the extent of your breakage by counting the number of failed methods. This argument is pretty convincing to me.
> 
> Well, in my library, if an assert error is thrown in a block (passed to the "it" method), the whole block is canceled and it will continue with the next block. So it's up to the user how the asserts should be laid out.

It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good).

- Jonathan M Davis

On 10/26/11 3:45 PM, Jonathan M Davis wrote:
> On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
>> On 2011-10-26 17:40, David Gileadi wrote:
>>> On 10/25/11 4:04 AM, Jacob Carlborg wrote:
>>>> On 2011-10-24 22:08, Jonathan M Davis wrote:
>>>>> On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:
>>>>>> I'm not sure why it just stops after the first failing unittest
>>>>>> though. What is the point of that 'failed' counter?
>>>>>
>>>>> It's a long standing issue that when one unit test fails within a
>>>>> module, no
>>>>> more within that module are run (though fortunately, a while back it
>>>>> was fixed
>>>>> so that other modules' unit tests will still run). As I recall,
>>>>> there
>>>>> had to
>>>>> be a change to the compiler to fix it, but I don't known/remember
>>>>> the
>>>>> details.
>>>>> Certainly, the issue still stands.
>>>>>
>>>>> - Jonathan M Davis
>>>>
>>>> A workaround is to catch AssertErrors, hook it up with some library
>>>> code
>>>> and you get a minimal unit test framework:
>>>> https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
>>>> ester.d
>>>>
>>>>
>>>>
>>>> Example of usage:
>>>>
>>>> https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d
>>>
>>> As an argument for continuing to run tests after one fails, I'm taking a
>>> TDD class and the instructor asserted that for unit tests you should
>>> generally only have one or two assertions per test method. His reasoning
>>> is that when something breaks you immediately know the extent of your
>>> breakage by counting the number of failed methods. This argument is
>>> pretty convincing to me.
>>
>> Well, in my library, if an assert error is thrown in a block (passed to
>> the "it" method), the whole block is canceled and it will continue with
>> the next block. So it's up to the user how the asserts should be laid out.
>
> It would be disastrous IMHO to continue to run a unittest block after an
> assert failed - at least in the general case. Too often further assertions
> relied on the state assured by the one that failed, so further failures just
> confuse things and give you too much data to have to sift through. As it
> stands with the built-in unit testing faciliities, you can put each assertion
> in its own unittest block if you really want each assertion to run on its own
> (though until it's fixed so that further unittest blocks within the module run
> after the first failure in that module, it wouldn't do you any good).
>
> - Jonathan M Davis

Agreed--the (rather obscure, sorry) point of my message was that having further unittest blocks run within the module even after one fails is a good idea.  Jacob's test frameworks sounds like a good effort in that direction, but built-in would be even better.

On 2011-10-26 21:45, Jonathan M Davis wrote:
> On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
>> Well, in my library, if an assert error is thrown in a block (passed to
>> the "it" method), the whole block is canceled and it will continue with
>> the next block. So it's up to the user how the asserts should be laid out.
>
> It would be disastrous IMHO to continue to run a unittest block after an
> assert failed - at least in the general case. Too often further assertions
> relied on the state assured by the one that failed, so further failures just
> confuse things and give you too much data to have to sift through. As it
> stands with the built-in unit testing faciliities, you can put each assertion
> in its own unittest block if you really want each assertion to run on its own
> (though until it's fixed so that further unittest blocks within the module run
> after the first failure in that module, it wouldn't do you any good).
>
> - Jonathan M Davis

My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block.

You can never convince me that the other solution is better as long as it doesn't work.

-- 
/Jacob Carlborg

On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:
> On 2011-10-26 21:45, Jonathan M Davis wrote:
> > On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
> >> Well, in my library, if an assert error is thrown in a block (passed
> >> to
> >> the "it" method), the whole block is canceled and it will continue
> >> with
> >> the next block. So it's up to the user how the asserts should be laid
> >> out.>
> > It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good).
> > 
> > - Jonathan M Davis
> 
> My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block.
> 
> You can never convince me that the other solution is better as long as it doesn't work.

The current solution works just fine if you want a unittest block to terminate
after a single assertion failure (as it should IMHO). It's just the running of
subsequent unittest blocks within a module which doesn't work.
And while I'd very much like it to work, I don't personally find it to be all
that much of a problem.

- Jonathan M Davis

On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:
> On 2011-10-26 21:45, Jonathan M Davis wrote:
> > On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:
> >> Well, in my library, if an assert error is thrown in a block (passed
> >> to
> >> the "it" method), the whole block is canceled and it will continue
> >> with
> >> the next block. So it's up to the user how the asserts should be laid
> >> out.>
> > It would be disastrous IMHO to continue to run a unittest block after an assert failed - at least in the general case. Too often further assertions relied on the state assured by the one that failed, so further failures just confuse things and give you too much data to have to sift through. As it stands with the built-in unit testing faciliities, you can put each assertion in its own unittest block if you really want each assertion to run on its own (though until it's fixed so that further unittest blocks within the module run after the first failure in that module, it wouldn't do you any good).
> > 
> > - Jonathan M Davis
> 
> My solution works now and if you want the whole unit test block to end when an assert error is thrown that is possible too. Just call the "it" method with a block once in a unit test block.
> 
> You can never convince me that the other solution is better as long as it doesn't work.

The current solution works just fine if you want a unittest block to terminate
after a single assertion failure (as it should IMHO). It's just the running of
subsequent unittest blocks within a module which doesn't work.
And while I'd very much like it to work, I don't personally find it to be all
that much of a problem.

- Jonathan M Davis