View mode: basic / threaded / horizontal-split · Log in · Help
November 08, 2008
Re: Improving unit tests
Derek Parnell Wrote:

> >> ** Nesting of unit test blocks.
> > 
> > You mean nesting unittest{  unittest{ .. } ..} What would that be for?
> 
> In conjunction with variable declaration scope within unittest blocks.

doesn't D have scope blocks?
November 08, 2008
Re: Improving unit tests
Christopher Wright wrote:
> I'm adding support for running particular subsets with the default test 
> runner right now (test runners can do this, but there's no command line 
> argument to support it).

Dunit now supports this.
November 08, 2008
Re: Improving unit tests
Christopher Wright wrote:
> Janderson wrote:
> 
>> 3) If you chose "more extensible unit tests" what language features 
>> would be needed to make this happen.
> 
> 1. Named unittests.
> There are a few reasons for this:
>     - Your code should be self-documenting. unittest {} tells you nothing.
>     - If a test fails, you want to know which test it is. A backtrace 
> library helps more than a unittest name, I admit, but both are useful.
>     - Naming for unittests integrates better with continuous integration 
> servers like CruiseControl and CCNET -- these expect unittests to have 
> fully qualified names.

I agree that this would be nice.  Perhaps unittest("name") for the syntax?

> 2. A pluggable unittest runner.
> With this, any functionality that unittests lack can be provided. The 
> unittest runner has to be in the runtime, of course.

There's one in druntime.

> 3. Each test should be its own function.
> Currently, all unittest blocks in any module are conglomerated into one 
> function, ModuleInfo.unitTest. ModuleInfo should have a list of 
> name/function pairs. This is required for named unittests, and for 
> reporting errors on a per-test basis.

This would be nice as well.  Recovering from unit tests currently skips 
all tests in a module after the test that failed, which isn't terribly 
desirable.

> 4. The unittest runner must be able to have runtime arguments.
> At work, we use test-driven development. We have about 5700 classes, 
> half of which are test classes. That's 7,200 tests currently. They take 
> ten minutes to run. You can't do TDD with ten minute lag for running tests.
> 
> Command line arguments are the easiest way of changing options at 
> runtime. The alternatives are some sort of interactive session with 
> stdio, an options file, or recompiling (which isn't runtime). None of 
> these is particularly good.

Hm... perhaps if argc/argv were passed to the unit test runner from the 
runtime?  This would leave the choice of switches up to the user.


Sean
November 09, 2008
Re: Improving unit tests
Sean Kelly wrote:
> Christopher Wright wrote:
>> Janderson wrote:
>>
>>> 3) If you chose "more extensible unit tests" what language features 
>>> would be needed to make this happen.
>>
>> 1. Named unittests.
>> There are a few reasons for this:
>>     - Your code should be self-documenting. unittest {} tells you 
>> nothing.
>>     - If a test fails, you want to know which test it is. A backtrace 
>> library helps more than a unittest name, I admit, but both are useful.
>>     - Naming for unittests integrates better with continuous 
>> integration servers like CruiseControl and CCNET -- these expect 
>> unittests to have fully qualified names.
> 
> I agree that this would be nice.  Perhaps unittest("name") for the syntax?
> 
>> 2. A pluggable unittest runner.
>> With this, any functionality that unittests lack can be provided. The 
>> unittest runner has to be in the runtime, of course.
> 
> There's one in druntime.
> 
>> 3. Each test should be its own function.
>> Currently, all unittest blocks in any module are conglomerated into 
>> one function, ModuleInfo.unitTest. ModuleInfo should have a list of 
>> name/function pairs. This is required for named unittests, and for 
>> reporting errors on a per-test basis.
> 
> This would be nice as well.  Recovering from unit tests currently skips 
> all tests in a module after the test that failed, which isn't terribly 
> desirable.

Couldn't there be some hidden global bool, isInUnitTest, which is set 
when the unit tests start to run, and which is set back to false when 
the last one ends. assert() could have different behaviour depending on 
whether it's in the unit testing phase, or not.
November 09, 2008
Re: Improving unit tests
Don wrote:
> assert() could have different behaviour depending on 
> whether it's in the unit testing phase, or not.

I agree.  I also think that you may want different types of asserts such as:

assert() //Standard

unitchecked_assert( ) //Only happens for unit checks.  I've found this 
very useful in my projects.

soft_assert( ) //Doesn't stop program function but reports error.  Users 
can of course override this to send the output elsewhere.  It can also 
be turned on for verbose check where it will stop in the debugger when 
it occurs.  Also this would not show up in unit checks.  Why? Because 
the unit checks should be able to pass invalid data into functions (ie a 
missing file name) and have the program continue.  Note softasserts are 
similar but not the same as exceptions.
November 09, 2008
Re: Improving unit tests
Don wrote:
> Sean Kelly wrote:
>> Christopher Wright wrote:
>>> 3. Each test should be its own function.
>>> Currently, all unittest blocks in any module are conglomerated into 
>>> one function, ModuleInfo.unitTest. ModuleInfo should have a list of 
>>> name/function pairs. This is required for named unittests, and for 
>>> reporting errors on a per-test basis.
>>
>> This would be nice as well.  Recovering from unit tests currently 
>> skips all tests in a module after the test that failed, which isn't 
>> terribly desirable.
> 
> Couldn't there be some hidden global bool, isInUnitTest, which is set 
> when the unit tests start to run, and which is set back to false when 
> the last one ends. assert() could have different behaviour depending on 
> whether it's in the unit testing phase, or not.

druntime has pluggable unit test runners, so that's entirely possible. 
You couldn't call that function assert, though.

On the other hand, you might find crashing in these situations if assert 
doesn't abort the current test:

unittest
{
	int[] array = getArray();
	assert (array.length == 1);
	assert (array[0] == something); // array bounds error? wtf?
}

So that's not a good idea.
November 09, 2008
Re: Improving unit tests
Don wrote:
> 
> Couldn't there be some hidden global bool, isInUnitTest, which is set 
> when the unit tests start to run, and which is set back to false when 
> the last one ends. assert() could have different behaviour depending on 
> whether it's in the unit testing phase, or not.

assert currently must throw when using DMD because it doesn't generate a 
fukk stack frame for the call.  For unit tests though, a unit tester 
could override assert only for the duration of unit testing.


Sean
November 09, 2008
Re: Improving unit tests
Sean Kelly wrote:
> Don wrote:
>>
>> Couldn't there be some hidden global bool, isInUnitTest, which is set 
>> when the unit tests start to run, and which is set back to false when 
>> the last one ends. assert() could have different behaviour depending 
>> on whether it's in the unit testing phase, or not.
> 
> assert currently must throw when using DMD because it doesn't generate a 
> fukk stack frame for the call.

I see that annoys the heck out of you :o).

Andrei
November 09, 2008
Re: Improving unit tests
Andrei Alexandrescu:
> unittest should in fact be put inside functions too

I agree.

Bye,
bearophile
November 09, 2008
Re: Improving unit tests
Andrei Alexandrescu wrote:
> Sean Kelly wrote:
>> Don wrote:
>>>
>>> Couldn't there be some hidden global bool, isInUnitTest, which is set 
>>> when the unit tests start to run, and which is set back to false when 
>>> the last one ends. assert() could have different behaviour depending 
>>> on whether it's in the unit testing phase, or not.
>>
>> assert currently must throw when using DMD because it doesn't generate 
>> a fukk stack frame for the call.
> 
> I see that annoys the heck out of you :o).

Oops.  that's what I get for typing that with only one hand and one eye 
on the computer :-)


Sean
1 2 3
Top | Discussion index | About this forum | D home