Derek Parnell Wrote:

> >> ** Nesting of unit test blocks.
> > 
> > You mean nesting unittest{  unittest{ .. } ..} What would that be for?
> 
> In conjunction with variable declaration scope within unittest blocks.

doesn't D have scope blocks?

Christopher Wright wrote:
> I'm adding support for running particular subsets with the default test runner right now (test runners can do this, but there's no command line argument to support it).

Dunit now supports this.

Christopher Wright wrote:
> Janderson wrote:
> 
>> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.
> 
> 1. Named unittests.
> There are a few reasons for this:
>     - Your code should be self-documenting. unittest {} tells you nothing.
>     - If a test fails, you want to know which test it is. A backtrace library helps more than a unittest name, I admit, but both are useful.
>     - Naming for unittests integrates better with continuous integration servers like CruiseControl and CCNET -- these expect unittests to have fully qualified names.

I agree that this would be nice.  Perhaps unittest("name") for the syntax?

> 2. A pluggable unittest runner.
> With this, any functionality that unittests lack can be provided. The unittest runner has to be in the runtime, of course.

There's one in druntime.

> 3. Each test should be its own function.
> Currently, all unittest blocks in any module are conglomerated into one function, ModuleInfo.unitTest. ModuleInfo should have a list of name/function pairs. This is required for named unittests, and for reporting errors on a per-test basis.

This would be nice as well.  Recovering from unit tests currently skips all tests in a module after the test that failed, which isn't terribly desirable.

> 4. The unittest runner must be able to have runtime arguments.
> At work, we use test-driven development. We have about 5700 classes, half of which are test classes. That's 7,200 tests currently. They take ten minutes to run. You can't do TDD with ten minute lag for running tests.
> 
> Command line arguments are the easiest way of changing options at runtime. The alternatives are some sort of interactive session with stdio, an options file, or recompiling (which isn't runtime). None of these is particularly good.

Hm... perhaps if argc/argv were passed to the unit test runner from the runtime?  This would leave the choice of switches up to the user.


Sean

Sean Kelly wrote:
> Christopher Wright wrote:
>> Janderson wrote:
>>
>>> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.
>>
>> 1. Named unittests.
>> There are a few reasons for this:
>>     - Your code should be self-documenting. unittest {} tells you nothing.
>>     - If a test fails, you want to know which test it is. A backtrace library helps more than a unittest name, I admit, but both are useful.
>>     - Naming for unittests integrates better with continuous integration servers like CruiseControl and CCNET -- these expect unittests to have fully qualified names.
> 
> I agree that this would be nice.  Perhaps unittest("name") for the syntax?
> 
>> 2. A pluggable unittest runner.
>> With this, any functionality that unittests lack can be provided. The unittest runner has to be in the runtime, of course.
> 
> There's one in druntime.
> 
>> 3. Each test should be its own function.
>> Currently, all unittest blocks in any module are conglomerated into one function, ModuleInfo.unitTest. ModuleInfo should have a list of name/function pairs. This is required for named unittests, and for reporting errors on a per-test basis.
> 
> This would be nice as well.  Recovering from unit tests currently skips all tests in a module after the test that failed, which isn't terribly desirable.

Couldn't there be some hidden global bool, isInUnitTest, which is set when the unit tests start to run, and which is set back to false when the last one ends. assert() could have different behaviour depending on whether it's in the unit testing phase, or not.

Don wrote:
> assert() could have different behaviour depending on whether it's in the unit testing phase, or not.

I agree.  I also think that you may want different types of asserts such as:

assert() //Standard

unitchecked_assert( ) //Only happens for unit checks.  I've found this very useful in my projects.

soft_assert( ) //Doesn't stop program function but reports error.  Users can of course override this to send the output elsewhere.  It can also be turned on for verbose check where it will stop in the debugger when it occurs.  Also this would not show up in unit checks.  Why? Because the unit checks should be able to pass invalid data into functions (ie a missing file name) and have the program continue.  Note softasserts are similar but not the same as exceptions.

Don wrote:
> Sean Kelly wrote:
>> Christopher Wright wrote:
>>> 3. Each test should be its own function.
>>> Currently, all unittest blocks in any module are conglomerated into one function, ModuleInfo.unitTest. ModuleInfo should have a list of name/function pairs. This is required for named unittests, and for reporting errors on a per-test basis.
>>
>> This would be nice as well.  Recovering from unit tests currently skips all tests in a module after the test that failed, which isn't terribly desirable.
> 
> Couldn't there be some hidden global bool, isInUnitTest, which is set when the unit tests start to run, and which is set back to false when the last one ends. assert() could have different behaviour depending on whether it's in the unit testing phase, or not.

druntime has pluggable unit test runners, so that's entirely possible. You couldn't call that function assert, though.

On the other hand, you might find crashing in these situations if assert doesn't abort the current test:

unittest
{
	int[] array = getArray();
	assert (array.length == 1);
	assert (array[0] == something); // array bounds error? wtf?
}

So that's not a good idea.

Don wrote:
> 
> Couldn't there be some hidden global bool, isInUnitTest, which is set when the unit tests start to run, and which is set back to false when the last one ends. assert() could have different behaviour depending on whether it's in the unit testing phase, or not.

assert currently must throw when using DMD because it doesn't generate a fukk stack frame for the call.  For unit tests though, a unit tester could override assert only for the duration of unit testing.


Sean

Sean Kelly wrote:
> Don wrote:
>>
>> Couldn't there be some hidden global bool, isInUnitTest, which is set when the unit tests start to run, and which is set back to false when the last one ends. assert() could have different behaviour depending on whether it's in the unit testing phase, or not.
> 
> assert currently must throw when using DMD because it doesn't generate a fukk stack frame for the call.

I see that annoys the heck out of you :o).

Andrei

Andrei Alexandrescu:
> unittest should in fact be put inside functions too

I agree.

Bye,
bearophile

Andrei Alexandrescu wrote:
> Sean Kelly wrote:
>> Don wrote:
>>>
>>> Couldn't there be some hidden global bool, isInUnitTest, which is set when the unit tests start to run, and which is set back to false when the last one ends. assert() could have different behaviour depending on whether it's in the unit testing phase, or not.
>>
>> assert currently must throw when using DMD because it doesn't generate a fukk stack frame for the call.
> 
> I see that annoys the heck out of you :o).

Oops.  that's what I get for typing that with only one hand and one eye on the computer :-)


Sean