> Someone who's a big unittesting fan should write up a proposal on
> this.  I think unittests are neat and all -- I probably don't use them
> as much as I should -- but I don't really know what's so great about
> named unittests or other things people mention that D's unittests
> lack.  I suspect Walter may be in the same boat.  You can't address a
> problem if you don't really understand it.
> --bb

Its funny, I was just thinking last night of starting a new thread about exactly that.  For me I only ever use unit tests in a simple way however I'd like to learn about move advanced features that D is missing.

I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?

Questions:
1) What features are missing from D's unit tests that you miss?

2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?

3) If you chose "more extensible unit tests" what language features would be needed to make this happen.

-Joel

Janderson Wrote:

>  > Someone who's a big unittesting fan should write up a proposal on
>  > this.  I think unittests are neat and all -- I probably don't use them
>  > as much as I should -- but I don't really know what's so great about
>  > named unittests or other things people mention that D's unittests
>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>  > problem if you don't really understand it.
>  > --bb
> 
> Its funny, I was just thinking last night of starting a new thread about exactly that.  For me I only ever use unit tests in a simple way however I'd like to learn about move advanced features that D is missing.
> 
> I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?
> 
> Questions:
> 1) What features are missing from D's unit tests that you miss?

Named unit tests
Reporting individual failures and continuing. Note that you can recover from module testing failures, but not from individual tests.
Compile-time unit tests, especially when making release builds.


> 2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?

I like having them built in.


> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.

I hope an author of one of the DUnit libraries answers this part.
> 
> -Joel

On Fri, 07 Nov 2008 16:55:14 -0500, Jason House <jason.james.house@gmail.com> wrote:

>Janderson Wrote:
>
>>  > Someone who's a big unittesting fan should write up a proposal on
>>  > this.  I think unittests are neat and all -- I probably don't use them
>>  > as much as I should -- but I don't really know what's so great about
>>  > named unittests or other things people mention that D's unittests
>>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>>  > problem if you don't really understand it.
>>  > --bb
>> 
>> Its funny, I was just thinking last night of starting a new thread about exactly that.  For me I only ever use unit tests in a simple way however I'd like to learn about move advanced features that D is missing.
>> 
>> I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?
>> 
>> Questions:
>> 1) What features are missing from D's unit tests that you miss?
>
>Named unit tests
>Reporting individual failures and continuing. Note that you can recover from module testing failures, but not from individual tests.
>Compile-time unit tests, especially when making release builds.
>

Nestable named unittest would be nice. If one group fails, report the error and move onto the next.

unittest ("XML") {
	unittest("elements") {
	    assert(isValidXml("<aaa />"));
	    assert(isValidXml("<aaa/>"));
	    assert(isValidXml("<aaa></aaa>"));
            ...
	}
	unittest("attributes") {
	    assert(isValidXml("<aaa abc="\x\"/>"));
	    assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
            ...
	}
	unittest("encoding") {
	    assert(encode("hello") is "hello");
	    assert(encode("a > b") == "a &gt; b");
	    ...
	}
}

> 
>> 2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?
>
>I like having them built in.

Agreed

> 
>> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.
>
>I hope an author of one of the DUnit libraries answers this part.
>> 
>> -Joel

Gide

On Fri, Nov 7, 2008 at 6:56 PM, Gide Nwawudu <gide@btinternet.com> wrote:
> Nestable named unittest would be nice. If one group fails, report the error and move onto the next.
>
> unittest ("XML") {
>        unittest("elements") {
>            assert(isValidXml("<aaa />"));
>            assert(isValidXml("<aaa/>"));
>            assert(isValidXml("<aaa></aaa>"));
>            ...
>        }
>        unittest("attributes") {
>            assert(isValidXml("<aaa abc="\x\"/>"));
>            assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
>            ...
>        }
>        unittest("encoding") {
>            assert(encode("hello") is "hello");
>            assert(encode("a > b") == "a &gt; b");
>            ...
>        }
> }

_Nice_.

On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:

> I'd like to learn about move advanced features that D is missing.

The D language does *not* do unit testing. At best, it provides a place in which the coder can put their unit testing code and an automated initiation of the coder's unit test code.


> I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?
> 
> Questions:
> 1) What features are missing from D's unit tests that you miss?

** A variation of the assert statement that does not halt execution upon finding an error. Instead, it just reports the error and moves on to the next test.

** A method of identifying the failing test other than by file-name and line-number.

** A method by by the number of failures and successes are recorded and
reported at the end of execution. Possibly event made availble in symbolic
form to the coder for whatever purposes they choose. eg. Keeping a database
of unit ttest results over time.

> 2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?

Language changes. Uni testing outside of the language is already possible by using the 'debug' or 'version' facilities, for examle.

> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.

In addition to those above...
** Being able to run unit tests on a per-module basis without the need of a
main() function.
** Being able to import and declare symbols so that they are only in scope
for their unit test block.
** Nesting of unit test blocks.

-- 
Derek Parnell
Melbourne, Australia
skype: derek.j.parnell

On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:
> On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:
>
>> I'd like to learn about move advanced features that D is missing.
>
> The D language does *not* do unit testing. At best, it provides a place in which the coder can put their unit testing code and an automated initiation of the coder's unit test code.
>
>
>> I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?
>>
>> Questions:
>> 1) What features are missing from D's unit tests that you miss?
>
> ** A variation of the assert statement that does not halt execution upon finding an error. Instead, it just reports the error and moves on to the next test.

Can't that be done by a library function?  I've seen in Python unittesting the use of "assert_" instead of the built-in "assert".  I assume it's for the same reason.

> ** A method of identifying the failing test other than by file-name and line-number.

Seems like a lib could do this too.
Just some sort of   BeginTests("Module identification") that's tied
into the definition of that assert_()

> ** A method by by the number of failures and successes are recorded and reported at the end of execution. Possibly event made availble in symbolic form to the coder for whatever purposes they choose. eg. Keeping a database of unit ttest results over time.

Can that be done with a lib?

> In addition to those above...
> ** Being able to run unit tests on a per-module basis without the need of a
> main() function.

Yeh, that one is annoying.  You shouldn't need to have a main to run unit tests.  If there isn't one the compiler should supply an empty one when unittesting.

> ** Being able to import and declare symbols so that they are only in scope for their unit test block.

Agreed.  version(UnitTests) { import module_for_tests; unittest { .. } }

Is annoying.  Being able to stick that import inside the unittest{ ... } would be much nicer.

> ** Nesting of unit test blocks.

You mean nesting unittest{  unittest{ .. } ..} What would that be for?

But on the subject of nesting, I've noticed it's not possible to put unittests _inside_ classes and structs.  Or inside functions to test nested functions.  I guess there is a problem of supplying an appropriate context there.  Not sure if it can be solved.  For classes I think it would be ok if you just treat unittest{} like a static method.

--bb

On Sat, 8 Nov 2008 10:42:16 +0900, Bill Baxter wrote:

> On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:

>> ** A variation of the assert statement that does not halt execution upon finding an error. Instead, it just reports the error and moves on to the next test.
> 
> Can't that be done by a library function?
 ...

> Seems like a lib could do this too.

 ...
> Can that be done with a lib?

Sure they can all be done via library functions. No problem with that. In
fact I have a test_equal() and test_true() functions that implement this
functionality.

>> ** Nesting of unit test blocks.
> 
> You mean nesting unittest{  unittest{ .. } ..} What would that be for?

In conjunction with variable declaration scope within unittest blocks.

-- 
Derek Parnell
Melbourne, Australia
skype: derek.j.parnell

Bill Baxter wrote:
> On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:
>> On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:
>>
>>> I'd like to learn about move advanced features that D is missing.
>> The D language does *not* do unit testing. At best, it provides a place in
>> which the coder can put their unit testing code and an automated initiation
>> of the coder's unit test code.
>>
>>
>>> I was originally thinking, maybe unit tests shouldn't be part of D to
>>> allow for innovation. However then I though, what about if D's unit
>>> tests where extensible though some language syntax?
>>>
>>> Questions:
>>> 1) What features are missing from D's unit tests that you miss?
>> ** A variation of the assert statement that does not halt execution upon
>> finding an error. Instead, it just reports the error and moves on to the
>> next test.
> 
> Can't that be done by a library function?  I've seen in Python
> unittesting the use of "assert_" instead of the built-in "assert".  I
> assume it's for the same reason.
> 
>> ** A method of identifying the failing test other than by file-name and
>> line-number.
> 
> Seems like a lib could do this too.
> Just some sort of   BeginTests("Module identification") that's tied
> into the definition of that assert_()
> 
>> ** A method by by the number of failures and successes are recorded and
>> reported at the end of execution. Possibly event made availble in symbolic
>> form to the coder for whatever purposes they choose. eg. Keeping a database
>> of unit ttest results over time.
> 
> Can that be done with a lib?

I agree with the spirit that we need to look more into what we can do with libraries. At this stage in D's development, this group is a crucible of new language features, which is good, but oughtn't deter us from looking at libraries.

>> In addition to those above...
>> ** Being able to run unit tests on a per-module basis without the need of a
>> main() function.
> 
> Yeh, that one is annoying.  You shouldn't need to have a main to run
> unit tests.  If there isn't one the compiler should supply an empty
> one when unittesting.

At a point I'd added --main to rdmd. It defines a vacuous void main(){} to the banana. (I removed it since because I thought it's too minor a feature.) When I talked to Walter about that, he said he'd consider adding such a switch to dmd.

>> ** Being able to import and declare symbols so that they are only in scope
>> for their unit test block.
> 
> Agreed.  version(UnitTests) { import module_for_tests; unittest { .. } }
>
> Is annoying.  Being able to stick that import inside the unittest{ ...
> } would be much nicer.

For a good time, try version(unittest). Yours truly's brainchild :o).

>> ** Nesting of unit test blocks.
> 
> You mean nesting unittest{  unittest{ .. } ..} What would that be for?
> 
> But on the subject of nesting, I've noticed it's not possible to put
> unittests _inside_ classes and structs.  Or inside functions to test
> nested functions.  I guess there is a problem of supplying an
> appropriate context there.  Not sure if it can be solved.  For classes
> I think it would be ok if you just treat unittest{} like a static
> method.

I agree these should be added to the language, and indeed as static blocks. Moreover, unittest should in fact be put inside functions too and treated as a "run-once" block. But maybe that goes a bit too far :o).


Andrei

Janderson wrote:
> Questions:
> 1) What features are missing from D's unit tests that you miss?

- Named tests
- Error reporting
- The ability to list unit tests
- The ability to get different types of output
- Continuous integration support
- Running a subset of the tests
- Not running main after running tests
- Not running tests before main, and not having to recompile to get this behavior
- A setup/teardown structure with common code to be run before and after a particular set of tests. This one won't happen without a major redesign of the builtin unittests, though.

Of those, Dunit has named tests and good error reporting. It has continuous integration support. It can list tests without running them. I'm adding support for running particular subsets with the default test runner right now (test runners can do this, but there's no command line argument to support it). And of course it has setup/teardown methods.

Dunit replaces main, so main doesn't run after it, and it doesn't run before main.

> 2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?

D should decouple unittests from the runtime.

> 3) If you chose "more extensible unit tests" what language features would be needed to make this happen.

1. Named unittests.
There are a few reasons for this:
	- Your code should be self-documenting. unittest {} tells you nothing.
	- If a test fails, you want to know which test it is. A backtrace library helps more than a unittest name, I admit, but both are useful.
	- Naming for unittests integrates better with continuous integration servers like CruiseControl and CCNET -- these expect unittests to have fully qualified names.

2. A pluggable unittest runner.
With this, any functionality that unittests lack can be provided. The unittest runner has to be in the runtime, of course.

3. Each test should be its own function.
Currently, all unittest blocks in any module are conglomerated into one function, ModuleInfo.unitTest. ModuleInfo should have a list of name/function pairs. This is required for named unittests, and for reporting errors on a per-test basis.

4. The unittest runner must be able to have runtime arguments.
At work, we use test-driven development. We have about 5700 classes, half of which are test classes. That's 7,200 tests currently. They take ten minutes to run. You can't do TDD with ten minute lag for running tests.

Command line arguments are the easiest way of changing options at runtime. The alternatives are some sort of interactive session with stdio, an options file, or recompiling (which isn't runtime). None of these is particularly good.

"Gide Nwawudu" <gide@btinternet.com> wrote in message news:i8k9h49nr5fk5l11qu8ujq5p42ck1kdmbd@4ax.com...
> On Fri, 07 Nov 2008 16:55:14 -0500, Jason House <jason.james.house@gmail.com> wrote:
>
>>Janderson Wrote:
>>
>>>  > Someone who's a big unittesting fan should write up a proposal on
>>>  > this.  I think unittests are neat and all -- I probably don't use
>>> them
>>>  > as much as I should -- but I don't really know what's so great about
>>>  > named unittests or other things people mention that D's unittests
>>>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>>>  > problem if you don't really understand it.
>>>  > --bb
>>>
>>> Its funny, I was just thinking last night of starting a new thread about exactly that.  For me I only ever use unit tests in a simple way however I'd like to learn about move advanced features that D is missing.
>>>
>>> I was originally thinking, maybe unit tests shouldn't be part of D to allow for innovation. However then I though, what about if D's unit tests where extensible though some language syntax?
>>>
>>> Questions:
>>> 1) What features are missing from D's unit tests that you miss?
>>
>>Named unit tests
>>Reporting individual failures and continuing. Note that you can recover
>>from module testing failures, but not from individual tests.
>>Compile-time unit tests, especially when making release builds.
>>
>
> Nestable named unittest would be nice. If one group fails, report the error and move onto the next.
>

Last time I tried to use D's unittests, I had a ton of assert statements. On the first run one of them triggered, obviously halting execution, and I immediately said to myself "this is stupid" and set out writing a "NonFatalAssert()" utility function that displayed assertion failures to Stdout, incremented a "numFailures" counter (that's later queried at the beginning of main(), along with the flushing of Stdout), and continued onward.

So yes, I'd say the ability keep the tests going after one failure (preferably with descriptive names and grouping) is pretty important.

>>
>>> 2) Should D decouple unittests from the language or should there be language changes to allow for for more extensible unit tests?
>>
>>I like having them built in.
>
> Agreed
>

Ditto