View mode: basic / threaded / horizontal-split · Log in · Help
November 07, 2008
Improving unit tests
> Someone who's a big unittesting fan should write up a proposal on
> this.  I think unittests are neat and all -- I probably don't use them
> as much as I should -- but I don't really know what's so great about
> named unittests or other things people mention that D's unittests
> lack.  I suspect Walter may be in the same boat.  You can't address a
> problem if you don't really understand it.
> --bb

Its funny, I was just thinking last night of starting a new thread about 
exactly that.  For me I only ever use unit tests in a simple way however 
I'd like to learn about move advanced features that D is missing.

I was originally thinking, maybe unit tests shouldn't be part of D to 
allow for innovation. However then I though, what about if D's unit 
tests where extensible though some language syntax?

Questions:
1) What features are missing from D's unit tests that you miss?

2) Should D decouple unittests from the language or should there be 
language changes to allow for for more extensible unit tests?

3) If you chose "more extensible unit tests" what language features 
would be needed to make this happen.

-Joel
November 07, 2008
Re: Improving unit tests
Janderson Wrote:

>  > Someone who's a big unittesting fan should write up a proposal on
>  > this.  I think unittests are neat and all -- I probably don't use them
>  > as much as I should -- but I don't really know what's so great about
>  > named unittests or other things people mention that D's unittests
>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>  > problem if you don't really understand it.
>  > --bb
> 
> Its funny, I was just thinking last night of starting a new thread about 
> exactly that.  For me I only ever use unit tests in a simple way however 
> I'd like to learn about move advanced features that D is missing.
> 
> I was originally thinking, maybe unit tests shouldn't be part of D to 
> allow for innovation. However then I though, what about if D's unit 
> tests where extensible though some language syntax?
> 
> Questions:
> 1) What features are missing from D's unit tests that you miss?

Named unit tests
Reporting individual failures and continuing. Note that you can recover from module testing failures, but not from individual tests.
Compile-time unit tests, especially when making release builds.


> 2) Should D decouple unittests from the language or should there be 
> language changes to allow for for more extensible unit tests?

I like having them built in.


> 3) If you chose "more extensible unit tests" what language features 
> would be needed to make this happen.

I hope an author of one of the DUnit libraries answers this part.
> 
> -Joel
November 07, 2008
Re: Improving unit tests
On Fri, 07 Nov 2008 16:55:14 -0500, Jason House
<jason.james.house@gmail.com> wrote:

>Janderson Wrote:
>
>>  > Someone who's a big unittesting fan should write up a proposal on
>>  > this.  I think unittests are neat and all -- I probably don't use them
>>  > as much as I should -- but I don't really know what's so great about
>>  > named unittests or other things people mention that D's unittests
>>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>>  > problem if you don't really understand it.
>>  > --bb
>> 
>> Its funny, I was just thinking last night of starting a new thread about 
>> exactly that.  For me I only ever use unit tests in a simple way however 
>> I'd like to learn about move advanced features that D is missing.
>> 
>> I was originally thinking, maybe unit tests shouldn't be part of D to 
>> allow for innovation. However then I though, what about if D's unit 
>> tests where extensible though some language syntax?
>> 
>> Questions:
>> 1) What features are missing from D's unit tests that you miss?
>
>Named unit tests
>Reporting individual failures and continuing. Note that you can recover from module testing failures, but not from individual tests.
>Compile-time unit tests, especially when making release builds.
>

Nestable named unittest would be nice. If one group fails, report the
error and move onto the next.

unittest ("XML") {
	unittest("elements") {
	    assert(isValidXml("<aaa />"));
	    assert(isValidXml("<aaa/>"));
	    assert(isValidXml("<aaa></aaa>"));
           ...
	}
	unittest("attributes") {
	    assert(isValidXml("<aaa abc="\x\"/>"));
	    assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
           ...
	}
	unittest("encoding") {
	    assert(encode("hello") is "hello");
	    assert(encode("a > b") == "a &gt; b");
	    ...
	}		
}

> 
>> 2) Should D decouple unittests from the language or should there be 
>> language changes to allow for for more extensible unit tests?
>
>I like having them built in.

Agreed

> 
>> 3) If you chose "more extensible unit tests" what language features 
>> would be needed to make this happen.
>
>I hope an author of one of the DUnit libraries answers this part.
>> 
>> -Joel

Gide
November 08, 2008
Re: Improving unit tests
On Fri, Nov 7, 2008 at 6:56 PM, Gide Nwawudu <gide@btinternet.com> wrote:
> Nestable named unittest would be nice. If one group fails, report the
> error and move onto the next.
>
> unittest ("XML") {
>        unittest("elements") {
>            assert(isValidXml("<aaa />"));
>            assert(isValidXml("<aaa/>"));
>            assert(isValidXml("<aaa></aaa>"));
>            ...
>        }
>        unittest("attributes") {
>            assert(isValidXml("<aaa abc="\x\"/>"));
>            assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
>            ...
>        }
>        unittest("encoding") {
>            assert(encode("hello") is "hello");
>            assert(encode("a > b") == "a &gt; b");
>            ...
>        }
> }

_Nice_.
November 08, 2008
Re: Improving unit tests
On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:

> I'd like to learn about move advanced features that D is missing.

The D language does *not* do unit testing. At best, it provides a place in
which the coder can put their unit testing code and an automated initiation
of the coder's unit test code.


> I was originally thinking, maybe unit tests shouldn't be part of D to 
> allow for innovation. However then I though, what about if D's unit 
> tests where extensible though some language syntax?
> 
> Questions:
> 1) What features are missing from D's unit tests that you miss?

** A variation of the assert statement that does not halt execution upon
finding an error. Instead, it just reports the error and moves on to the
next test.

** A method of identifying the failing test other than by file-name and
line-number. 

** A method by by the number of failures and successes are recorded and
reported at the end of execution. Possibly event made availble in symbolic
form to the coder for whatever purposes they choose. eg. Keeping a database
of unit ttest results over time.

> 2) Should D decouple unittests from the language or should there be 
> language changes to allow for for more extensible unit tests?

Language changes. Uni testing outside of the language is already possible
by using the 'debug' or 'version' facilities, for examle.

> 3) If you chose "more extensible unit tests" what language features 
> would be needed to make this happen.

In addition to those above...
** Being able to run unit tests on a per-module basis without the need of a
main() function.
** Being able to import and declare symbols so that they are only in scope
for their unit test block.
** Nesting of unit test blocks.

-- 
Derek Parnell
Melbourne, Australia
skype: derek.j.parnell
November 08, 2008
Re: Improving unit tests
On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:
> On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:
>
>> I'd like to learn about move advanced features that D is missing.
>
> The D language does *not* do unit testing. At best, it provides a place in
> which the coder can put their unit testing code and an automated initiation
> of the coder's unit test code.
>
>
>> I was originally thinking, maybe unit tests shouldn't be part of D to
>> allow for innovation. However then I though, what about if D's unit
>> tests where extensible though some language syntax?
>>
>> Questions:
>> 1) What features are missing from D's unit tests that you miss?
>
> ** A variation of the assert statement that does not halt execution upon
> finding an error. Instead, it just reports the error and moves on to the
> next test.

Can't that be done by a library function?  I've seen in Python
unittesting the use of "assert_" instead of the built-in "assert".  I
assume it's for the same reason.

> ** A method of identifying the failing test other than by file-name and
> line-number.

Seems like a lib could do this too.
Just some sort of   BeginTests("Module identification") that's tied
into the definition of that assert_()

> ** A method by by the number of failures and successes are recorded and
> reported at the end of execution. Possibly event made availble in symbolic
> form to the coder for whatever purposes they choose. eg. Keeping a database
> of unit ttest results over time.

Can that be done with a lib?

> In addition to those above...
> ** Being able to run unit tests on a per-module basis without the need of a
> main() function.

Yeh, that one is annoying.  You shouldn't need to have a main to run
unit tests.  If there isn't one the compiler should supply an empty
one when unittesting.

> ** Being able to import and declare symbols so that they are only in scope
> for their unit test block.

Agreed.  version(UnitTests) { import module_for_tests; unittest { .. } }

Is annoying.  Being able to stick that import inside the unittest{ ...
} would be much nicer.

> ** Nesting of unit test blocks.

You mean nesting unittest{  unittest{ .. } ..} What would that be for?

But on the subject of nesting, I've noticed it's not possible to put
unittests _inside_ classes and structs.  Or inside functions to test
nested functions.  I guess there is a problem of supplying an
appropriate context there.  Not sure if it can be solved.  For classes
I think it would be ok if you just treat unittest{} like a static
method.

--bb
November 08, 2008
Re: Improving unit tests
On Sat, 8 Nov 2008 10:42:16 +0900, Bill Baxter wrote:

> On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:

>> ** A variation of the assert statement that does not halt execution upon
>> finding an error. Instead, it just reports the error and moves on to the
>> next test.
> 
> Can't that be done by a library function? 
...

> Seems like a lib could do this too.

... 
> Can that be done with a lib?

Sure they can all be done via library functions. No problem with that. In
fact I have a test_equal() and test_true() functions that implement this
functionality.

>> ** Nesting of unit test blocks.
> 
> You mean nesting unittest{  unittest{ .. } ..} What would that be for?

In conjunction with variable declaration scope within unittest blocks.

-- 
Derek Parnell
Melbourne, Australia
skype: derek.j.parnell
November 08, 2008
Re: Improving unit tests
Bill Baxter wrote:
> On Sat, Nov 8, 2008 at 9:48 AM, Derek Parnell <derek@psych.ward> wrote:
>> On Fri, 07 Nov 2008 08:25:04 -0800, Janderson wrote:
>>
>>> I'd like to learn about move advanced features that D is missing.
>> The D language does *not* do unit testing. At best, it provides a place in
>> which the coder can put their unit testing code and an automated initiation
>> of the coder's unit test code.
>>
>>
>>> I was originally thinking, maybe unit tests shouldn't be part of D to
>>> allow for innovation. However then I though, what about if D's unit
>>> tests where extensible though some language syntax?
>>>
>>> Questions:
>>> 1) What features are missing from D's unit tests that you miss?
>> ** A variation of the assert statement that does not halt execution upon
>> finding an error. Instead, it just reports the error and moves on to the
>> next test.
> 
> Can't that be done by a library function?  I've seen in Python
> unittesting the use of "assert_" instead of the built-in "assert".  I
> assume it's for the same reason.
> 
>> ** A method of identifying the failing test other than by file-name and
>> line-number.
> 
> Seems like a lib could do this too.
> Just some sort of   BeginTests("Module identification") that's tied
> into the definition of that assert_()
> 
>> ** A method by by the number of failures and successes are recorded and
>> reported at the end of execution. Possibly event made availble in symbolic
>> form to the coder for whatever purposes they choose. eg. Keeping a database
>> of unit ttest results over time.
> 
> Can that be done with a lib?

I agree with the spirit that we need to look more into what we can do 
with libraries. At this stage in D's development, this group is a 
crucible of new language features, which is good, but oughtn't deter us 
from looking at libraries.

>> In addition to those above...
>> ** Being able to run unit tests on a per-module basis without the need of a
>> main() function.
> 
> Yeh, that one is annoying.  You shouldn't need to have a main to run
> unit tests.  If there isn't one the compiler should supply an empty
> one when unittesting.

At a point I'd added --main to rdmd. It defines a vacuous void main(){} 
to the banana. (I removed it since because I thought it's too minor a 
feature.) When I talked to Walter about that, he said he'd consider 
adding such a switch to dmd.

>> ** Being able to import and declare symbols so that they are only in scope
>> for their unit test block.
> 
> Agreed.  version(UnitTests) { import module_for_tests; unittest { .. } }
>
> Is annoying.  Being able to stick that import inside the unittest{ ...
> } would be much nicer.

For a good time, try version(unittest). Yours truly's brainchild :o).

>> ** Nesting of unit test blocks.
> 
> You mean nesting unittest{  unittest{ .. } ..} What would that be for?
> 
> But on the subject of nesting, I've noticed it's not possible to put
> unittests _inside_ classes and structs.  Or inside functions to test
> nested functions.  I guess there is a problem of supplying an
> appropriate context there.  Not sure if it can be solved.  For classes
> I think it would be ok if you just treat unittest{} like a static
> method.

I agree these should be added to the language, and indeed as static 
blocks. Moreover, unittest should in fact be put inside functions too 
and treated as a "run-once" block. But maybe that goes a bit too far :o).


Andrei
November 08, 2008
Re: Improving unit tests
Janderson wrote:
> Questions:
> 1) What features are missing from D's unit tests that you miss?

- Named tests
- Error reporting
- The ability to list unit tests
- The ability to get different types of output
- Continuous integration support
- Running a subset of the tests
- Not running main after running tests
- Not running tests before main, and not having to recompile to get this 
behavior
- A setup/teardown structure with common code to be run before and after 
a particular set of tests. This one won't happen without a major 
redesign of the builtin unittests, though.

Of those, Dunit has named tests and good error reporting. It has 
continuous integration support. It can list tests without running them. 
I'm adding support for running particular subsets with the default test 
runner right now (test runners can do this, but there's no command line 
argument to support it). And of course it has setup/teardown methods.

Dunit replaces main, so main doesn't run after it, and it doesn't run 
before main.

> 2) Should D decouple unittests from the language or should there be 
> language changes to allow for for more extensible unit tests?

D should decouple unittests from the runtime.

> 3) If you chose "more extensible unit tests" what language features 
> would be needed to make this happen.

1. Named unittests.
There are a few reasons for this:
	- Your code should be self-documenting. unittest {} tells you nothing.
	- If a test fails, you want to know which test it is. A backtrace 
library helps more than a unittest name, I admit, but both are useful.
	- Naming for unittests integrates better with continuous integration 
servers like CruiseControl and CCNET -- these expect unittests to have 
fully qualified names.

2. A pluggable unittest runner.
With this, any functionality that unittests lack can be provided. The 
unittest runner has to be in the runtime, of course.

3. Each test should be its own function.
Currently, all unittest blocks in any module are conglomerated into one 
function, ModuleInfo.unitTest. ModuleInfo should have a list of 
name/function pairs. This is required for named unittests, and for 
reporting errors on a per-test basis.

4. The unittest runner must be able to have runtime arguments.
At work, we use test-driven development. We have about 5700 classes, 
half of which are test classes. That's 7,200 tests currently. They take 
ten minutes to run. You can't do TDD with ten minute lag for running tests.

Command line arguments are the easiest way of changing options at 
runtime. The alternatives are some sort of interactive session with 
stdio, an options file, or recompiling (which isn't runtime). None of 
these is particularly good.
November 08, 2008
Re: Improving unit tests
"Gide Nwawudu" <gide@btinternet.com> wrote in message 
news:i8k9h49nr5fk5l11qu8ujq5p42ck1kdmbd@4ax.com...
> On Fri, 07 Nov 2008 16:55:14 -0500, Jason House
> <jason.james.house@gmail.com> wrote:
>
>>Janderson Wrote:
>>
>>>  > Someone who's a big unittesting fan should write up a proposal on
>>>  > this.  I think unittests are neat and all -- I probably don't use 
>>> them
>>>  > as much as I should -- but I don't really know what's so great about
>>>  > named unittests or other things people mention that D's unittests
>>>  > lack.  I suspect Walter may be in the same boat.  You can't address a
>>>  > problem if you don't really understand it.
>>>  > --bb
>>>
>>> Its funny, I was just thinking last night of starting a new thread about
>>> exactly that.  For me I only ever use unit tests in a simple way however
>>> I'd like to learn about move advanced features that D is missing.
>>>
>>> I was originally thinking, maybe unit tests shouldn't be part of D to
>>> allow for innovation. However then I though, what about if D's unit
>>> tests where extensible though some language syntax?
>>>
>>> Questions:
>>> 1) What features are missing from D's unit tests that you miss?
>>
>>Named unit tests
>>Reporting individual failures and continuing. Note that you can recover 
>>from module testing failures, but not from individual tests.
>>Compile-time unit tests, especially when making release builds.
>>
>
> Nestable named unittest would be nice. If one group fails, report the
> error and move onto the next.
>

Last time I tried to use D's unittests, I had a ton of assert statements. On 
the first run one of them triggered, obviously halting execution, and I 
immediately said to myself "this is stupid" and set out writing a 
"NonFatalAssert()" utility function that displayed assertion failures to 
Stdout, incremented a "numFailures" counter (that's later queried at the 
beginning of main(), along with the flushing of Stdout), and continued 
onward.

So yes, I'd say the ability keep the tests going after one failure 
(preferably with descriptive names and grouping) is pretty important.

>>
>>> 2) Should D decouple unittests from the language or should there be
>>> language changes to allow for for more extensible unit tests?
>>
>>I like having them built in.
>
> Agreed
>

Ditto
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home