Lutger Blijdestijn:

> Actually the unix convention is to give exit code 0 as an indicator of success, so there is feedback. It is very usable for scripting.

But currently something like that is not present in the D unittest system.


> But with the change Sean suggested - and I assume an extension point in druntime - there would be enough for a more human friendly tool to be built on top of the current D unittesting system.

We'll see.

Bye,
bearophile

bearophile wrote:

> Lutger Blijdestijn:
> 
>> Actually the unix convention is to give exit code 0 as an indicator of success, so there is feedback. It is very usable for scripting.
> 
> But currently something like that is not present in the D unittest system.

rdmd --main -unittest somemodule.d

Jonathan M Davis wrote:
> On Tuesday, November 16, 2010 13:33:54 bearophile wrote:
>> Jonathan M Davis:
>>> Most of the rest (if not all of it) could indeed be done in a library.
>> I am not sure it could be done nicely too :-)
> 
> That would depend on what you're trying to do. Printing test success or failure is as simple as adding the approprate scope statement to the beginning of each unittest block. A bit tedious perhaps, but not hard.
> 
>>> Right now
>>> unit tests follow the unix convention of saying nothing on success,
>> That's an usability failure. Humans expect feedback, because you can't tell
>> apart "unittests run and succeed" from "unittests not even run". That Unix
>> convention is bad here. And Unix commands sometimes have a -v (verbose)
>> command that gives feedback, while D unittests don't have this option.
> 
> I'm afraid that I have to disagree there. Having all of the successes print out would, in many cases, just be useless output flooding the console. I have no problem with making it possible for unit tests to report success, but I wouldn't want that to be the default. It's quite clear when a test fails, and that's what is necessary in order to fix test failures.
> 
> I can see why a beginner might want the positive feedback that a test has succeeded, but personally, I would just find it annoying. The only real advantage would be that it would indicate where in the unit tests the program was, and that's only particularly important if you have a _lot_ of them and they take a long time to run.

I think:   "%d unit tests passed in %d modules"
would be enough.

On 18-nov-10, at 09:11, Don wrote:

> Jonathan M Davis wrote:
>> On Tuesday, November 16, 2010 13:33:54 bearophile wrote:
>>> Jonathan M Davis:
>>>> Most of the rest (if not all of it) could indeed be done in a library.
>>> I am not sure it could be done nicely too :-)
>> That would depend on what you're trying to do. Printing test success or failure is as simple as adding the approprate scope statement to the beginning of each unittest block. A bit tedious perhaps, but not hard.
>>>> Right now
>>>> unit tests follow the unix convention of saying nothing on success,
>>> That's an usability failure. Humans expect feedback, because you can't tell
>>> apart "unittests run and succeed" from "unittests not even run". That Unix
>>> convention is bad here. And Unix commands sometimes have a -v (verbose)
>>> command that gives feedback, while D unittests don't have this option.
>> I'm afraid that I have to disagree there. Having all of the successes print out would, in many cases, just be useless output flooding the console. I have no problem with making it possible for unit tests to report success, but I wouldn't want that to be the default. It's quite clear when a test fails, and that's what is necessary in order to fix test failures.
>> I can see why a beginner might want the positive feedback that a test has succeeded, but personally, I would just find it annoying. The only real advantage would be that it would indicate where in the unit tests the program was, and that's only particularly important if you have a _lot_ of them and they take a long time to run.
>
> I think:   "%d unit tests passed in %d modules"
> would be enough.

This was already discussed, I think that optimal solution would be to have a testing function a bit like tangos, the testing functions knows how the module is called. Tango one always prints the module, but that is easy to change.

What I use is my own testing framework, in it i have defined as default main function that checks commandline arguments, so that one can for example pass --print-level=error and see only the errors...
See http://dsource.org/projects/blip/wiki/BlipOverview for an example of using it.
This means having a special file to compile, that generates an executable dedicated to testing, but this maps well to how I do tests.
In fact I often keep the tests separated from the code, I even hide them behind templates to avoid excessive template instantiation in some cases because they are large and would slow down the compilation...

The current default unittest function runs very early (before main), so it is not possible to use that and use commandline arguments (which is "correct" because in the current model unittests can be activated for *any* executable, and should not disturb its run).
It should be possible to write a test function that just sets up things for a later real unittest run that starts from main and can parse the commandline arguments, thus solving all these discussions...

Fawzi

On 16/11/2010 21:11, Gide Nwawudu wrote:
> On Tue, 16 Nov 2010 10:54:50 -0800, Jonathan M Davis
> <jmdavisProg@gmx.com>  wrote:
>
>> On Tuesday, November 16, 2010 07:53:01 Sean Kelly wrote:
>>> bearophile Wrote:
>>>> He also gives a quite useful unittest that the student implementation
>>>> must pass, this is a good usage of unittests. The unit test ends like
>>>> this:
>>>>
>>>> ...
>>>>
>>>>      writeln("unit test passed");
>>>>
>>>> }
>>>>
>>>> Indeed, a person needs feedback that the unittests have run (and have
>>>> succeed), I have used similar things in my dlbs1 (but more refined).
>>>> This kind of need for feedback is so natural that something like that
>>>> will be better somehow done on default by D.
>>>
>>> I'd like unit tests to be optionally named: unittest("name").  The rest
>>> could then be done in library code.
>>
>> That would indeed be great. With that done, it could become possible to run unit
>> tests by name (though that would likely mean more changes for the compiler) in
>> an IDE or whatnot. There's at least one bug report suggesting it, IIRC, though I
>> think that it was suggested that they be named without the quotes.
>> unittest(name) should probably translate to something like unittest_name in
>> whatever scope it's in.
>>
>> In any case, it's a good enhancement request that hasn't come to fruition yet.
>> Most of the rest (if not all of it) could indeed be done in a library. Right now
>> unit tests follow the unix convention of saying nothing on success, which I
>> think is best for the current set up - particularly when it's not all that hard
>> to add code yourself which prints out success if you really want it to.
>>
>> - Jonathan M Davis
>
> Already requested, see
> http://d.puremagic.com/issues/show_bug.cgi?id=2749
>
> Gide

I also agree having named unittests might be quite useful. I'm not sure about the nesting though, its seems to me using the D module itself as nesting is perfectly fine.

-- 
Bruno Medeiros - Software Engineer

On Thursday 18 November 2010 04:47:43 Bruno Medeiros wrote:
> On 16/11/2010 21:11, Gide Nwawudu wrote:
> > On Tue, 16 Nov 2010 10:54:50 -0800, Jonathan M Davis
> > 
> > <jmdavisProg@gmx.com>  wrote:
> >> On Tuesday, November 16, 2010 07:53:01 Sean Kelly wrote:
> >>> bearophile Wrote:
> >>>> He also gives a quite useful unittest that the student implementation must pass, this is a good usage of unittests. The unit test ends like this:
> >>>> 
> >>>> ...
> >>>> 
> >>>>      writeln("unit test passed");
> >>>> 
> >>>> }
> >>>> 
> >>>> Indeed, a person needs feedback that the unittests have run (and have succeed), I have used similar things in my dlbs1 (but more refined). This kind of need for feedback is so natural that something like that will be better somehow done on default by D.
> >>> 
> >>> I'd like unit tests to be optionally named: unittest("name").  The rest could then be done in library code.
> >> 
> >> That would indeed be great. With that done, it could become possible to run unit tests by name (though that would likely mean more changes for the compiler) in an IDE or whatnot. There's at least one bug report suggesting it, IIRC, though I think that it was suggested that they be named without the quotes. unittest(name) should probably translate to something like unittest_name in whatever scope it's in.
> >> 
> >> In any case, it's a good enhancement request that hasn't come to fruition yet. Most of the rest (if not all of it) could indeed be done in a library. Right now unit tests follow the unix convention of saying nothing on success, which I think is best for the current set up - particularly when it's not all that hard to add code yourself which prints out success if you really want it to.
> >> 
> >> - Jonathan M Davis
> > 
> > Already requested, see http://d.puremagic.com/issues/show_bug.cgi?id=2749
> > 
> > Gide
> 
> I also agree having named unittests might be quite useful. I'm not sure about the nesting though, its seems to me using the D module itself as nesting is perfectly fine.

I see no point in nesting. Name unit tests, however, would make stack traces from unit tests actually useable (since you'd know which test they were from when they weren't actually assertion failures in the tests themselves which give you a line number in the tests), and it helps open the doors for allowing external tools to access them directly.

- Jonathan M Davis

Fawzi Mohamed <fawzi@gmx.ch> wrote:
> On 18-nov-10, at 09:11, Don wrote:
> 
>> Jonathan M Davis wrote:
>>> On Tuesday, November 16, 2010 13:33:54 bearophile wrote:
>>>> Jonathan M Davis:
>>>>> Most of the rest (if not all of it) could indeed be done in a
> > > > > >>>> library.
>>>> I am not sure it could be done nicely too :-)
>>> That would depend on what you're trying to do. Printing test  >>
> > > success or failure is as simple as adding the approprate scope  >> statement to the beginning of each unittest block. A bit tedious
> > > >> perhaps, but not hard.
>>>>> Right now
>>>>> unit tests follow the unix convention of saying nothing on
> > > > > success,
>>>> That's an usability failure. Humans expect feedback, because you
> > > > >>> can't tell
>>>> apart "unittests run and succeed" from "unittests not even run".
> > > > >>> That Unix
>>>> convention is bad here. And Unix commands sometimes have a -v  >>>
> > > > (verbose)
>>>> command that gives feedback, while D unittests don't have this  >>>
> > > > option.
>>> I'm afraid that I have to disagree there. Having all of the  >>
> > > successes print out would, in many cases, just be useless output
> > > >> flooding the console. I have no problem with making it possible
> > > > > for  >> unit tests to report success, but I wouldn't want that to be the  >> default. It's quite clear when a test fails, and that's what is  >> necessary in order to fix test failures.
>>> I can see why a beginner might want the positive feedback that a  >>
> > > test has succeeded, but personally, I would just find it annoying.
> > >  >> The only real advantage would be that it would indicate where
> > > in  >> the unit tests the program was, and that's only
> > > particularly  >> important if you have a _lot_ of them and they
> > > take a long time to  >> run.
>> 
>> I think:   "%d unit tests passed in %d modules"
>> would be enough.
> 
> This was already discussed, I think that optimal solution would be to
> have a testing function a bit like tangos, the testing functions knows
>  how the module is called. Tango one always prints the module, but
> that  is easy to change.
> 
> What I use is my own testing framework, in it i have defined as
> default main function that checks commandline arguments, so that one
> can for example pass --print-level=error and see only the errors...
> See http://dsource.org/projects/blip/wiki/BlipOverview for an example
> of using it.
> This means having a special file to compile, that generates an
> executable dedicated to testing, but this maps well to how I do tests.
> In fact I often keep the tests separated from the code, I even hide
> them behind templates to avoid excessive template instantiation in
> some cases because they are large and would slow down the
> compilation...
> 
> The current default unittest function runs very early (before main),
> so it is not possible to use that and use commandline arguments (which
>  is "correct" because in the current model unittests can be activated
> for *any* executable, and should not disturb its run).
> It should be possible to write a test function that just sets up
> things for a later real unittest run that starts from main and can
> parse the commandline arguments, thus solving all these discussions...

You can access the commandline args via Runtime.args. This works within unittests.

bearophile wrote:
> Lutger Blijdestijn:
> 
>> Actually the unix convention is to give exit code 0 as an indicator of success, so there is feedback. It is very usable for scripting.
> 
> But currently something like that is not present in the D unittest system.

Yes, it is. Unit test failures return a non-zero exit code.

Walter:

> Yes, it is. Unit test failures return a non-zero exit code.

I see, good. But real unit test systems don't just return that value, they give a more visible feedback, like the number of failed/passed tests. One line of text that shows those numbers is a good start.

Bye,
bearophile

> But real unit test systems don't just return that value, they give a more visible feedback, like the number of failed/passed tests. One line of text that shows those numbers is a good start.

Or maybe that's not the job of a compiler...

Bye,
bearophile