On Friday 26 November 2010 17:17:56 bearophile wrote:
> Walter:
> > Yes, it is. Unit test failures return a non-zero exit code.
> 
> I see, good. But real unit test systems don't just return that value, they give a more visible feedback, like the number of failed/passed tests. One line of text that shows those numbers is a good start.

I'm a firm believer that D unit tests should not change how they fundamentally work at this point. I don't _want_ it to report the number of tests that passed. That information is not at all useful. And if I want to do something like set up something to run dmd periodically and report only when the tests start failing, then the current situation is perfect.

I _do_ think that the unit testing stuff should be expanded to have named unit tests and do whatever is necessary to make it possible for external tools to run the unit tests. Then you could have an external tool that did things like only run specific unit tests and report which tests succeeded and which failed.

So, I do think that it should be possible to build external tools on D's unit testing framework, but I think that it's quite good as it is now, and I wouldn't want to see it drastically changed.

The two changes that I want to see to the current framework are

1. Make it so that every unittest block runs in a module instead of them stopping once one failed. The current situation is better than stopping _all_ unittest blocks once a single test fails, but it's still not granular enough. I believe that this change is planned, but work has to be done to make it possible, and that work hasn't been done yet.

2. Make it possible to name unittest blocks. This would make exceptions that escape unittest blocks _far_ more useful. Names like unittest45 really aren't useful at all. I can't even figure out how it does the numbering. Also, having named unittest blocks would be a necessity for properly allowing external tools to run the unit tests.

I do _not_ want to see D programs printing out anything more than they do when running unit tests. I'm totally open to external tools which do more (it would particularly good for IDEs to be able to run unittest blocks individually), but I do not want to see the basic framework change how it prints feedback on test successes and failures. It works great as it is now.

- Jonathan M Davis

bearophile wrote:
> Walter:
> 
>> Yes, it is. Unit test failures return a non-zero exit code.
> 
> I see, good.

I wish you'd check these things before confidently posting incorrect assertions about how D behaves.

Jonathan M Davis wrote:
> I'm a firm believer that D unit tests should not change how they fundamentally work at this point. I don't _want_ it to report the number of tests that passed. 

That's right. The number that fail is completely useless window dressing.

> That information is not at all useful. And if I want to do something like set up something to run dmd periodically and report only when the tests start failing, then the current situation is perfect.
> 
> I _do_ think that the unit testing stuff should be expanded to have named unit tests and do whatever is necessary to make it possible for external tools to run the unit tests. Then you could have an external tool that did things like only run specific unit tests and report which tests succeeded and which failed.
> 
> So, I do think that it should be possible to build external tools on D's unit testing framework, but I think that it's quite good as it is now, and I wouldn't want to see it drastically changed.
> 
> The two changes that I want to see to the current framework are
> 
> 1. Make it so that every unittest block runs in a module instead of them stopping once one failed. The current situation is better than stopping _all_ unittest blocks once a single test fails, but it's still not granular enough. I believe that this change is planned, but work has to be done to make it possible, and that work hasn't been done yet.

I believe that is the current behavior. For a time, it kept getting broken by changes to druntime, so it may be broken yet again, but that is the way it is supposed to work.


> 2. Make it possible to name unittest blocks. This would make exceptions that escape unittest blocks _far_ more useful. Names like unittest45 really aren't useful at all. I can't even figure out how it does the numbering.

If the line numbers given out are incorrect, those are compiler bugs and I'd like to get them fixed.

> Also, having named unittest blocks would be a necessity for properly allowing external tools to run the unit tests.

I don't really know how that would work.


> I do _not_ want to see D programs printing out anything more than they do when running unit tests. I'm totally open to external tools which do more (it would particularly good for IDEs to be able to run unittest blocks individually), but I do not want to see the basic framework change how it prints feedback on test successes and failures. It works great as it is now.

Yes, and if anyone wants more, a writefln("your message here") works just fine.

On Friday 26 November 2010 18:52:59 Walter Bright wrote:
> Jonathan M Davis wrote:
> > I'm a firm believer that D unit tests should not change how they fundamentally work at this point. I don't _want_ it to report the number of tests that passed.
> 
> That's right. The number that fail is completely useless window dressing.
> 
> > That information is not at all useful. And if I want to do something like set up something to run dmd periodically and report only when the tests start failing, then the current situation is perfect.
> > 
> > I _do_ think that the unit testing stuff should be expanded to have named unit tests and do whatever is necessary to make it possible for external tools to run the unit tests. Then you could have an external tool that did things like only run specific unit tests and report which tests succeeded and which failed.
> > 
> > So, I do think that it should be possible to build external tools on D's unit testing framework, but I think that it's quite good as it is now, and I wouldn't want to see it drastically changed.
> > 
> > The two changes that I want to see to the current framework are
> > 
> > 1. Make it so that every unittest block runs in a module instead of them stopping once one failed. The current situation is better than stopping _all_ unittest blocks once a single test fails, but it's still not granular enough. I believe that this change is planned, but work has to be done to make it possible, and that work hasn't been done yet.
> 
> I believe that is the current behavior. For a time, it kept getting broken by changes to druntime, so it may be broken yet again, but that is the way it is supposed to work.

It's not. This program

import std.stdio;

bool func()
{
    return false;
}

void main()
{
}

unittest
{
    writeln("test 1");
    assert(func());
}

unittest
{
    writeln("test 2");
}

prints

test 1
core.exception.AssertError@test(15): unittest failure
----------------
./test(onAssertErrorMsg+0x34) [0x808c0f4]
./test(onUnittestErrorMsg+0x18) [0x807f3b8]
./test(_d_unittestm+0x22) [0x807d252]
./test(void test.__unittest_fail(int)) [0x807acfa]
./test(void test.__unittest1()) [0x807ac3b]
./test(void test.__modtest()) [0x807ace0]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x807f56c]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41)
[0x807cd7d]
./test(runModuleUnitTests+0x87) [0x807f487]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x807d448]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x807d370]
./test(main+0x96) [0x807d316]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf75b7c76]
./test() [0x807ab51]

The second unittest block is never run. I believe that the last time that this got discussed on the Phobos list, Sean said that some fundamental changes had to be made to either dmd or druntime (I _think_ it was dmd, but I'm not sure) to make it possible to run subsequent unit tests in a module after one of them has failed. If a test in a module fails, then the tests in other modules run, but not the rest of the tests in the module that had the failure.

> > 2. Make it possible to name unittest blocks. This would make exceptions that escape unittest blocks _far_ more useful. Names like unittest45 really aren't useful at all. I can't even figure out how it does the numbering.
> 
> If the line numbers given out are incorrect, those are compiler bugs and I'd like to get them fixed.

Not the line number. I'm talking about the function name that you get stack traces. Take this program for instance:

void func()
{
    throw new Exception("An exception threw.");
}

void main()
{
}

unittest
{
    func();
}

If you run the unit tests, you get

object.Exception: An exception threw.
----------------
./test(void test.func()) [0x80599fe]
./test(void test.__unittest1()) [0x8059a10]
./test(void test.__modtest()) [0x8059a1c]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x805dcdc]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41)
[0x805ba9d]
./test(runModuleUnitTests+0x87) [0x805dbf7]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c168]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c090]
./test(main+0x96) [0x805c036]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf7595c76]
./test() [0x8059921]


Notic __unittest1(). Here, there's only one unit test, so it's not a big deal, but once you have a lot of them, it's not really feasible to associate that function name with a unittest block. If the unittest blocks were named, something like this

unitest(mytest)
{
}

then you could get a stack trace like this

object.Exception: An exception threw.
----------------
./test(void test.func()) [0x80599fe]
./test(void test.__unittest_mytest()) [0x8059a10]
./test(void test.__modtest()) [0x8059a1c]
./test(extern (C) bool core.runtime.runModuleUnitTests()) [0x805dcdc]
./test(_D6object10ModuleInfo7opApplyFMDFKPS6object10ModuleInfoZiZi+0x41)
[0x805ba9d]
./test(runModuleUnitTests+0x87) [0x805dbf7]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c168]
./test(extern (C) int rt.dmain2.main(int, char**)) [0x805c090]
./test(main+0x96) [0x805c036]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf7595c76]
./test() [0x8059921]

With that stack trace, I can easily determine which test threw the exception. It's not a problem with assertions within the test - since they'll give the file and line number of the assertion in the test - but if any function called by the test throws an exception for whatever reason (including assertions in contracts), then you get a stack trace that borders on useless, because you can't easily determine which unittest block threw the exception. With named unittest blocks, that problem can be fixed.

As it stands, I don't even know how the number for the unittest is picked. It does not appear that __unittestX() necessarily corresponds to the Xth unit test in a module lexographically. Named unit tests are really the way that the problem should be fixed.

I'm not saying that naming unittest blocks should be mandatory, but I think that it should be possible. Unnamed unittest blocks result in functions with names like __unittest1() like they do now, whereas named unittest blocks would  result in names like __unittest_testname() where testname is the name of the unittest block. It would make stack traces far easier to deal with, and if external tools are ever going to have any hope of calling unit test functions, then I think that it's a necessity

>  > Also, having named unittest blocks would be a necessity for properly
>  > allowing
> 
> external tools to run the unit tests.
> 
> I don't really know how that would work.

I'm not sure either. But ideally, it would be possible to have an external tool run a program which runs all of the static constructors and whatever else has to be run before the unittest blocks can run, and then specifically run each unittest block that it wants to, in whatever order it wants to, without necessarily running all of them.

For instance, if you use JUnit with Eclipse, you can tell it to specifically run all of the unit tests in a file. Or you can tell it to run only a single unit test. Or you can tell it to run them all. Right now, with D, you can only ever run them all, and you can't really integrate them into an IDE. I think that that should become possible at some point. Unfortunately, however, I don't know enough to say exactly how that would work. I don't even know how it works with JUnit and Eclipse. It is potentially a very important feature though.

> > I do _not_ want to see D programs printing out anything more than they do when running unit tests. I'm totally open to external tools which do more (it would particularly good for IDEs to be able to run unittest blocks individually), but I do not want to see the basic framework change how it prints feedback on test successes and failures. It works great as it is now.
> 
> Yes, and if anyone wants more, a writefln("your message here") works just
> fine.

Yes. It is a bit of work to make all of your unit tests print success or failure, but it's not hard, just tedious. I see no reason to make everyone have to put up with extraneous output just because some people want it to print a list of successes and failures. If you really want that, you can do it yourself. The one part I can think of that you can't do yourself very well is print the total number of successes and failures (unless you print a running total with each test). But it's possible to build printing functionality on top of a framework that only prints out on failure, whereas it's not possible to get rid of printing with a framework that always prints.

- Jonathan M Davis