On Saturday, December 31, 2011 18:57:14 Andrew Wiley wrote:
> On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis <jmdavisProg@gmx.com>
wrote:
> > On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
> >> On 2011-12-31 11:37, Jonathan M Davis wrote:
> >> > On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
> >> >>> I think that the AssertError's message (which includes the
> >> >>> file and
> >> >>> line number of the failure) and its stack trace are plenty.
> >> >>> It's
> >> >>> exactly what you need and nothing else.
> >> >>> 
> >> >>> - Jonathan M Davis
> >> >> 
> >> >> I want to have such a summary.
> >> > 
> >> > I don't see any reason to put that in the standard library.
> >> > There's
> >> > nothing wrong with 3rd party solutions which give additional
> >> > functionality, but D's unit test framework is designed to be
> >> > minimilistic, and I don't think that adding anything beyond what
> >> > it
> >> > does now in terms of summary makes any sense. The only major issue
> >> > in
> >> > that regard IMHO is the fact that no further unittest blocks
> >> > within a
> >> > module are run after one fails. Even if it did, I still don't
> >> > think
> >> > that a fancier summary would be worth having - especially in the
> >> > standard library.
> >> 
> >> BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished?
> > 
> > My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved.
> 
> The counterargument is that the language doesn't really provide a framework - it actually provides anonymous parameterless global functions that will be run before main is invoked if code is compiled with -unittest. That isn't considered a framework in any language I've ever used, but it adds just enough functionality to allow a well-integrated fully-featured library solution. Would making such a library solution part of the standard library really be a problem? I'm mostly ambivalent on this issue because I haven't had time to look closely at the proposed framework, but your argument seems to be that all unittesting functionality needs to be built into the language. I don't think that should be necessary or required.

For the most part, I don't think that we need anything more. It should be fixed so that all unittest blocks within a module are run even if one fails. This should be fixed in the compiler and runtime rather than having to hack it together with a library solution, since it's supposed to work that way in the first place but doesn't because of some changes that need to be made to the compiler (some hooks of some kind are needed IIRC). Named unit tests would also be nice, but it would be _far_ cleaner to add those to the language than try and add them with a library solution (e.g. unittest(test_name){}). It would also be nice if it were possible to hook in something that would run specific unit tests, but again, such hooks would then need to be provided by the compiler and/or runtime - unless you want to simply create a function for each unit test and have each unittest block call a specific function? That's an ugly solution IMHO.

All of those changes would be fairly minor in terms of how they affect how unit tests function and would be far cleaner IMHO to have in the language. I think that on some level, the unit test framework in the language has failed if we have to add library solutions on top of it to get such basic functionality.

> > As for a "nice report," I don't see anything wrong with just using the stack traces (which include the file, line number, and error message of the assertion failure). That's all the information that's needed. Anything else is superfluous IMHO. Now, if there were something nicer that could be generally agreed upon and added to druntime such that the standard unit test facilities used it, then fine. I don't see any point to it, but at least in that case, the standard library is still using the standard unit test framework. What I really don't want to see is Phobos essentially building a new unit test framework on top of the existing one. Any issues that need to be addressed with the unit test framework for the standard library should be addressed in the standard framework. Any additional framework stuff should be left to 3rd parties.
> 
> As I said above, I wouldn't consider what we have to be a framework,
> but it's definitely enough to build an excellent library solution on
> top of.
> As for a report, the problem is that an assertion error isn't what you
> want when you're running, say, a continuous integration server (or,
> say, a pull request tester). What you really want is a detailed
> explanation of what unittests broke, what the tests were testing, and
> how the result differed from what was expected. You want to be able to
> have a reasonable idea of what went wrong *without* having to look at
> someone else's code and figure out exactly what they're testing every
> time.

I don't really get what you'd need beyond knowing which test failed. The AssertErrors tell you that. It would definitely be nicer if the unittest blocks had proper names to go with them, but they're not necessary for knowing which test failed. The file and line number give you that. I don't see how you're going to know what actually failed unless you actually look at the test. If the tests have names and are well named, then that gives you a better idea of what isn't working before you look at the code, but ultimately, you have to look at the code to know what went wrong. And if you can name unittest blocks, then that problem is solved anyway, since then you'd have a name for the unit test failure to give in any report that you wanted to create.

- Jonathan M Davis

Jonathan M Davis wrote:

> I think
> that on some level, the unit test framework in the language has failed if
> we have to add library solutions on top of it to get such basic
> functionality.

That's what we are saying: The unit test framework fails for us and a library solution is perfectly possible, even one that lets everyone use the build in functionality if he wants.

You don't find it useful, but others do. I'd say: many others do. If something is considered useful by many it should have a place in the standard library.

On 2011-12-31 21:56, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
>> BTW, what would be so wrong if the unit tests for the standard library
>> displayed a nice report when finished?
>
> My primary issue here is that I don't think that we should be adding stuff to
> Phobos which is essentially a new unit test framework on top of the built in
> one. If 3rd party stuff wants to do that. Fine. But the standard library should
> use the standard facilities. If the standard facilities aren't sufficient, then
> they should be improved.

First, that's how programming works. You build new abstractions on top of existing ones. Second, I would called the built in support for unit testing in D a "unit test framework". If add a unit test framework to Phobos it would be standard facilities and it wouldn't be a problem.

> As for a "nice report," I don't see anything wrong with just using the stack
> traces (which include the file, line number, and error message of the assertion
> failure). That's all the information that's needed. Anything else is
> superfluous IMHO. Now, if there were something nicer that could be generally
> agreed upon and added to druntime such that the standard unit test facilities
> used it, then fine. I don't see any point to it, but at least in that case, the
> standard library is still using the standard unit test framework. What I
> really don't want to see is Phobos essentially building a new unit test
> framework on top of the existing one. Any issues that need to be addressed
> with the unit test framework for the standard library should be addressed in
> the standard framework. Any additional framework stuff should be left to 3rd
> parties.
>
> - Jonathan M Davis

What's wrong with being able to run the unit tests from your editor, have the unit test framework output HTML (or similar), displayed in your editor and then you can click on links in the stack trace to get to the source code. If you don't see why that's useful that we can just end this discussion now.

-- 
/Jacob Carlborg

On 2011-12-31 21:57, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
>> Yes but what happens when there are many failed tests, i.e. may
>> AssertErrors that have been thrown? It will just print all after each
>> other and you have to count them yourself if you want to know how many
>> failed tests there are?
>
> What does the number of failures really matter? You just need to know which
> ones failed and where. The AssertErrors give you that.
>
> - Jonathan M Davis

It can be nice for statistics. And again, how will it be displayed, just all AssertErrors after each other?

-- 
/Jacob Carlborg

On 2011-12-31 22:01, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 16:04:12 Jacob Carlborg wrote:
>> It would be possible to implement named unit tests only in library code.
>> It would not have as nice syntax as if it was implemented in the
>> language but still possible.
>>
>> In Ruby on Rails I run single unit tests all the time. Why would I run
>> all the unit tests, which can take five minutes, when I just can run one
>> unit test and it takes just one second?
>>
>> When your doing test/behavior driven development (T/BDD) it's certainly
>> nice to be able to run single unit tests, because you run it all the time.
>
> Yes. I agree that it would be nice, but for it to be done at all cleanly, the
> language, compiler, and druntime need to be improved to make it possible.
> However, at least syntactically, such changes should be completely backwards
> compatible, so they can be added at a future date. Regardless, I don't think
> that it's a problem that Phobos should be trying to solve.
>
> - Jonathan M Davis

Ok, if you would rather have all this in the language I would say no do that. But I know other people in the community that usually prefer to do a library solution if possible.

-- 
/Jacob Carlborg

On 2012-01-01 01:57, Andrew Wiley wrote:
> On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis<jmdavisProg@gmx.com>  wrote:
>> On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
>>> On 2011-12-31 11:37, Jonathan M Davis wrote:
>>>> On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
>>>>>> I think that the AssertError's message (which includes the file and
>>>>>> line number of the failure) and its stack trace are plenty. It's
>>>>>> exactly what you need and nothing else.
>>>>>>
>>>>>> - Jonathan M Davis
>>>>>
>>>>> I want to have such a summary.
>>>>
>>>> I don't see any reason to put that in the standard library. There's
>>>> nothing wrong with 3rd party solutions which give additional
>>>> functionality, but D's unit test framework is designed to be
>>>> minimilistic, and I don't think that adding anything beyond what it
>>>> does now in terms of summary makes any sense. The only major issue in
>>>> that regard IMHO is the fact that no further unittest blocks within a
>>>> module are run after one fails. Even if it did, I still don't think
>>>> that a fancier summary would be worth having - especially in the
>>>> standard library.
>>>
>>> BTW, what would be so wrong if the unit tests for the standard library
>>> displayed a nice report when finished?
>>
>> My primary issue here is that I don't think that we should be adding stuff to
>> Phobos which is essentially a new unit test framework on top of the built in
>> one. If 3rd party stuff wants to do that. Fine. But the standard library should
>> use the standard facilities. If the standard facilities aren't sufficient, then
>> they should be improved.
>
> The counterargument is that the language doesn't really provide a
> framework - it actually provides anonymous parameterless global
> functions that will be run before main is invoked if code is compiled
> with -unittest. That isn't considered a framework in any language I've
> ever used, but it adds just enough functionality to allow a
> well-integrated fully-featured library solution. Would making such a
> library solution part of the standard library really be a problem?
> I'm mostly ambivalent on this issue because I haven't had time to look
> closely at the proposed framework, but your argument seems to be that
> all unittesting functionality needs to be built into the language. I
> don't think that should be necessary or required.
>
>
>> As for a "nice report," I don't see anything wrong with just using the stack
>> traces (which include the file, line number, and error message of the assertion
>> failure). That's all the information that's needed. Anything else is
>> superfluous IMHO. Now, if there were something nicer that could be generally
>> agreed upon and added to druntime such that the standard unit test facilities
>> used it, then fine. I don't see any point to it, but at least in that case, the
>> standard library is still using the standard unit test framework. What I
>> really don't want to see is Phobos essentially building a new unit test
>> framework on top of the existing one. Any issues that need to be addressed
>> with the unit test framework for the standard library should be addressed in
>> the standard framework. Any additional framework stuff should be left to 3rd
>> parties.
>
> As I said above, I wouldn't consider what we have to be a framework,
> but it's definitely enough to build an excellent library solution on
> top of.
> As for a report, the problem is that an assertion error isn't what you
> want when you're running, say, a continuous integration server (or,
> say, a pull request tester). What you really want is a detailed
> explanation of what unittests broke, what the tests were testing, and
> how the result differed from what was expected. You want to be able to
> have a reasonable idea of what went wrong *without* having to look at
> someone else's code and figure out exactly what they're testing every
> time.

Exactly, I agree.

-- 
/Jacob Carlborg

On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
> Ok, if you would rather have all this in the language I would say no do that. But I know other people in the community that usually prefer to do a library solution if possible.

If all we're talking about is named unit tests and running all of the unittest blocks within a module even if one fails, those aren't huge changes to the language. Both have been discussed before, and the only reason that the latter hasn't been implemented yet is that it requires changes to the compiler. So, I don't see any reason to make those a library solution. If we're talking something massively more complicated than that, then yes, a library solution starts making more sense. I also think that it then doesn't necessarily make sense to put it in the standard library.

If the issue is how the tests print out on failure, that could probably be done in the language as well, depending on what you were talking about changing. If you want to do something massively complicated, however, then you'd probably have to come up with a library solution, but again, I see no reason to have it in the standard library if you're doing anything fancy with how the test failures print out.

- Jonathan M Davis

On Sunday, January 01, 2012 15:32:34 Jacob Carlborg wrote:
> On 2011-12-31 21:57, Jonathan M Davis wrote:
> > On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
> >> Yes but what happens when there are many failed tests, i.e. may AssertErrors that have been thrown? It will just print all after each other and you have to count them yourself if you want to know how many failed tests there are?
> > 
> > What does the number of failures really matter? You just need to know which ones failed and where. The AssertErrors give you that.
> > 
> > - Jonathan M Davis
> 
> It can be nice for statistics. And again, how will it be displayed, just all AssertErrors after each other?

Yes.

- Jonathan M Davis

On Sunday, January 01, 2012 15:31:18 Jacob Carlborg wrote:
> What's wrong with being able to run the unit tests from your editor, have the unit test framework output HTML (or similar), displayed in your editor and then you can click on links in the stack trace to get to the source code. If you don't see why that's useful that we can just end this discussion now.

If you want fancier unit test facilities in your own code. Fine. I just don't think that they should be in the standard library. I think that on the whole, D's built-in unit test framework works fine as it is (barring a few tweaks such as making it so that all unittest blocks within a module run), and that further additions are a needless complication to the standard unit testing facilities and better left to 3rd party solutions where you can do whatever you want with them.

- Jonathan M Davis

On 2012-01-02 00:14, Jonathan M Davis wrote:
> On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
>> Ok, if you would rather have all this in the language I would say no do
>> that. But I know other people in the community that usually prefer to do
>> a library solution if possible.
>
> If all we're talking about is named unit tests and running all of the unittest
> blocks within a module even if one fails, those aren't huge changes to the
> language. Both have been discussed before, and the only reason that the latter
> hasn't been implemented yet is that it requires changes to the compiler. So, I
> don't see any reason to make those a library solution. If we're talking
> something massively more complicated than that, then yes, a library solution
> starts making more sense. I also think that it then doesn't necessarily make
> sense to put it in the standard library.
>
> If the issue is how the tests print out on failure, that could probably be
> done in the language as well, depending on what you were talking about
> changing. If you want to do something massively complicated, however, then
> you'd probably have to come up with a library solution, but again, I see no
> reason to have it in the standard library if you're doing anything fancy with
> how the test failures print out.
>
> - Jonathan M Davis

I see it as three things I want done.

* Named unit tests
* Continue to run unit tests after a failed one
* A nice report at the end, preferably configurable

It would be nice if the first two were implemented in the language. If the third one is implemented in the runtime it could perhaps be configurable. One implements an interface, or similar, and can output the unit test report how he/she wants.

-- 
/Jacob Carlborg