View mode: basic / threaded / horizontal-split · Log in · Help
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Saturday, December 31, 2011 18:57:14 Andrew Wiley wrote:
> On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis <jmdavisProg@gmx.com> 
wrote:
> > On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
> >> On 2011-12-31 11:37, Jonathan M Davis wrote:
> >> > On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
> >> >>> I think that the AssertError's message (which includes the
> >> >>> file and
> >> >>> line number of the failure) and its stack trace are plenty.
> >> >>> It's
> >> >>> exactly what you need and nothing else.
> >> >>> 
> >> >>> - Jonathan M Davis
> >> >> 
> >> >> I want to have such a summary.
> >> > 
> >> > I don't see any reason to put that in the standard library.
> >> > There's
> >> > nothing wrong with 3rd party solutions which give additional
> >> > functionality, but D's unit test framework is designed to be
> >> > minimilistic, and I don't think that adding anything beyond what
> >> > it
> >> > does now in terms of summary makes any sense. The only major issue
> >> > in
> >> > that regard IMHO is the fact that no further unittest blocks
> >> > within a
> >> > module are run after one fails. Even if it did, I still don't
> >> > think
> >> > that a fancier summary would be worth having - especially in the
> >> > standard library.
> >> 
> >> BTW, what would be so wrong if the unit tests for the standard library
> >> displayed a nice report when finished?
> > 
> > My primary issue here is that I don't think that we should be adding
> > stuff to Phobos which is essentially a new unit test framework on top
> > of the built in one. If 3rd party stuff wants to do that. Fine. But the
> > standard library should use the standard facilities. If the standard
> > facilities aren't sufficient, then they should be improved.
> 
> The counterargument is that the language doesn't really provide a
> framework - it actually provides anonymous parameterless global
> functions that will be run before main is invoked if code is compiled
> with -unittest. That isn't considered a framework in any language I've
> ever used, but it adds just enough functionality to allow a
> well-integrated fully-featured library solution. Would making such a
> library solution part of the standard library really be a problem?
> I'm mostly ambivalent on this issue because I haven't had time to look
> closely at the proposed framework, but your argument seems to be that
> all unittesting functionality needs to be built into the language. I
> don't think that should be necessary or required.

For the most part, I don't think that we need anything more. It should be fixed 
so that all unittest blocks within a module are run even if one fails. This 
should be fixed in the compiler and runtime rather than having to hack it 
together with a library solution, since it's supposed to work that way in the 
first place but doesn't because of some changes that need to be made to the 
compiler (some hooks of some kind are needed IIRC). Named unit tests would 
also be nice, but it would be _far_ cleaner to add those to the language than 
try and add them with a library solution (e.g. unittest(test_name){}). It 
would also be nice if it were possible to hook in something that would run 
specific unit tests, but again, such hooks would then need to be provided by 
the compiler and/or runtime - unless you want to simply create a function for 
each unit test and have each unittest block call a specific function? That's an 
ugly solution IMHO.

All of those changes would be fairly minor in terms of how they affect how unit 
tests function and would be far cleaner IMHO to have in the language. I think 
that on some level, the unit test framework in the language has failed if we 
have to add library solutions on top of it to get such basic functionality.

> > As for a "nice report," I don't see anything wrong with just using the
> > stack traces (which include the file, line number, and error message of
> > the assertion failure). That's all the information that's needed.
> > Anything else is superfluous IMHO. Now, if there were something nicer
> > that could be generally agreed upon and added to druntime such that the
> > standard unit test facilities used it, then fine. I don't see any point
> > to it, but at least in that case, the standard library is still using
> > the standard unit test framework. What I really don't want to see is
> > Phobos essentially building a new unit test framework on top of the
> > existing one. Any issues that need to be addressed with the unit test
> > framework for the standard library should be addressed in the standard
> > framework. Any additional framework stuff should be left to 3rd
> > parties.
> 
> As I said above, I wouldn't consider what we have to be a framework,
> but it's definitely enough to build an excellent library solution on
> top of.
> As for a report, the problem is that an assertion error isn't what you
> want when you're running, say, a continuous integration server (or,
> say, a pull request tester). What you really want is a detailed
> explanation of what unittests broke, what the tests were testing, and
> how the result differed from what was expected. You want to be able to
> have a reasonable idea of what went wrong *without* having to look at
> someone else's code and figure out exactly what they're testing every
> time.

I don't really get what you'd need beyond knowing which test failed. The 
AssertErrors tell you that. It would definitely be nicer if the unittest blocks 
had proper names to go with them, but they're not necessary for knowing which 
test failed. The file and line number give you that. I don't see how you're 
going to know what actually failed unless you actually look at the test. If 
the tests have names and are well named, then that gives you a better idea of 
what isn't working before you look at the code, but ultimately, you have to 
look at the code to know what went wrong. And if you can name unittest blocks, 
then that problem is solved anyway, since then you'd have a name for the unit 
test failure to give in any report that you wanted to create.

- Jonathan M Davis
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
Jonathan M Davis wrote:

> I think
> that on some level, the unit test framework in the language has failed if
> we have to add library solutions on top of it to get such basic
> functionality.

That's what we are saying: The unit test framework fails for us and a 
library solution is perfectly possible, even one that lets everyone use
the build in functionality if he wants.

You don't find it useful, but others do. I'd say: many others do. If 
something is considered useful by many it should have a place in the 
standard library.
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On 2011-12-31 21:56, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
>> BTW, what would be so wrong if the unit tests for the standard library
>> displayed a nice report when finished?
>
> My primary issue here is that I don't think that we should be adding stuff to
> Phobos which is essentially a new unit test framework on top of the built in
> one. If 3rd party stuff wants to do that. Fine. But the standard library should
> use the standard facilities. If the standard facilities aren't sufficient, then
> they should be improved.

First, that's how programming works. You build new abstractions on top 
of existing ones. Second, I would called the built in support for unit 
testing in D a "unit test framework". If add a unit test framework to 
Phobos it would be standard facilities and it wouldn't be a problem.

> As for a "nice report," I don't see anything wrong with just using the stack
> traces (which include the file, line number, and error message of the assertion
> failure). That's all the information that's needed. Anything else is
> superfluous IMHO. Now, if there were something nicer that could be generally
> agreed upon and added to druntime such that the standard unit test facilities
> used it, then fine. I don't see any point to it, but at least in that case, the
> standard library is still using the standard unit test framework. What I
> really don't want to see is Phobos essentially building a new unit test
> framework on top of the existing one. Any issues that need to be addressed
> with the unit test framework for the standard library should be addressed in
> the standard framework. Any additional framework stuff should be left to 3rd
> parties.
>
> - Jonathan M Davis

What's wrong with being able to run the unit tests from your editor, 
have the unit test framework output HTML (or similar), displayed in your 
editor and then you can click on links in the stack trace to get to the 
source code. If you don't see why that's useful that we can just end 
this discussion now.

-- 
/Jacob Carlborg
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On 2011-12-31 21:57, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
>> Yes but what happens when there are many failed tests, i.e. may
>> AssertErrors that have been thrown? It will just print all after each
>> other and you have to count them yourself if you want to know how many
>> failed tests there are?
>
> What does the number of failures really matter? You just need to know which
> ones failed and where. The AssertErrors give you that.
>
> - Jonathan M Davis

It can be nice for statistics. And again, how will it be displayed, just 
all AssertErrors after each other?

-- 
/Jacob Carlborg
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On 2011-12-31 22:01, Jonathan M Davis wrote:
> On Saturday, December 31, 2011 16:04:12 Jacob Carlborg wrote:
>> It would be possible to implement named unit tests only in library code.
>> It would not have as nice syntax as if it was implemented in the
>> language but still possible.
>>
>> In Ruby on Rails I run single unit tests all the time. Why would I run
>> all the unit tests, which can take five minutes, when I just can run one
>> unit test and it takes just one second?
>>
>> When your doing test/behavior driven development (T/BDD) it's certainly
>> nice to be able to run single unit tests, because you run it all the time.
>
> Yes. I agree that it would be nice, but for it to be done at all cleanly, the
> language, compiler, and druntime need to be improved to make it possible.
> However, at least syntactically, such changes should be completely backwards
> compatible, so they can be added at a future date. Regardless, I don't think
> that it's a problem that Phobos should be trying to solve.
>
> - Jonathan M Davis

Ok, if you would rather have all this in the language I would say no do 
that. But I know other people in the community that usually prefer to do 
a library solution if possible.

-- 
/Jacob Carlborg
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On 2012-01-01 01:57, Andrew Wiley wrote:
> On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis<jmdavisProg@gmx.com>  wrote:
>> On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
>>> On 2011-12-31 11:37, Jonathan M Davis wrote:
>>>> On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
>>>>>> I think that the AssertError's message (which includes the file and
>>>>>> line number of the failure) and its stack trace are plenty. It's
>>>>>> exactly what you need and nothing else.
>>>>>>
>>>>>> - Jonathan M Davis
>>>>>
>>>>> I want to have such a summary.
>>>>
>>>> I don't see any reason to put that in the standard library. There's
>>>> nothing wrong with 3rd party solutions which give additional
>>>> functionality, but D's unit test framework is designed to be
>>>> minimilistic, and I don't think that adding anything beyond what it
>>>> does now in terms of summary makes any sense. The only major issue in
>>>> that regard IMHO is the fact that no further unittest blocks within a
>>>> module are run after one fails. Even if it did, I still don't think
>>>> that a fancier summary would be worth having - especially in the
>>>> standard library.
>>>
>>> BTW, what would be so wrong if the unit tests for the standard library
>>> displayed a nice report when finished?
>>
>> My primary issue here is that I don't think that we should be adding stuff to
>> Phobos which is essentially a new unit test framework on top of the built in
>> one. If 3rd party stuff wants to do that. Fine. But the standard library should
>> use the standard facilities. If the standard facilities aren't sufficient, then
>> they should be improved.
>
> The counterargument is that the language doesn't really provide a
> framework - it actually provides anonymous parameterless global
> functions that will be run before main is invoked if code is compiled
> with -unittest. That isn't considered a framework in any language I've
> ever used, but it adds just enough functionality to allow a
> well-integrated fully-featured library solution. Would making such a
> library solution part of the standard library really be a problem?
> I'm mostly ambivalent on this issue because I haven't had time to look
> closely at the proposed framework, but your argument seems to be that
> all unittesting functionality needs to be built into the language. I
> don't think that should be necessary or required.
>
>
>> As for a "nice report," I don't see anything wrong with just using the stack
>> traces (which include the file, line number, and error message of the assertion
>> failure). That's all the information that's needed. Anything else is
>> superfluous IMHO. Now, if there were something nicer that could be generally
>> agreed upon and added to druntime such that the standard unit test facilities
>> used it, then fine. I don't see any point to it, but at least in that case, the
>> standard library is still using the standard unit test framework. What I
>> really don't want to see is Phobos essentially building a new unit test
>> framework on top of the existing one. Any issues that need to be addressed
>> with the unit test framework for the standard library should be addressed in
>> the standard framework. Any additional framework stuff should be left to 3rd
>> parties.
>
> As I said above, I wouldn't consider what we have to be a framework,
> but it's definitely enough to build an excellent library solution on
> top of.
> As for a report, the problem is that an assertion error isn't what you
> want when you're running, say, a continuous integration server (or,
> say, a pull request tester). What you really want is a detailed
> explanation of what unittests broke, what the tests were testing, and
> how the result differed from what was expected. You want to be able to
> have a reasonable idea of what went wrong *without* having to look at
> someone else's code and figure out exactly what they're testing every
> time.

Exactly, I agree.

-- 
/Jacob Carlborg
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
> Ok, if you would rather have all this in the language I would say no do
> that. But I know other people in the community that usually prefer to do
> a library solution if possible.

If all we're talking about is named unit tests and running all of the unittest 
blocks within a module even if one fails, those aren't huge changes to the 
language. Both have been discussed before, and the only reason that the latter 
hasn't been implemented yet is that it requires changes to the compiler. So, I 
don't see any reason to make those a library solution. If we're talking 
something massively more complicated than that, then yes, a library solution 
starts making more sense. I also think that it then doesn't necessarily make 
sense to put it in the standard library.

If the issue is how the tests print out on failure, that could probably be 
done in the language as well, depending on what you were talking about 
changing. If you want to do something massively complicated, however, then 
you'd probably have to come up with a library solution, but again, I see no 
reason to have it in the standard library if you're doing anything fancy with 
how the test failures print out.

- Jonathan M Davis
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Sunday, January 01, 2012 15:32:34 Jacob Carlborg wrote:
> On 2011-12-31 21:57, Jonathan M Davis wrote:
> > On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
> >> Yes but what happens when there are many failed tests, i.e. may
> >> AssertErrors that have been thrown? It will just print all after each
> >> other and you have to count them yourself if you want to know how many
> >> failed tests there are?
> > 
> > What does the number of failures really matter? You just need to know
> > which ones failed and where. The AssertErrors give you that.
> > 
> > - Jonathan M Davis
> 
> It can be nice for statistics. And again, how will it be displayed, just
> all AssertErrors after each other?

Yes.

- Jonathan M Davis
January 01, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On Sunday, January 01, 2012 15:31:18 Jacob Carlborg wrote:
> What's wrong with being able to run the unit tests from your editor,
> have the unit test framework output HTML (or similar), displayed in your
> editor and then you can click on links in the stack trace to get to the
> source code. If you don't see why that's useful that we can just end
> this discussion now.

If you want fancier unit test facilities in your own code. Fine. I just don't 
think that they should be in the standard library. I think that on the whole, 
D's built-in unit test framework works fine as it is (barring a few tweaks such 
as making it so that all unittest blocks within a module run), and that 
further additions are a needless complication to the standard unit testing 
facilities and better left to 3rd party solutions where you can do whatever 
you want with them.

- Jonathan M Davis
January 02, 2012
Re: CURL Wrapper: Congratulations Next up: std.serialize
On 2012-01-02 00:14, Jonathan M Davis wrote:
> On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
>> Ok, if you would rather have all this in the language I would say no do
>> that. But I know other people in the community that usually prefer to do
>> a library solution if possible.
>
> If all we're talking about is named unit tests and running all of the unittest
> blocks within a module even if one fails, those aren't huge changes to the
> language. Both have been discussed before, and the only reason that the latter
> hasn't been implemented yet is that it requires changes to the compiler. So, I
> don't see any reason to make those a library solution. If we're talking
> something massively more complicated than that, then yes, a library solution
> starts making more sense. I also think that it then doesn't necessarily make
> sense to put it in the standard library.
>
> If the issue is how the tests print out on failure, that could probably be
> done in the language as well, depending on what you were talking about
> changing. If you want to do something massively complicated, however, then
> you'd probably have to come up with a library solution, but again, I see no
> reason to have it in the standard library if you're doing anything fancy with
> how the test failures print out.
>
> - Jonathan M Davis

I see it as three things I want done.

* Named unit tests
* Continue to run unit tests after a failed one
* A nice report at the end, preferably configurable

It would be nice if the first two were implemented in the language. If 
the third one is implemented in the runtime it could perhaps be 
configurable. One implements an interface, or similar, and can output 
the unit test report how he/she wants.

-- 
/Jacob Carlborg
1 2 3 4 5
Top | Discussion index | About this forum | D home