On 4/14/11 10:34 AM, spir wrote:
> On 04/14/2011 04:22 PM, Andrei Alexandrescu wrote:
>> On 4/14/11 6:52 AM, spir wrote:
>>> On 04/14/2011 01:11 AM, bearophile wrote:
>>>>> * Since most of them don't actually output anything, the program
>>>>> > now detects output.length == 0 and says "Program run
>>>>> > successfully." upon completion to give some feedback.
>>>> This is a problem. Unittest code is not meant to produce an output,
>>>> while online snippets to try are meant to nearly always produce a
>>>> meaningful output.
>>>
>>> All my unittest blocks produce meaningful output; there exist firstly
>>> for that. Am I an heretic?
>>
>> No, you're just doing it wrong. Heretic might be sometimes good.
>>
>> http://www.linfo.org/rule_of_silence.html
>
> Right, I know that and totally disagree.

This is a good cue to revisit your opinion.

> This rule is IMO the exact
> opposite of human-friendliness (numerous authors on usability and
> friends share this opinion, indeed).

This may be a misunderstanding of a valid human friendliness concern with a simple mistake. In addition, you reject an authoritative text with an appeal to an undefined authority.

Applying "let me output something and eye it" at the level of each unit test does not scale because it makes it extremely difficult to identify issues once one test of many produces unexpected output. There are programs (such as "expect") that automate that task. The ultimate goal is to make errors noisy and easy to identify.

> Also, this is not what I'm talking about: I'm not here expecting just
> "all fine, brother", but various data helping me and understand how
> things actually worked; to be short, data useful to the coder during
> development or debug. See also reply to Steven's post.

I understand, and I repeat: that approach works great for exploration, but won't scale for systematic unit testing.


Andrei

On 4/14/11 12:00 PM, Steven Schveighoffer wrote:
> On Thu, 14 Apr 2011 12:48:26 -0400, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>
>> On 4/14/11 9:03 AM, Steven Schveighoffer wrote:
>>> Sometimes, I worry that my unit tests or asserts aren't running. Every
>>> once in a while, I have to change one to fail to make sure that code is
>>> compiling (this is especially true when I'm doing version statements or
>>> templates). It would be nice if there was a -assertprint mode which
>>> showed asserts actually running (only for the module compiled with that
>>> switch, of course).
>>
>> Could this be achieved within the language?
>
> I think you need to do it at the compiler level to make it useful. For
> example, an assert like:
>
> assert(container.length == 5);
>
> To print this out properly, I'd want to see that the assert passed, but
> also what the test was.

What's needed here is a "text of expression" feature similar to C's "#". That would help enforce and other artifacts too. I'm thinking along the lines of:

void myassert(string expr = __traits(text, condition))(bool condition) {
  ...
}

with, of course, a simpler syntax.

> I think this would be an awesome feature. One of the cool things about
> the new way phobos runs unit tests (one module at a time) is that if a
> particular test fails, I can just build/run it over and over again until
> it passes. This is huge since to run the full phobos unit tests can take
> a while.

That's exactly how Phobos unittest work on Linux, and that was the motivation behind making it work that way.


Andrei

> On 04/14/2011 04:22 PM, Andrei Alexandrescu wrote:
> > On 4/14/11 6:52 AM, spir wrote:
> >> On 04/14/2011 01:11 AM, bearophile wrote:
> >>>> * Since most of them don't actually output anything, the program
> >>>> 
> >>>> > now detects output.length == 0 and says "Program run successfully." upon completion to give some feedback.
> >>> 
> >>> This is a problem. Unittest code is not meant to produce an output, while online snippets to try are meant to nearly always produce a meaningful output.
> >> 
> >> All my unittest blocks produce meaningful output; there exist firstly for that. Am I an heretic?
> > 
> > No, you're just doing it wrong. Heretic might be sometimes good.
> > 
> > http://www.linfo.org/rule_of_silence.html
> 
> Right, I know that and totally disagree. This rule is IMO the exact
> opposite of human-friendliness (numerous authors on usability and friends
> share this opinion, indeed).
> Also, this is not what I'm talking about: I'm not here expecting just "all
> fine, brother", but various data helping me and understand how things
> actually worked; to be short, data useful to the coder during development
> or debug. See also reply to Steven's post.

And I have no clue what you'd want to print. Normally, if the assertion passes, everything's fine. Seeing output from it would just be noise. Now, being able to verify that a particular unit test block is running when you're dealing with version blocks could be useful periodically, but all you really need to verify is that the test is run, not what it's doing. If you need to debug it, you debug it. And if you need to debug it, usually the assertions would be failing anyway. Having debug output printing all of the time would just get in the way and make it harder to find what you need when you're actually debugging, since you'd have too much debug output if everything's printing debug output. So, I really don't understand what you'd be looking to print.

I can understand someone wanting the list of tests printed so that they could see where in the tests the program is, but generally, all you need to know is whether the tests failed, and seeing the failed assertions gives you that. So, ultimately, I don't really understand the push for stuff being printed during unit tests unless you're explicitly debugging a unit test.

- Jonathan M Davis

On Thu, 14 Apr 2011 13:16:33 -0400, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 4/14/11 12:00 PM, Steven Schveighoffer wrote:
>> On Thu, 14 Apr 2011 12:48:26 -0400, Andrei Alexandrescu
>> <SeeWebsiteForEmail@erdani.org> wrote:
>>
>>> On 4/14/11 9:03 AM, Steven Schveighoffer wrote:
>>>> Sometimes, I worry that my unit tests or asserts aren't running. Every
>>>> once in a while, I have to change one to fail to make sure that code is
>>>> compiling (this is especially true when I'm doing version statements or
>>>> templates). It would be nice if there was a -assertprint mode which
>>>> showed asserts actually running (only for the module compiled with that
>>>> switch, of course).
>>>
>>> Could this be achieved within the language?
>>
>> I think you need to do it at the compiler level to make it useful. For
>> example, an assert like:
>>
>> assert(container.length == 5);
>>
>> To print this out properly, I'd want to see that the assert passed, but
>> also what the test was.
>
> What's needed here is a "text of expression" feature similar to C's "#". That would help enforce and other artifacts too. I'm thinking along the lines of:
>
> void myassert(string expr = __traits(text, condition))(bool condition) {
>    ...
> }
>
> with, of course, a simpler syntax.

It still isn't exactly right.  Assert has some special properties that cannot be duplicated exactly with library code.  Such as true lazy evaluation and elimination during release mode.

But yes, it would be nice to be able to get a string of the parameters of a function (macro?).  It would certainly make this more feasible.

However, all this is ignoring one simple thing -- the hundreds of thousands of lines of code (mostly from std.datetime ;) ) that already have asserts.  Hooking assert directly automatically gives us a tool for printf debugging without changing any code.

I would point out that even for this solution, we are still modifying the compiler.  Any particular reason why adding a new trait is more desirable than modifying assert?  I certainly feel that the auto-inclusion of all the existing asserts would far outweigh the extensibility of adding a trait (which could probably also be added at the same time).

>
>> I think this would be an awesome feature. One of the cool things about
>> the new way phobos runs unit tests (one module at a time) is that if a
>> particular test fails, I can just build/run it over and over again until
>> it passes. This is huge since to run the full phobos unit tests can take
>> a while.
>
> That's exactly how Phobos unittest work on Linux, and that was the motivation behind making it work that way.

Yeah, I know, that's why I brought it up :)

-Steve

> On Thu, 14 Apr 2011 12:35:49 -0400, Daniel Gibson <metalcaedes@gmail.com>
> 
> wrote:
> > Am 14.04.2011 17:47, schrieb Steven Schveighoffer:
> >> On Thu, 14 Apr 2011 11:28:39 -0400, spir <denis.spir@gmail.com> wrote:
> >>> On 04/14/2011 04:03 PM, Steven Schveighoffer wrote:
> >>>> Sometimes, I worry that my unit tests or asserts aren't running.
> >>>> Every once in
> >>>> a while, I have to change one to fail to make sure that code is
> >>>> compiling (this
> >>>> is especially true when I'm doing version statements or templates).
> >>>> It would
> >>>> be nice if there was a -assertprint mode which showed asserts
> >>>> actually running
> >>>> (only for the module compiled with that switch, of course).
> >>> 
> >>> Man, I'm very pleased to read someone else advocating for optionally verbose assertions.
> >>> 
> >>> This could use 2 arguments instead of a predicate:
> >>> assert(expressions, value);
> >>> 
> >>> Example use:
> >> [snip]
> >> 
> >> I don't think we can change assert syntax now. What I was looking was for something more like:
> >> 
> >> assert(x == y);
> >> 
> >> prints out
> >> 
> >> "asserting x == y: true"
> >> 
> >> for asserts that pass when you have the 'verbose assert' flag turned on. This should be easy to do in the compiler by just translating
> >> 
> >> assert(expr);
> >> 
> >> to something like:
> >> 
> >> auto result = evaluateAssert(expr);
> >> print("asserting expr: ", result ? "true" : "false");
> >> assert(result);
> >> 
> >> -Steve
> > 
> > Another possibility are named unittests and printing out "running test
> > 'foobar' was successful".
> > One message for every assert is too verbose IMHO.
> 
> I'm thinking of the code examples that have asserts in them. I totally like the idea of code examples in ddoc being compiled as unit tests. This ensures your examples don't get out of date or are not invalid (as many of phobos' examples are). Nothing is worse when learning a language to encounter errors or unexpected results from the sanctioned examples. This means they have to look like unit tests, but also be useful for learning (see my above issue).
> 
> A compromise might be to be able to name unit tests, and then run a specific unit test by name on execution only. This allows you to ensure a specific unit test (and specific asserts in that unit test) are running without having to see all the asserts from the other unit tests, but also allows using asserts for general example code/testing.
> 
> This also could mean that you could simply directly compile a phobos module with -assertverbose -unittest and do:
> 
> ./algorithm --run-unit-test std.algorithm.sort_example
> 
> to see all the asserts working for that example.

I want named unit tests regardless, because then you'd get recognizable function names in the stack traces from exceptions that escape unit test blocks. Being able to print out which unit test is running or just finished by using a particular flag would definitely make some people around here happier though (much as it definitely shouldn't be the default behavior).

- Jonathan M Davis

On Thu, 14 Apr 2011 13:27:32 -0400, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

>> On Thu, 14 Apr 2011 12:35:49 -0400, Daniel Gibson <metalcaedes@gmail.com>
>>
>> wrote:
>> > Am 14.04.2011 17:47, schrieb Steven Schveighoffer:
>> >> On Thu, 14 Apr 2011 11:28:39 -0400, spir <denis.spir@gmail.com>  
>> wrote:
>> >>> On 04/14/2011 04:03 PM, Steven Schveighoffer wrote:
>> >>>> Sometimes, I worry that my unit tests or asserts aren't running.
>> >>>> Every once in
>> >>>> a while, I have to change one to fail to make sure that code is
>> >>>> compiling (this
>> >>>> is especially true when I'm doing version statements or templates).
>> >>>> It would
>> >>>> be nice if there was a -assertprint mode which showed asserts
>> >>>> actually running
>> >>>> (only for the module compiled with that switch, of course).
>> >>>
>> >>> Man, I'm very pleased to read someone else advocating for optionally
>> >>> verbose assertions.
>> >>>
>> >>> This could use 2 arguments instead of a predicate:
>> >>> assert(expressions, value);
>> >>>
>> >>> Example use:
>> >> [snip]
>> >>
>> >> I don't think we can change assert syntax now. What I was looking was
>> >> for something more like:
>> >>
>> >> assert(x == y);
>> >>
>> >> prints out
>> >>
>> >> "asserting x == y: true"
>> >>
>> >> for asserts that pass when you have the 'verbose assert' flag turned
>> >> on. This should be easy to do in the compiler by just translating
>> >>
>> >> assert(expr);
>> >>
>> >> to something like:
>> >>
>> >> auto result = evaluateAssert(expr);
>> >> print("asserting expr: ", result ? "true" : "false");
>> >> assert(result);
>> >>
>> >> -Steve
>> >
>> > Another possibility are named unittests and printing out "running test
>> > 'foobar' was successful".
>> > One message for every assert is too verbose IMHO.
>>
>> I'm thinking of the code examples that have asserts in them. I totally
>> like the idea of code examples in ddoc being compiled as unit tests. This
>> ensures your examples don't get out of date or are not invalid (as many of
>> phobos' examples are). Nothing is worse when learning a language to
>> encounter errors or unexpected results from the sanctioned examples. This
>> means they have to look like unit tests, but also be useful for learning
>> (see my above issue).
>>
>> A compromise might be to be able to name unit tests, and then run a
>> specific unit test by name on execution only. This allows you to ensure a
>> specific unit test (and specific asserts in that unit test) are running
>> without having to see all the asserts from the other unit tests, but also
>> allows using asserts for general example code/testing.
>>
>> This also could mean that you could simply directly compile a phobos
>> module with -assertverbose -unittest and do:
>>
>> ./algorithm --run-unit-test std.algorithm.sort_example
>>
>> to see all the asserts working for that example.
>
> I want named unit tests regardless, because then you'd get recognizable
> function names in the stack traces from exceptions that escape unit test
> blocks. Being able to print out which unit test is running or just finished by
> using a particular flag would definitely make some people around here happier
> though (much as it definitely shouldn't be the default behavior).

I agree it should not be the default.

What I'm trying to do is find a solution that makes examples with asserts useful for learning (i.e. show that something is happening), and try and make it also be useful for unit test debugging.

Obviously, during normal unit tests, you want only errors to print out.

-Steve

> On Thu, 14 Apr 2011 13:16:33 -0400, Andrei Alexandrescu
> 
> <SeeWebsiteForEmail@erdani.org> wrote:
> > On 4/14/11 12:00 PM, Steven Schveighoffer wrote:
> >> On Thu, 14 Apr 2011 12:48:26 -0400, Andrei Alexandrescu
> >> 
> >> <SeeWebsiteForEmail@erdani.org> wrote:
> >>> On 4/14/11 9:03 AM, Steven Schveighoffer wrote:
> >>>> Sometimes, I worry that my unit tests or asserts aren't running. Every
> >>>> once in a while, I have to change one to fail to make sure that code
> >>>> is
> >>>> compiling (this is especially true when I'm doing version statements
> >>>> or
> >>>> templates). It would be nice if there was a -assertprint mode which
> >>>> showed asserts actually running (only for the module compiled with
> >>>> that
> >>>> switch, of course).
> >>> 
> >>> Could this be achieved within the language?
> >> 
> >> I think you need to do it at the compiler level to make it useful. For example, an assert like:
> >> 
> >> assert(container.length == 5);
> >> 
> >> To print this out properly, I'd want to see that the assert passed, but also what the test was.
> > 
> > What's needed here is a "text of expression" feature similar to C's "#". That would help enforce and other artifacts too. I'm thinking along the lines of:
> > 
> > void myassert(string expr = __traits(text, condition))(bool condition) {
> > 
> > ...
> > 
> > }
> > 
> > with, of course, a simpler syntax.
> 
> It still isn't exactly right. Assert has some special properties that cannot be duplicated exactly with library code. Such as true lazy evaluation and elimination during release mode.
> 
> But yes, it would be nice to be able to get a string of the parameters of a function (macro?). It would certainly make this more feasible.
> 
> However, all this is ignoring one simple thing -- the hundreds of thousands of lines of code (mostly from std.datetime ;) ) that already have asserts. Hooking assert directly automatically gives us a tool for printf debugging without changing any code.

Actually, std.datetime doesn't use assert much. It mostly uses a version of assertPred which it has as a private function. Also, I've been rewriting the unit tests, and that will result in far fewer actual lines with assert or _assertPred thanks to more looping and whatnot. It does increase the need for printing out the actual results from an assertion though (which _assertPred does on failure). And what you're asking for here is at least related to the requested improvements to assert which resulted in assertPred being scrapped.

- Jonathan M Davis

On Thu, 14 Apr 2011 13:38:11 -0400, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> Actually, std.datetime doesn't use assert much. It mostly uses a version of
> assertPred which it has as a private function. Also, I've been rewriting the
> unit tests, and that will result in far fewer actual lines with assert or
> _assertPred thanks to more looping and whatnot. It does increase the need for
> printing out the actual results from an assertion though (which _assertPred
> does on failure). And what you're asking for here is at least related to the
> requested improvements to assert which resulted in assertPred being scrapped.

I forgot that assert isn't actually used much in std.datetime!

-Steve

On 4/14/11 12:26 PM, Steven Schveighoffer wrote:
> Any particular reason why adding a new trait is more
> desirable than modifying assert?

Absolutely!

Andrei

On Thu, 14 Apr 2011 13:54:00 -0400, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 4/14/11 12:26 PM, Steven Schveighoffer wrote:
>> Any particular reason why adding a new trait is more
>> desirable than modifying assert?
>
> Absolutely!

Maybe I worded my question wrong.  What I meant was what *is* the particular reason.

And keep in mind, when I say modifying assert, I mean changing the way assert compiles, not changing it's usage.

-Steve