On Monday 24 January 2011 06:34:49 Jonathan M Davis wrote:
> In case you didn't know, I have a set of unit test helper functions which have been being reviewed for possible inclusion in phobos. Here's an update.
> 
> Most recent code: http://is.gd/F1OHat
> 
> Okay. I took the previous suggestions into consideration and adjusted the code a bit more. However, most of the changes are to the documentation (though there are some changes to the code). Some of the code duplication was removed, and the way that some of the assertPred functions' errors are formatted has been altered so that values line up vertically, making them easier to compare. The big change is the docs though. There's now a fake version of assertPred at the top with an overall description for assertPred followed by the individual versions with as little documentation as seemed appropriate while still getting all of the necessary information across. A couple of the functions still have irritatingly long example sections, but anything less wouldn't get the functionality across.
> 
> In any case. Here's the updated code. Review away. Andrei set the vote deadline for February 7th, at which point, if it passes majority vote, then it will go into Phobos. The number of functions is small enough now (thanks to having consolidated most of them into the fantastically versatile assertPred) that it looks like it will likely go in std.exception if the vote passes rather than becoming a new module. So, the std.unittests title has now become a bit of a misnomer, but that's what I've been calling it, so it seemed appropriate to continue to label it that way in the thread's title.
> 
> - Jonathan M Davis

By the way, I should point out that this requires the newest (now git!) version of druntime to work, since I made assertThrown chain the exception which was thrown to the AssertError on failure. It doesn't do much good at the moment, since only the AssertError gets printed, but maybe that'll change at some point. Regardless, I had to make it so that AssertError would take another Throwable for it to work, so you'll need the version from the git repository ( https://github.com/D-Programming-Language/druntime ) for it work.

The examples (and therefore the unit tests) require the git version of Phobos as well ( https://github.com/D-Programming-Language/phobos ), since some of them use std.datetime. The code itself doesn't require a newer version of Phobos though.

- Jonathan M Davis

Maybe if we get that "any" function any time soon (pardon the pun), we could simplify those constraints a little bit:

enum string[] binaryOps = ["+", "-", "*", "/", "%", "^^", "&", "|", "^", "<<", ">>", ">>>", "~"];

void assertPred(string op, L, R, E)
               (L lhs, R rhs, E expected, lazy string msg = null,
string file = __FILE__, size_t line = __LINE__)
    if(any(op, binaryOps) &&
       __traits(compiles, mixin("lhs " ~ op ~ " rhs")) &&
       __traits(compiles, mixin("(lhs " ~ op ~ " rhs) == expected")) &&
       isPrintable!L &&
       isPrintable!R)
{
}

I don't recall what any will look like exactly, but something like that could work I guess.

On Monday 24 January 2011 07:48:25 Andrej Mitrovic wrote:
> Maybe if we get that "any" function any time soon (pardon the pun), we could simplify those constraints a little bit:
> 
> enum string[] binaryOps = ["+", "-", "*", "/", "%", "^^", "&", "|", "^", "<<", ">>", ">>>", "~"];
> 
> void assertPred(string op, L, R, E)
>                (L lhs, R rhs, E expected, lazy string msg = null,
> string file = __FILE__, size_t line = __LINE__)
>     if(any(op, binaryOps) &&
>        __traits(compiles, mixin("lhs " ~ op ~ " rhs")) &&
>        __traits(compiles, mixin("(lhs " ~ op ~ " rhs) == expected")) &&
>        isPrintable!L &&
>        isPrintable!R)
> {
> }
> 
> I don't recall what any will look like exactly, but something like that could work I guess.

Assuming that it worked with CTFE, of course (surprisingly little in std.algorithm works in CTFE from what I've seen). Ultimately, I don' think that it matters all that much, since you need a list of the acceptable operators somewhere regardless, but using any _would_ be a bit cleaner.

- Jonathan M Davis

Jonathan M Davis wrote:
> In case you didn't know, I have a set of unit test helper functions which have been being reviewed for possible inclusion in phobos. Here's an update.
> 
> Most recent code: http://is.gd/F1OHat
> 
> Okay. I took the previous suggestions into consideration and adjusted the code a bit more. However, most of the changes are to the documentation (though there are some changes to the code). Some of the code duplication was removed, and the way that some of the assertPred functions' errors are formatted has been altered so that values line up vertically, making them easier to compare. The big change is the docs though. There's now a fake version of assertPred at the top with an overall description for assertPred followed by the individual versions with as little documentation as seemed appropriate while still getting all of the necessary information across. A couple of the functions still have irritatingly long example sections, but anything less wouldn't get the functionality across.
> 
> In any case. Here's the updated code. Review away. Andrei set the vote deadline for February 7th, at which point, if it passes majority vote, then it will go into Phobos. The number of functions is small enough now (thanks to having consolidated most of them into the fantastically versatile assertPred) that it looks like it will likely go in std.exception if the vote passes rather than becoming a new module. So, the std.unittests title has now become a bit of a misnomer, but that's what I've been calling it, so it seemed appropriate to continue to label it that way in the thread's title.

I wonder whether there is a nice way to have unittests included in the
documentation but also executed. There are lots of examples in the
module (search for 'Verify Examples').
I like to avoid this duplication. Has anybody an idea how to achieve
this? Often the unittests themselves are a pretty good code
documentation.

Jens

On Monday, January 24, 2011 09:55:52 Jens Mueller wrote:
> Jonathan M Davis wrote:
> > In case you didn't know, I have a set of unit test helper functions which have been being reviewed for possible inclusion in phobos. Here's an update.
> > 
> > Most recent code: http://is.gd/F1OHat
> > 
> > Okay. I took the previous suggestions into consideration and adjusted the code a bit more. However, most of the changes are to the documentation (though there are some changes to the code). Some of the code duplication was removed, and the way that some of the assertPred functions' errors are formatted has been altered so that values line up vertically, making them easier to compare. The big change is the docs though. There's now a fake version of assertPred at the top with an overall description for assertPred followed by the individual versions with as little documentation as seemed appropriate while still getting all of the necessary information across. A couple of the functions still have irritatingly long example sections, but anything less wouldn't get the functionality across.
> > 
> > In any case. Here's the updated code. Review away. Andrei set the vote deadline for February 7th, at which point, if it passes majority vote, then it will go into Phobos. The number of functions is small enough now (thanks to having consolidated most of them into the fantastically versatile assertPred) that it looks like it will likely go in std.exception if the vote passes rather than becoming a new module. So, the std.unittests title has now become a bit of a misnomer, but that's what I've been calling it, so it seemed appropriate to continue to label it that way in the thread's title.
> 
> I wonder whether there is a nice way to have unittests included in the
> documentation but also executed. There are lots of examples in the
> module (search for 'Verify Examples').
> I like to avoid this duplication. Has anybody an idea how to achieve
> this? Often the unittests themselves are a pretty good code
> documentation.

I think that it's been discussed a time or two, but nothing has been done about it. It wouldn't be entirely straightforward to do. Essentially, either a unittest block would have to be generated from the Examples section in the documentation, or you'd have to have some way to indicate that a particular unittest block got put into the documentation as an Examples section. It's certainly true that it would be ideal to have a way to avoid the duplication, but we don't have one at the moment, and it hasn't yet been a high enough priority to sort out how to do it and implement it.

- Jonathan M Davis

Jonathan M Davis wrote:
> On Monday, January 24, 2011 09:55:52 Jens Mueller wrote:
> > Jonathan M Davis wrote:
> > > In case you didn't know, I have a set of unit test helper functions which have been being reviewed for possible inclusion in phobos. Here's an update.
> > > 
> > > Most recent code: http://is.gd/F1OHat
> > > 
> > > Okay. I took the previous suggestions into consideration and adjusted the code a bit more. However, most of the changes are to the documentation (though there are some changes to the code). Some of the code duplication was removed, and the way that some of the assertPred functions' errors are formatted has been altered so that values line up vertically, making them easier to compare. The big change is the docs though. There's now a fake version of assertPred at the top with an overall description for assertPred followed by the individual versions with as little documentation as seemed appropriate while still getting all of the necessary information across. A couple of the functions still have irritatingly long example sections, but anything less wouldn't get the functionality across.
> > > 
> > > In any case. Here's the updated code. Review away. Andrei set the vote deadline for February 7th, at which point, if it passes majority vote, then it will go into Phobos. The number of functions is small enough now (thanks to having consolidated most of them into the fantastically versatile assertPred) that it looks like it will likely go in std.exception if the vote passes rather than becoming a new module. So, the std.unittests title has now become a bit of a misnomer, but that's what I've been calling it, so it seemed appropriate to continue to label it that way in the thread's title.
> > 
> > I wonder whether there is a nice way to have unittests included in the
> > documentation but also executed. There are lots of examples in the
> > module (search for 'Verify Examples').
> > I like to avoid this duplication. Has anybody an idea how to achieve
> > this? Often the unittests themselves are a pretty good code
> > documentation.
> 
> I think that it's been discussed a time or two, but nothing has been done about it. It wouldn't be entirely straightforward to do. Essentially, either a unittest block would have to be generated from the Examples section in the documentation, or you'd have to have some way to indicate that a particular unittest block got put into the documentation as an Examples section. It's certainly true that it would be ideal to have a way to avoid the duplication, but we don't have one at the moment, and it hasn't yet been a high enough priority to sort out how to do it and implement it.

I see. I understand that it does not have high priority. Just wondered whether ...

Jens

On 1/24/11 1:50 PM, Jens Mueller wrote:
> Jonathan M Davis wrote:
>> I think that it's been discussed a time or two, but nothing has been done about
>> it. It wouldn't be entirely straightforward to do. Essentially, either a
>> unittest block would have to be generated from the Examples section in the
>> documentation, or you'd have to have some way to indicate that a particular
>> unittest block got put into the documentation as an Examples section. It's
>> certainly true that it would be ideal to have a way to avoid the duplication,
>> but we don't have one at the moment, and it hasn't yet been a high enough
>> priority to sort out how to do it and implement it.
>
> I see. I understand that it does not have high priority. Just wondered
> whether ...
>
> Jens

The change is much simpler than what Jonathan suggests. A change can be made such that any unittest preceded by a documentation comment is automatically considered an example.

/**
 Example:
*/
unittest
{
    writeln("This is how it works.");
}


Andrei

On 1/24/11 2:15 PM, Andrei Alexandrescu wrote:
> On 1/24/11 1:50 PM, Jens Mueller wrote:
>> Jonathan M Davis wrote:
>>> I think that it's been discussed a time or two, but nothing has been
>>> done about
>>> it. It wouldn't be entirely straightforward to do. Essentially, either a
>>> unittest block would have to be generated from the Examples section
>>> in the
>>> documentation, or you'd have to have some way to indicate that a
>>> particular
>>> unittest block got put into the documentation as an Examples section.
>>> It's
>>> certainly true that it would be ideal to have a way to avoid the
>>> duplication,
>>> but we don't have one at the moment, and it hasn't yet been a high
>>> enough
>>> priority to sort out how to do it and implement it.
>>
>> I see. I understand that it does not have high priority. Just wondered
>> whether ...
>>
>> Jens
>
> The change is much simpler than what Jonathan suggests. A change can be
> made such that any unittest preceded by a documentation comment is
> automatically considered an example.
>
> /**
> Example:
> */
> unittest
> {
> writeln("This is how it works.");
> }
>
>
> Andrei

BTW I consider this a very important topic. We have _plenty_ of examples that don't work and are not mechanically verifiable. The reasons range from minor typos to language changes to implementation limitations. Generally this is what they call "documentation rot". This is terrible PR for the language.

Changing ddoc to recognize documentation unittests would fix this matter once and forever.

Last but not least, the "----" separators for code samples are awful because no editor recognizes them for anything - they confuse the hell out of Emacs for one thing.


Andrei

On Mon, 24 Jan 2011 15:20:13 -0500, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 1/24/11 2:15 PM, Andrei Alexandrescu wrote:
>> On 1/24/11 1:50 PM, Jens Mueller wrote:
>>> Jonathan M Davis wrote:
>>>> I think that it's been discussed a time or two, but nothing has been
>>>> done about
>>>> it. It wouldn't be entirely straightforward to do. Essentially, either a
>>>> unittest block would have to be generated from the Examples section
>>>> in the
>>>> documentation, or you'd have to have some way to indicate that a
>>>> particular
>>>> unittest block got put into the documentation as an Examples section.
>>>> It's
>>>> certainly true that it would be ideal to have a way to avoid the
>>>> duplication,
>>>> but we don't have one at the moment, and it hasn't yet been a high
>>>> enough
>>>> priority to sort out how to do it and implement it.
>>>
>>> I see. I understand that it does not have high priority. Just wondered
>>> whether ...
>>>
>>> Jens
>>
>> The change is much simpler than what Jonathan suggests. A change can be
>> made such that any unittest preceded by a documentation comment is
>> automatically considered an example.
>>
>> /**
>> Example:
>> */
>> unittest
>> {
>> writeln("This is how it works.");
>> }
>>
>>
>> Andrei
>
> BTW I consider this a very important topic. We have _plenty_ of examples that don't work and are not mechanically verifiable. The reasons range from minor typos to language changes to implementation limitations. Generally this is what they call "documentation rot". This is terrible PR for the language.
>
> Changing ddoc to recognize documentation unittests would fix this matter once and forever.
>
> Last but not least, the "----" separators for code samples are awful because no editor recognizes them for anything - they confuse the hell out of Emacs for one thing.

This only makes sense if:

1. The unit test immediately follows the item being documented
2. The unit test *only* tests that item.

The second one could be pretty annoying.  Consider cases where several functions interact (I've seen this many times on Microsoft's Documentation), and it makes sense to make one example that covers all of them.  Having them 'testable' means creating several identical unit tests.

One way to easily fix this is to allow an additional parameter to the comment:

/**
Example(Foo.foo(int), Foo.bar(int)):
*/
unittest
{
   auto foo = new Foo;
   foo.foo(5);
   foo.bar(6);
   assert(foo.toString() == "bazunga!");
}

The above means, copy the example to both Foo.foo(int) and Foo.bar(int)

An alternative that is more verbose, but probably more understandable:

/**
Example:
Covers Foo.foo(int)
Covers Foo.bar(int)
*/

Of course, a lack of target just means it applies to the item just documented.

One other thing, using writefln is considered bad form in unit tests (you want *no* output if the unit test works).  But many examples might want to demonstrate how e.g. an object interacts with writefln.  Any suggestions?  The assert line above is not very pretty for example...

-Steve

Andrei Alexandrescu wrote:
> On 1/24/11 1:50 PM, Jens Mueller wrote:
> >Jonathan M Davis wrote:
> >>I think that it's been discussed a time or two, but nothing has been done about it. It wouldn't be entirely straightforward to do. Essentially, either a unittest block would have to be generated from the Examples section in the documentation, or you'd have to have some way to indicate that a particular unittest block got put into the documentation as an Examples section. It's certainly true that it would be ideal to have a way to avoid the duplication, but we don't have one at the moment, and it hasn't yet been a high enough priority to sort out how to do it and implement it.
> >
> >I see. I understand that it does not have high priority. Just wondered whether ...
> >
> >Jens
> 
> The change is much simpler than what Jonathan suggests. A change can be made such that any unittest preceded by a documentation comment is automatically considered an example.
> 
> /**
>  Example:
> */
> unittest
> {
>     writeln("This is how it works.");
> }

That does not work for me.
$ dmd -unittest -D -Dftest.html test.d

I get an empty example.
<dl><dt><big>void <u>__unittest2</u>();
</big></dt>
<dd><b>Example:</b><br>
<br><br>

</dd>
</dl>

Jens