On Thursday, December 29, 2011 00:06:57 Brad Anderson wrote:
> Forgive me if this is a silly question but a conversation in IRC got me wondering if compiler could check for shared/__gshared use (and any other thread unsafe operation) in each unittest and run those that aren't using them concurrently? Obviously not all at once but at least more than one at a time (perhaps set through user configuration).

That could result in non-deterministic failures. Granted, unittest blocks should usually be independent, but there's nothing that guarantees that they are, so running them in a non-derministic order could result in non- deterministic failures.

Also, what about unit tests for stuff like std.pararellism or std.concurrency - anything that actually specifically uses threads in its implementation? Running tests in their own threads could cause issues with that.

Each module in Phobos has its own executable for running its unit tests, and there has been talk of making _those_ parallelizable, and I think that parallelizing things on that level makes a lot more sense than trying to build it into the language.

- Jonathan M Davis

On Wednesday, 28 December 2011 at 22:21:09 UTC, Jacob Carlborg wrote:
> On 2011-12-28 22:19, jdrewsen wrote:
>> On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg wrote:
>>> On 2011-12-27 03:01, dsimcha wrote:
>>>> By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been
>>>> accepted into Phobos. Thanks to Jonas for his hard work and his
>>>> persistence through the multiple rounds of review that it took to get
>>>> this module up to Phobos's high and increasing quality standard.
>>>>
>>>> Keep the good work coming. Next in line, if it's ready, is Jacob
>>>> Carlborg's std.serialize. Jacob, please post here when you've got
>>>> something ready to go.
>>>
>>> Project page (two tutorials): http://dsource.org/projects/orange
>>> Repository: https://github.com/jacob-carlborg/orange
>>>
>>> Documentation:
>>>
>>> http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html
>>>
>>>
>>> (Don't forget the "Package" tab)
>>>
>>> Unit tests are available in the "tests" directory. These unit tests
>>> are not like regular unit tests that test individual functions. These
>>> unit tests are on a higher level.
>>>
>>> The most important part to review is the "serialization" package. The
>>> rest is mostly utility modules.
>>
>> After I quick look:
>>
>> I think that all other modules/packages under the orange package should
>> either be integrated into other existing phobos modules or as new
>> phobos modules since they are really not serialization specific.
>
> I agree with that.
>
>> Furthermore I think that all suppport for tango in the code should be
>> stripped before going into phobos.
>>
>> /Jonas
>
> That is obvious. I don't know if this have been clear or not but I've requested a pre-review. A review that should be as a regular review but before I do a Phobos integration. If it gets accepted into Phobos I'll do the necessary modifications (remove all Tango related code and so on) to integrate it into Phobos and then we can have a second review. I don't want to waste time on removing the Tango support if I don't know for sure that it will be accepted.

Ok that wasn't clear to me. I've updated the review queue:

http://prowiki.org/wiki4d/wiki.cgi?ReviewQueue

/Jonas

On 2011-12-29 06:11, Jonathan M Davis wrote:
> On Wednesday, December 28, 2011 23:07:51 Jacob Carlborg wrote:
>> I think it is, don't know what others think. What it does is it catches
>> AssertErrors so other unit tests can continue to run and then gives a
>> nice report at the end.
>
> I'm against it. I think that the compiler/runtime should be fixed so that each
> unit test block is run in a module even if one fails. That would solve the
> problem quite nicely IMHO, and that's already _supposed_ to be how it works.
> It just isn't properly implemented in that regard yet. And I'm against
> unittest blocks running any code after a single failure. So, I don't think
> that any additional unit testing framework is necessary.
>
> - Jonathan M Davis

Then the compiler need as well to collect all possible asserts and then somehow present them to the user. My library implementation already does this. Since this is completely implemented in library code it would be possible to have different formatters, a basic for outputting to the console and a more advanced with HTML output.

Less important but can be quite useful sometimes is that with my framework you add contexts to the unit tests, just like RSpec for those familiar with it.

If you haven't already, I suggest you take a look at the documentation for the unit test framework:

http://dl.dropbox.com/u/18386187/orange_docs/orange.test.UnitTester.html



For example:

import orange.test.UnitTester;

int sum (int x, int y)
{
    return x * y;
}

unittest ()
{
    describe("sum") in {
         it("should return the sum of the two given arguments") in {
              assert(sum(1, 2) == 3);
         }
    }
}

void main ()
{
    run;
}

If a test fails the framework will print out the context, the stack trace and a snippet from the failing test, something like this:

 sum
   - should return the sum of the given arguments

 Failures:
     1) sum should return the sum of the given arguments
        # main.d:44
        Stack trace:
        tango.core.Exception.AssertException@main(44): Assertion failure


 describe("sum") in {
 	it("should return the sum of the given arguments") in {
 		assert(sum(1, 2) == 3);
 	};
 };

 1 test, 1 failure

-- 
/Jacob Carlborg

On 2011-12-27 03:01, dsimcha wrote:
> By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been
> accepted into Phobos. Thanks to Jonas for his hard work and his
> persistence through the multiple rounds of review that it took to get
> this module up to Phobos's high and increasing quality standard.
>
> Keep the good work coming. Next in line, if it's ready, is Jacob
> Carlborg's std.serialize. Jacob, please post here when you've got
> something ready to go.

If we're going with my serialization library as the next item to review, should we have a review manager and a new thread?

-- 
/Jacob Carlborg

On Thursday, 29 December 2011 at 12:49:55 UTC, Jacob Carlborg wrote:
> For example:
>
> import orange.test.UnitTester;
>
> int sum (int x, int y)
> {
>    return x * y;
> }
>
> unittest ()
> {
>    describe("sum") in {
>         it("should return the sum of the two given arguments") in {
>              assert(sum(1, 2) == 3);
>         }
>    }
> }
>
> void main ()
> {
>    run;
> }
>
> If a test fails the framework will print out the context, the stack trace and a snippet from the failing test, something like this:
>
> sum
>   - should return the sum of the given arguments
>
> Failures:
>     1) sum should return the sum of the given arguments
>        # main.d:44
>        Stack trace:
>        tango.core.Exception.AssertException@main(44): Assertion failure
>
>
> describe("sum") in {
> 	it("should return the sum of the given arguments") in {
> 		assert(sum(1, 2) == 3);
> 	};
> };
>
> 1 test, 1 failure

Operator overloading abuse, ahoy!

On 2011-12-29 13:59, Jakob Ovrum wrote:
> On Thursday, 29 December 2011 at 12:49:55 UTC, Jacob Carlborg wrote:
>> For example:
>>
>> import orange.test.UnitTester;
>>
>> int sum (int x, int y)
>> {
>> return x * y;
>> }
>>
>> unittest ()
>> {
>> describe("sum") in {
>> it("should return the sum of the two given arguments") in {
>> assert(sum(1, 2) == 3);
>> }
>> }
>> }
>>
>> void main ()
>> {
>> run;
>> }
>>
>> If a test fails the framework will print out the context, the stack
>> trace and a snippet from the failing test, something like this:
>>
>> sum
>> - should return the sum of the given arguments
>>
>> Failures:
>> 1) sum should return the sum of the given arguments
>> # main.d:44
>> Stack trace:
>> tango.core.Exception.AssertException@main(44): Assertion failure
>>
>>
>> describe("sum") in {
>> it("should return the sum of the given arguments") in {
>> assert(sum(1, 2) == 3);
>> };
>> };
>>
>> 1 test, 1 failure
>
> Operator overloading abuse, ahoy!

Yeah, I know. The "describe" and "it" functions can be changed to take the delegate as an argument instead.

-- 
/Jacob Carlborg

When I first came to D, I read "unittest support" and thought that would be nice. But after I tried it, I realized it sucks and wrote something similar to Jacobs unittest framework.

> I'm against it. I think that the compiler/runtime should be fixed so that each unit test block is run in a module even if one fails. That would solve the problem quite nicely IMHO,
Which problem? That you can't get a good summary or backtrace? No
descriptions of the unit test that failed? That other unit tests in the
module will not run after an error? That you can't run some
specific unit-tests only?

> And
> I'm against unittest blocks running any code after a single failure.
On the other hand, I think this is absolutely necessary, especially if you have big modules.

Imagine a high level unit-tests fails and you can't see the failure in the low level helper functions that nails down the error. However every library implemented unit test framework could just stop after the first failure, if configured properly. Should not be the default though.

> So, I
> don't think that any additional unit testing framework is necessary.

I really think it is and will use one for my D code. Since both worlds could live together peacefully there is absolutely no reason not to include one in phobos.

On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
> When I first came to D, I read "unittest support" and thought that would be nice. But after I tried it, I realized it sucks and wrote something similar to Jacobs unittest framework.
> 
> > I'm against it. I think that the compiler/runtime should be fixed so
> > that
> > each unit test block is run in a module even if one fails. That would
> > solve the problem quite nicely IMHO,
> 
> Which problem? That you can't get a good summary or backtrace? No
> descriptions of the unit test that failed? That other unit tests in the
> module will not run after an error? That you can't run some
> specific unit-tests only?

You get a proper error message backtrace on unit test failure. The only problem is that once a single unittest block within a module fails, no further unittest blocks within that module are run (unittest blocks from subsequent modules are run but no more from the one with the failure). As such, once you get a single failure within a module, you can't see any further failures. That _is_ a problem, but it's the only one that I'm aware of. And that needs to be fixed in the runtime, _not_ by adding more complex library support. It's a known issue which requires some additional support by the compiler (additional hooks of some kind IIRC) for the runtime to be fixed, but it is the runtime that should be fixed rather than trying to fix it with a library.

> > And
> > I'm against unittest blocks running any code after a single failure.
> 
> On the other hand, I think this is absolutely necessary, especially if you have big modules.
> 
> Imagine a high level unit-tests fails and you can't see the failure in the low level helper functions that nails down the error. However every library implemented unit test framework could just stop after the first failure, if configured properly. Should not be the default though.
> 
> > So, I
> > don't think that any additional unit testing framework is necessary.
> 
> I really think it is and will use one for my D code. Since both worlds could live together peacefully there is absolutely no reason not to include one in phobos.

It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further.

- Jonathan M Davis

On 2011-12-30 19:49, Jonathan M Davis wrote:
> On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
>> I really think it is and will use one for my D code. Since both worlds
>> could live together peacefully there is absolutely no reason not to include
>> one in phobos.
>
> It's one thing to use a fancier framework on your own. It's quite another to
> put it in the standard library. D's unit testing framework is designed to be
> straightforward and simple. On the whole, it does the job quite well. And once
> the issue of not running subsequent unittest blocks within a module after a
> failure in that module is fixed, I see no benefit from adding any additional
> library support. It just complicates things further.
>
> - Jonathan M Davis

Will that be able to give a proper report of all failed tests in a nice format?

-- 
/Jacob Carlborg

On Friday, December 30, 2011 21:38:07 Jacob Carlborg wrote:
> On 2011-12-30 19:49, Jonathan M Davis wrote:
> > On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
> >> I really think it is and will use one for my D code. Since both worlds could live together peacefully there is absolutely no reason not to include one in phobos.
> > 
> > It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further.
> > 
> > - Jonathan M Davis
> 
> Will that be able to give a proper report of all failed tests in a nice format?

I think that the AssertError's message (which includes the file and line number of the failure) and its stack trace are plenty. It's exactly what you need and nothing else.

- Jonathan M Davis