Is there a standard and/or acceptable way to make sure that pre-conditions, post-conditions, or invariants _fail_ when running unit tests? That is, lets say I had a function like this

void func(int x)
in
{
    assert(x < 8);
}
body
{
  //...
}


and I wanted to test to make sure that func() couldn't be called with any int greater or equal to 8, what would I do? The best that I can think of is to catch an AssertError and ignore it. e.g.

unittest
{
    try
    {
        func(8);
        assert(0);
    }
    catch(AssertionError ae)
    {}
}


But catching AssertionErrors is generally a bad idea. As I understand it, when anything derived from Error is thrown, code is not left in a proper state, with stuff like destructors being skipped. Would it be okay to catch it in a case like this, or is it a really bad idea? If it's a bad idea, I haven't a clue how to verify that pre-condition, post-conditions, and invariants are correct.

- Jonathan M Davis

Jonathan M Davis wrote:

> Is there a standard and/or acceptable way to make sure that pre-conditions, post-conditions, or invariants _fail_ when running unit tests? That is, lets say I had a function like this
> 
> void func(int x)
> in
> {
>     assert(x < 8);
> }
> body
> {
>   //...
> }
> 
> 
> and I wanted to test to make sure that func() couldn't be called with any int greater or equal to 8, what would I do? The best that I can think of is to catch an AssertError and ignore it. e.g.
> 
> unittest
> {
>     try
>     {
>         func(8);
>         assert(0);
>     }
>     catch(AssertionError ae)
>     {}
> }
> 
> 
> But catching AssertionErrors is generally a bad idea. As I understand it, when anything derived from Error is thrown, code is not left in a proper state, with stuff like destructors being skipped. Would it be okay to catch it in a case like this, or is it a really bad idea? If it's a bad idea, I haven't a clue how to verify that pre-condition, post-conditions, and invariants are correct.
> 
> - Jonathan M Davis

Thats a good one. It used to be so that dmd would assume a halt and you could get segfaults for continuing on. I *believe* this has changed and it is ok to catch AssertError in this scenario. The only thing I can think of is that nothrow functions can be optimized by the compiler, but are not typechecked for throwing Error.

Even if this is correct, you still have the problem foo could call other functions that assert, which invalidates at least two contract checks. This may require some care. Perhaps an idea is to define a PreconditionError and PostconditionError and wrap them in a function similar to enforce, so you can differentiate more easily where the error is coming from. This way you also get better error messages and have one point where to change the behavior if this may be required.

On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis <jmdavisprog@gmail.com> wrote:

> Is there a standard and/or acceptable way to make sure that pre-conditions,
> post-conditions, or invariants _fail_ when running unit tests? That is, lets say
> I had a function like this
>
> void func(int x)
> in
> {
>     assert(x < 8);
> }
> body
> {
>   //...
> }
>
>
> and I wanted to test to make sure that func() couldn't be called with any int
> greater or equal to 8, what would I do?

Hm... unit testing your unit tests :)

input contracts and unit tests are supposed to be simple, provable code so you don't have to test them.  The above function is obviously a simple example, you don't really need to unit test it (right?), so what would a complicated in contract look like?

It's also a good idea to avoid complex expressions in unit tests.  If you have a complex expression, split it out into several lines to avoid having to work through the logic in your head.  Performance/minimal LOC is not a goal you need in unit tests/contracts.

-Steve

On Monday, August 16, 2010 05:43:08 Steven Schveighoffer wrote:
> On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
> 
> <jmdavisprog@gmail.com> wrote:
> > Is there a standard and/or acceptable way to make sure that
> > pre-conditions,
> > post-conditions, or invariants _fail_ when running unit tests? That is,
> > lets say
> > I had a function like this
> > 
> > void func(int x)
> > in
> > {
> > 
> >     assert(x < 8);
> > 
> > }
> > body
> > {
> > 
> >   //...
> > 
> > }
> > 
> > 
> > and I wanted to test to make sure that func() couldn't be called with
> > any int
> > greater or equal to 8, what would I do?
> 
> Hm... unit testing your unit tests :)
> 
> input contracts and unit tests are supposed to be simple, provable code so you don't have to test them.  The above function is obviously a simple example, you don't really need to unit test it (right?), so what would a complicated in contract look like?
> 
> It's also a good idea to avoid complex expressions in unit tests.  If you have a complex expression, split it out into several lines to avoid having to work through the logic in your head.  Performance/minimal LOC is not a goal you need in unit tests/contracts.
> 
> -Steve

Ideally, unit tests and contracts would be simple. However, I do believe that there is some value in verifying that you wrote you contracts correctly. Ideally, you could use a wrapper function/template to call the function to verify that an AssertError was thrown, and the unit test would then be simple.

Of greater value is testing that normal exceptions are thrown when they're supposed to - especially for bad input. Testing that is essentially the same as testing for AssertErrors except that they're regular exceptions, so you don't have the whole issue with Errors not getting cleaned up properly. In the case of normal exceptions though, it is arguably part of the API (albeit not part of the signature), while for AssertError it's a contract which isn't really part of the API so much as verifying that your code is correct.

In any case, I definitely see some value in testing that sort of thing. You don't necessarily want to write unit tests for all of your contracts, but sometimes it can be valuable. I would argue though that you generally _do_ want to write code for normal exceptions that are thrown due to bad input (like with enforce) because you want to guarantee the behavior of the function, and that is part of the behavior and stays regardless of the -release flag.

- Jonathan M Davis

On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis <jmdavisprog@gmail.com> wrote:

> Ideally, unit tests and contracts would be simple. However, I do believe that
> there is some value in verifying that you wrote you contracts correctly.
> Ideally, you could use a wrapper function/template to call the function to
> verify that an AssertError was thrown, and the unit test would then be simple.

I don't see why that wouldn't work.  If you really feel the need to test your tests.

One thing I've found invaluable, especially when writing unit tests for templated items, is to write the unit tests *inside* the item.  Then instantiate the item with all the different parameter types you want.

For example, in dcollections, all my unit tests are generic, and work with all value types that are integral.  Then at the end of the file I just have:

unittest
{
   ArrayList!ubyte al1;
   ArrayList!byte al2;
   ArrayList!ushort al3;
   ...
}

In all, I have about 8-10 different flavors of unit tests that run on each collection type.  I found some very obscure compiler/runtime bugs that way :)

> Of greater value is testing that normal exceptions are thrown when they're
> supposed to - especially for bad input. Testing that is essentially the same as
> testing for AssertErrors except that they're regular exceptions, so you don't
> have the whole issue with Errors not getting cleaned up properly. In the case of
> normal exceptions though, it is arguably part of the API (albeit not part of the
> signature), while for AssertError it's a contract which isn't really part of the
> API so much as verifying that your code is correct.

I whole-heartedly agree, and the code I use for this is:

bool caughtException = false;
try
{
   functionThatShouldThrow(badInput);
}
catch(SpecificException e)
{
   caughtException = true;
}
assert(caughtException);

An example in dcollections: http://www.dsource.org/projects/dcollections/browser/branches/d2/dcollections/HashMap.d#L757

Note another useful idiom I found: In some cases, your unit tests only work for certain template instantiations.  In dcollections, all my input to initialize the collections is done with integral literals.  So if you happened to instantiate an ArrayList!string, for instance, the code would fail to compile, because you can't add (1, 2, 3, 4, 5) to that array list.

So I define an enum bool doUnittest in each object, which looks something like this:

enum bool doUnittest = isIntegral!V;

Then I use the doUnittest flag to statically disable all unit tests for things like ArrayList of string.  You could probably define multiple booleans if necessary if you had unittests that did support strings.

The only annoyance is this boolean is still in the release code, but it is only used at compile time and it does not add to the object size, it's just an extra data point in the static code.

>
> In any case, I definitely see some value in testing that sort of thing. You don't
> necessarily want to write unit tests for all of your contracts, but sometimes it
> can be valuable. I would argue though that you generally _do_ want to write code
> for normal exceptions that are thrown due to bad input (like with enforce)
> because you want to guarantee the behavior of the function, and that is part of
> the behavior and stays regardless of the -release flag.

I agree, you should test your normal exceptions, especially if they are thrown on invalid user input (i.e. deterministic).  contract asserts, I'm not so sure.  But I can see a reason to do that.

-Steve

On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis <jmdavisprog@gmail.com> wrote:


> and I wanted to test to make sure that func() couldn't be called with any int
> greater or equal to 8, what would I do? The best that I can think of is to catch
> an AssertError and ignore it. e.g.
>
> unittest
> {
>     try
>     {
>         func(8);
>         assert(0);
>     }
>     catch(AssertionError ae)
>     {}
> }

BTW, I just realized, this code doesn't test anything.  You want something like this:

unittest
{
   bool caughtAssert = false;
   try
   {
      func(8);
   }
   catch(AssertionError ae)
   {
      caughtAssert = true;
   }
   assert(caughtAssert);
}

-Steve

Steven Schveighoffer wrote:
> On Mon, 16 Aug 2010 12:57:22 -0400, Jonathan M Davis
> <jmdavisprog@gmail.com> wrote:

>> Of greater value is testing that normal exceptions are thrown when
>> they're
>> supposed to - especially for bad input. Testing that is essentially
>> the same as
>> testing for AssertErrors except that they're regular exceptions, so
>> you don't
>> have the whole issue with Errors not getting cleaned up properly. In
>> the case of
>> normal exceptions though, it is arguably part of the API (albeit not
>> part of the
>> signature), while for AssertError it's a contract which isn't really
>> part of the
>> API so much as verifying that your code is correct.
>
> I whole-heartedly agree, and the code I use for this is:
>
> bool caughtException = false;
> try
> {
>    functionThatShouldThrow(badInput);
> }
> catch(SpecificException e)
> {
>    caughtException = true;
> }
> assert(caughtException);

I've used the same. :) We should have something like ensure_throws:

  ensure_throws!SpecificException(functionThatShouldThrow(badInput));

Perhaps some of the existing unit test libraries already have that.

Ali

On Monday, August 16, 2010 14:36:13 Steven Schveighoffer wrote:
> On Sat, 14 Aug 2010 23:49:18 -0400, Jonathan M Davis
> 
> <jmdavisprog@gmail.com> wrote:
> > and I wanted to test to make sure that func() couldn't be called with
> > any int
> > greater or equal to 8, what would I do? The best that I can think of is
> > to catch
> > an AssertError and ignore it. e.g.
> > 
> > unittest
> > {
> > 
> >     try
> >     {
> > 
> >         func(8);
> >         assert(0);
> > 
> >     }
> >     catch(AssertionError ae)
> >     {}
> > 
> > }
> 
> BTW, I just realized, this code doesn't test anything.  You want something like this:
> 
> unittest
> {
>     bool caughtAssert = false;
>     try
>     {
>        func(8);
>     }
>     catch(AssertionError ae)
>     {
>        caughtAssert = true;
>     }
>     assert(caughtAssert);
> }
> 
> -Steve

Ah, that would be true, since assert throws an AssertException. Whoops. It does work for normal exceptions though (assuming that you changed the type in the catch to the appropriate exception type).

- Jonathan M Davis

On Monday, August 16, 2010 14:46:47 Ali Çehreli wrote:
> I've used the same. :) We should have something like ensure_throws:
> 
>    ensure_throws!SpecificException(functionThatShouldThrow(badInput));
> 
> Perhaps some of the existing unit test libraries already have that.

http://d.puremagic.com/issues/show_bug.cgi?id=4644