Jump to page: 1 2 3
Thread overview
How do you test pre-/post-conditions and invariants?
Feb 25, 2011
Magnus Lie Hetland
Feb 25, 2011
spir
Feb 25, 2011
Ary Manzana
Feb 25, 2011
Magnus Lie Hetland
Feb 25, 2011
spir
Feb 25, 2011
Jesse Phillips
Feb 25, 2011
Magnus Lie Hetland
Feb 25, 2011
Jonathan M Davis
Feb 25, 2011
Magnus Lie Hetland
Feb 25, 2011
spir
Feb 26, 2011
Jonathan M Davis
Feb 26, 2011
Magnus Lie Hetland
Feb 26, 2011
Jonathan M Davis
Feb 26, 2011
David Nadlinger
Feb 26, 2011
Magnus Lie Hetland
Feb 26, 2011
David Nadlinger
Feb 27, 2011
Jonathan M Davis
Feb 26, 2011
Magnus Lie Hetland
Feb 27, 2011
Jonathan M Davis
Feb 28, 2011
Magnus Lie Hetland
Feb 26, 2011
spir
February 25, 2011
Or, more generally, how do you test asserts (which is what I'm using in my preconditions etc.)?

As far as I can see, collectException() won't collect errors, which is what assert() throws -- so what's the standard way of writing unit tests for preconditions that use assert? (I.e., test that they will, in fact, throw when you break them.)

-- 
Magnus Lie Hetland
http://hetland.org

February 25, 2011
On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:
> Or, more generally, how do you test asserts (which is what I'm using in my
> preconditions etc.)?
>
> As far as I can see, collectException() won't collect errors, which is what
> assert() throws -- so what's the standard way of writing unit tests for
> preconditions that use assert? (I.e., test that they will, in fact, throw when
> you break them.)

See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

February 25, 2011
On 2/25/11 1:48 PM, spir wrote:
> On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:
>> Or, more generally, how do you test asserts (which is what I'm using
>> in my
>> preconditions etc.)?
>>
>> As far as I can see, collectException() won't collect errors, which is
>> what
>> assert() throws -- so what's the standard way of writing unit tests for
>> preconditions that use assert? (I.e., test that they will, in fact,
>> throw when
>> you break them.)
>
> See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.
>
> Denis

By the way, the link to "Adding Contracts to Java" at the bottom of the page is broken.
February 25, 2011
Magnus Lie Hetland Wrote:

> Or, more generally, how do you test asserts (which is what I'm using in my preconditions etc.)?
> 
> As far as I can see, collectException() won't collect errors, which is what assert() throws -- so what's the standard way of writing unit tests for preconditions that use assert? (I.e., test that they will, in fact, throw when you break them.)

There has been talk of a std.unittest that would help with this, but for now:

unittest {
    try {
         callMeWrong(wrong);
         assert(0);
    catch(AssertError e) {
    }
}

}
February 25, 2011
On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:
> Or, more generally, how do you test asserts (which is what I'm using in
> my preconditions etc.)?
> 
> As far as I can see, collectException() won't collect errors, which is
> what assert() throws -- so what's the standard way of writing unit
> tests for preconditions that use assert? (I.e., test that they will, in
> fact, throw when you break them.)

I think that the reality of the matter is the most of the time people _don't_ check them. And on some level, it doesn't make sense to. It's kind of like asking how people test their unit tests. Unit tests are already testing code. Do you want to be testing them on top of that? And if you do, do you test _that_ code? Where do you stop?

Pre-conditions, post-conditions, and invariants are testing the class or struct that they're on, and to some extent, the code that uses them. So, if you're looking to test them, you're looking to test test code.

And testing post-conditions and invariants in the manner that you're trying to do borders on impossible. What are you going to do, repeat the post-condition or invariant test on the result of the function or on the state of the object that the function was called on after the function was called? That's just doing the test twice. You might as well just re-read the post-conditions and invariants to make sure that you wrote them correctly.

I do see value in testing pre-conditions if you're using exceptions rather than assertions (which means that you're not use in blocks). In that case, you're testing the API to make sure that it does what it's supposed to do. But if you're dealing with assertions, then it's really test code as opposed to API code, and I don't see the same value in testing that. You'd just be testing test code.

Now, assuming that you _do_ want to test that sort of thing (as you obviously want to), some new unit testing functions which would help were recently reviewed on the D newsgroup and voted for inclusion in Phobos. There's a pull request for them, but they haven't been merged in yet, and unless you use the development version of Phobos from git, you'll have to wait for the next release regardless. Those changes _do_ make it so that you can use collectException to collect an Error (though it defaults to catching Exceptions only), but they also include assertThrown and assertNotThrown which effectively assert that the Exception or Error that you expected to be thrown (or not) from a particular expression or function call was indeed thrown (or not). So, you _can_ use that with AssertError to verify your pre-conditions.

However, I would point out that catching Errors is generally a _bad_ idea. If you're careful in unit testing code, you should be okay, but _don't_ do it in normal code, and if you're not careful, you'll still get into trouble with unit testing code. Unlike with Exceptions, when an Error is thrown, scope statements and finally blocks do not run. I don't believe that destructors are run either. It's pretty much assumed that you can't recover from an Error, so no attempt to recover is made. Your program runs a high risk of being in an invalid state after an Error is thrown - AssertError included. In a unit test, if you're dealing with simple stuff, then you're probably okay, but if you have any code in there that would be affected by scope statements, finally blocks, destructors, etc. not running, then you don't want to be catching any Errors and then trying to continue.

So, functions which will help you with such testing are on their way, but they weren't released with the latest release of dmd, and once they have been released, you're going to need to be careful if you use them to test AssertErrors or any other kind of Error.

- Jonathan M Davis
February 25, 2011
On 2011-02-25 19:18:40 +0100, Jesse Phillips said:

> There has been talk of a std.unittest that would help with this, but for now:

> unittest {
>     try {
>          callMeWrong(wrong);
>          assert(0);
>     catch(AssertError e) {
>     }
> }
> 
> }

Ah. I used something like...

auto thrown = 0;
try foo() catch (AssertError) thrown++;
...
assert(thrown == k);

I guess I could wrap it up a bit, or something.

-- 
Magnus Lie Hetland
http://hetland.org

February 25, 2011
On 2011-02-25 17:48:54 +0100, spir said:

> On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:
>> Or, more generally, how do you test asserts (which is what I'm using in my
>> preconditions etc.)?
>> 
>> As far as I can see, collectException() won't collect errors, which is what
>> assert() throws -- so what's the standard way of writing unit tests for
>> preconditions that use assert? (I.e., test that they will, in fact, throw when
>> you break them.)
> 
> See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.
> 
> Denis

Hm. I'm not sure how this answers my question. I know how pre/postconditions etc. work -- I was asking for how to test them in a unit test (i.e., ensure that they'll kick in if you provide faulty intput, for example). Maybe I'm missing something in the doc you referred me to?

-- 
Magnus Lie Hetland
http://hetland.org

February 25, 2011
On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:

> On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:
>> Or, more generally, how do you test asserts (which is what I'm using in
>> my preconditions etc.)?
>> 
>> As far as I can see, collectException() won't collect errors, which is
>> what assert() throws -- so what's the standard way of writing unit
>> tests for preconditions that use assert? (I.e., test that they will, in
>> fact, throw when you break them.)
> 
> I think that the reality of the matter is the most of the time people _don't_
> check them. And on some level, it doesn't make sense to. It's kind of like
> asking how people test their unit tests. Unit tests are already testing code. Do
> you want to be testing them on top of that? And if you do, do you test _that_
> code? Where do you stop?

I guess so. But you could say the same thing about other cases where you throw an exception when you detect that something is wrong -- but those are normally tested, right? Also, the difference here is that the precondition is written as a general "test", whereas my actual test would have specific cases.

For example, I have a test that checks that I don't add the same object twice to some structure, and the check involves some traversal -- code that could potentially be wrong. I wanted to make sure that it wasn't by explicitly adding the same object twice -- code (i.e., my unit test) that most likely could not be wrong.

But I do see your point.

[snip]
> And testing post-conditions and invariants in the manner that you're trying to
> do borders on impossible. What are you going to do, repeat the post-condition or
> invariant test on the result of the function or on the state of the object that
> the function was called on after the function was called? That's just doing the
> test twice.

Right.

> You might as well just re-read the post-conditions and invariants to
> make sure that you wrote them correctly.
> 
> I do see value in testing pre-conditions if you're using exceptions rather than
> assertions (which means that you're not use in blocks). In that case, you're
> testing the API to make sure that it does what it's supposed to do. But if
> you're dealing with assertions, then it's really test code as opposed to API
> code, and I don't see the same value in testing that. You'd just be testing test
> code.

OK. For the practical reason, I refer you to my explanation above. But I guess it's a style issue -- and I'm fine with not testing these things, by all means.

[snip]
> Those changes _do_ make it so that you can use collectException to
> collect an Error (though it defaults to catching Exceptions only), but they also
> include assertThrown and assertNotThrown which effectively assert that the
> Exception or Error that you expected to be thrown (or not) from a particular
> expression or function call was indeed thrown (or not).
> So, you _can_ use that with AssertError to verify your pre-conditions.

OK, thanks.

> However, I would point out that catching Errors is generally a _bad_ idea.
[snip lots of useful stuff]

Thanks for educating me :D

I guess the conclusion will be that I'll focus on keeping my preconditions really simple. (And any utility functions I use in them can then get unit tests of their own instead ;)

-- 
Magnus Lie Hetland
http://hetland.org

February 25, 2011
On 02/25/2011 11:20 PM, Magnus Lie Hetland wrote:
> On 2011-02-25 17:48:54 +0100, spir said:
>
>> On 02/25/2011 04:30 PM, Magnus Lie Hetland wrote:
>>> Or, more generally, how do you test asserts (which is what I'm using in my
>>> preconditions etc.)?
>>>
>>> As far as I can see, collectException() won't collect errors, which is what
>>> assert() throws -- so what's the standard way of writing unit tests for
>>> preconditions that use assert? (I.e., test that they will, in fact, throw when
>>> you break them.)
>>
>> See the page on DbC: http://www.digitalmars.com/d/2.0/dbc.html.
>>
>> Denis
>
> Hm. I'm not sure how this answers my question. I know how pre/postconditions
> etc. work -- I was asking for how to test them in a unit test (i.e., ensure
> that they'll kick in if you provide faulty intput, for example). Maybe I'm
> missing something in the doc you referred me to?

No, sorry, /I/ read your post too superficially: Hadn't caught you wanted to "meta-check" the checks.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

February 25, 2011
On 02/25/2011 11:33 PM, Magnus Lie Hetland wrote:
> On 2011-02-25 20:04:10 +0100, Jonathan M Davis said:
>
>> On Friday, February 25, 2011 07:30:50 Magnus Lie Hetland wrote:
>>> Or, more generally, how do you test asserts (which is what I'm using in
>>> my preconditions etc.)?
>>>
>>> As far as I can see, collectException() won't collect errors, which is
>>> what assert() throws -- so what's the standard way of writing unit
>>> tests for preconditions that use assert? (I.e., test that they will, in
>>> fact, throw when you break them.)
>>
>> I think that the reality of the matter is the most of the time people _don't_
>> check them. And on some level, it doesn't make sense to. It's kind of like
>> asking how people test their unit tests. Unit tests are already testing code. Do
>> you want to be testing them on top of that? And if you do, do you test _that_
>> code? Where do you stop?
>
> I guess so. But you could say the same thing about other cases where you throw
> an exception when you detect that something is wrong -- but those are normally
> tested, right? Also, the difference here is that the precondition is written as
> a general "test", whereas my actual test would have specific cases.
>
> For example, I have a test that checks that I don't add the same object twice
> to some structure, and the check involves some traversal -- code that could
> potentially be wrong. I wanted to make sure that it wasn't by explicitly adding
> the same object twice -- code (i.e., my unit test) that most likely could not
> be wrong.
>
> But I do see your point.


Now that you speak of it, I do agree with you. Just like for code paths or specific inputs that should throw an exception. I systematically check these bits in unittests. It's very easy to have them right, if only by distraction because they are not the points on which with we put most attention, and there often is a test invertion somewhere (and we don't watch them run). And actually, I often find an error somewhere in there.
What I do is trigger them in unitests, check all works as expected (ie an error is raised, and the message is sensible), then comment out but /keep/ the testing code.
Dunno if this makes sense for DbC checkings.


Denis
-- 
_________________
vita es estrany
spir.wikidot.com

« First   ‹ Prev
1 2 3