On Saturday, March 10, 2012 09:04:42 H. S. Teoh wrote:
> On Sat, Mar 10, 2012 at 08:48:32AM -0800, Jonathan M Davis wrote:
> > On Saturday, March 10, 2012 16:53:42 Jacob Carlborg wrote:
> > > On 2012-03-09 18:59, H. S. Teoh wrote:
> [.[..]
> 
> > > > I agree, though, that catching Errors outside of unittests is a very, very bad idea in general.
> > > 
> > > I don't see what's so bad in making AssertError an exception instead of an error.
> > 
> > Then
> > 
> > catch(Excetion e) {}
> > 
> > would catch it. This would be a huge problem in normal code. Assertions are supposed to kill the program. This is specifically mentioned in TDPL.
> 
> [...]
> 
> It seems that what we need is for assert errors called from within a unittest to throw Exceptions, whereas assert errors not called from a unittest should throw Errors. Something like:
> 
> 	auto func(T args...)
> 	in {
> 		if (args.are_bad()) {
> 			static if (__called_from_unittest__)
> 				throw new AssertException(...);
> 			else
> 				throw new AssertError(...);
> 		}
> 	} body { ... }

Maybe, maybe not. Certainly, having normal assertions throw Exceptions would be _very_ bad, but I'm not entirely convinced that throwing Exceptions instead of Errors in unit tests would be all that much better. They can still have catch(Exception e){} in them. And the fact that you'd then be dealing with two separate types - both coming from assert - could be a problem. If you were going to change something, it would probably be better to make it so that AssertErrors are guaranteed to hit scope statements, finally blocks, and destructors inside of unittest blocks, but that would probably complicate the compiler a fair bit (or maybe it would be druntime code), since it would have to handle the AssertError throwing logic differently depending on whether it's in a unittest block or not.

> However. The unittest is usually testing for a *specific* assert, so turning *all* asserts into an exception is very bad. For example, if the unittest is checking for assert errors caused by bad arguments, but while running the code it throws an assert error caused by memory corruption. It would be *very* bad for the unittest to catch this error and then continue running, because now the program is in a bad state.
> 
> Hmm. The more I think about it, the more I'm leaning towards simplifying contracts so that most, if not all, of any complicated checks happen in an external function (i.e. outside the contract), which can be unittested separately. Unittesting things that throw assertion errors is just a minefield rife with potentially very nasty problems.

If you really want to be testing your tests, then it probably does make a lot of sense to use separate functions for them which you can then unit test. And assuming that the compiler will let you, if you wanted to, you could then templatize them on exception type and use enforce rather than assert. So, in the contracts themselves, you do testFunc!AssertError(args), but in the tests you do testFunc!MyException(args). If there are definitely no destructors, scope statements, and finally blocks in your test functions though, you might as well just catch AssertError.

- Jonathan M Davis

On Sat, Mar 10, 2012 at 11:09:04AM -0800, Jonathan M Davis wrote:
> On Saturday, March 10, 2012 09:04:42 H. S. Teoh wrote:
[...]
> > Hmm. The more I think about it, the more I'm leaning towards simplifying contracts so that most, if not all, of any complicated checks happen in an external function (i.e. outside the contract), which can be unittested separately. Unittesting things that throw assertion errors is just a minefield rife with potentially very nasty problems.
> 
> If you really want to be testing your tests, then it probably does make a lot of sense to use separate functions for them which you can then unit test. And assuming that the compiler will let you, if you wanted to, you could then templatize them on exception type and use enforce rather than assert. So, in the contracts themselves, you do testFunc!AssertError(args), but in the tests you do testFunc!MyException(args). If there are definitely no destructors, scope statements, and finally blocks in your test functions though, you might as well just catch AssertError.
[...]

My point was that contracts won't *need* to be tested at all, if all they do is call a couple o' functions to do the actual complicated verification. You can then just unittest said functions, and be assured that the contract does what you think it does.

	bool veryComplicatedVerification(real[] args, real result) {
		...
	}

	unittest {
		assert(veryComplicatedVerification(sample_data,
			known_good_result));
		assert(!veryComplicatedVerification(sample_data,
			known_bad_result));
	}

	real veryComplicatedComputation(real[] args)
	out(result) {
		assert(veryComplicatedVerification(args, result));
	} body {
		auto result = lotsOfDotDotDotMagic(args);
		return result;
	}


T

-- 
If you compete with slaves, you become a slave. -- Norbert Wiener