On 3/14/13 4:57 PM, Timon Gehr wrote:
> On 03/14/2013 06:51 PM, Andrei Alexandrescu wrote:
>> On 3/14/13 1:48 PM, H. S. Teoh wrote:
>>> I don't agree. Phobos is a prime example. Does Phobos have unittests?
>>> Yes, and lots of them. Does it still have non-compilable template
>>> instantiations? Yes, because unittests can't cover all possibilities --
>>> there are too many possible combinations of template arguments. There
>>> are bound to be untested combinations which don't work but we're unaware
>>> of.
>>
>> If you found a few, that would be great. I don't think you'll have an
>> easy time.
>>
>> Andrei
>>
>
> Challenge accepted. Clearly the Phobos developers do not instantiate
> their templates before shipping them. :o)
>
> The following breaks most of std.range, and most of std.algorithm could
> likely be broken too, but I am too lazy to investigate.
[snip]

Nice point. The problem is an immutable TrollFace should not qualify pass the range tests. This could happen if there was a similar bug in a concept definition. So your argument does not move the ball on concepts vs static if.

Andrei

On 3/14/2013 1:57 PM, Timon Gehr wrote:
> Challenge accepted. Clearly the Phobos developers do not instantiate their
> templates before shipping them. :o)
>
> The following breaks most of std.range, and most of std.algorithm could likely
> be broken too, but I am too lazy to investigate.

Compiling with -cov -unittest shows:

std.range 86% covered

"good"

std.algorithm 96% covered

"excellent"

On 3/14/13 2:38 PM, Jonathan M Davis wrote:
> On Thursday, March 14, 2013 13:26:18 Andrei Alexandrescu wrote:
>> Walter has measured coverage of Phobos unittests a couple of times, it's
>> very high. But I agree it would be nice to have it as a target.
>
> Though this is exactly a case where 100% unit test coverage doesn't mean much.
> All that means is that each path has been instantiated and run. It doesn't
> mean that it's covered enough possible instantiations to properly test the
> template.

Concepts won't help there either.

Andrei

On 3/14/13 2:29 PM, Jonathan M Davis wrote:
> The main problem is not people who don't unit test their templates but the
> fact that it's pretty much impossible to cover every possible instantiation of
> a template, and so it can be pretty easy to miss stuff.

Concepts don't do that either. I do agree we could and should do a better job at testing templates against bare minimum structures that pass their constraints (e.g. test input range algorithms with true input range, which are indeed one-pass). Again, concepts won't help there either - it's not like we claim to support input ranges and then call .save against them. That's not a frequent bug.

Andrei

On Thu, Mar 14, 2013 at 10:26:54PM +0100, bearophile wrote: [[...]
> And 100% code coverage is not enough even for regular code, that's why they have invented this for Java:
> 
> http://dev.theladders.com/2013/02/mutation-testing-with-pit-a-step-beyond-normal-code-coverage/
[...]

We need this in D. I want this in D. :)

Recently, while writing a new D module, the thought occurred to me that perhaps my unittests aren't enough. Sure, it's good to have tons of unittests, but how effective are they? Do they really fail when I think they should, or do they just silently pass anyway when a bug is introduced?

This paper answered that question for me. :)

I think maybe the dustmite code can be adapted to do mutation testing, esp. the part that identifies what can be deleted from the code? I'm not sure exactly how dustmite identifies declarations/statements to be deleted, but that could be a start on doing mutation testing.


T

-- 
Why can't you just be a nonconformist like everyone else? -- YHL

On 3/14/13 4:37 PM, Dmitry Olshansky wrote:
> Point taken. That doesn't detract us from:
> a) fixing issues with -cov

Yes please (are there bugzilla entries etc)?

> It counts time a LOC is executed. Would nice to add instantations
> counter (determined at compile time) as well so as to see if
> declarations are all covered. That + CTFE-only counted as 0.

Not getting this. Example?

> b) unifying template fuzzy testing in Phobos
>
> We have lots of these wheels reinvented across Phobos alone.

Like those dummy ranges in std.algorithm?

> c) prototyping other higher-level tools to aid debugging generic code

Yeppers!


Andrei

On Thu, Mar 14, 2013 at 05:54:05PM -0400, Andrei Alexandrescu wrote:
> On 3/14/13 2:38 PM, Jonathan M Davis wrote:
> >On Thursday, March 14, 2013 13:26:18 Andrei Alexandrescu wrote:
> >>Walter has measured coverage of Phobos unittests a couple of times, it's very high. But I agree it would be nice to have it as a target.
> >
> >Though this is exactly a case where 100% unit test coverage doesn't mean much.  All that means is that each path has been instantiated and run. It doesn't mean that it's covered enough possible instantiations to properly test the template.
> 
> Concepts won't help there either.
[...]

It does help. For example, if the code wrongly assumes mutability for a particular template type, then it may work for most test cases (frankly, I find const/immutable unittest coverage in Phobos very poor) but fail when some daring user passes const(T) instead of T to the template. For example, you may have accidentally written the equivalent of:

	auto func(T)(T t) {
		Unqual!T u = t; // <-- spot the bug
		...
	}

Under a concepts system, this would be caught early because the compiler would detect a concept mismatch (Unqual!T != T) when analysing the template code.

Currently, though, if there is no unittest that tries instantiating the template with const(T), the bug goes undetected, because in all *tested* instantiations, Unqual!T == T.


T

-- 
Famous last words: I wonder what will happen if I do *this*...

On 3/14/2013 3:02 PM, H. S. Teoh wrote:
> Under a concepts system, this would be caught early because the compiler
> would detect a concept mismatch (Unqual!T != T) when analysing the
> template code.

How would that be different from simply adding a template constraint?

On 3/14/13 6:02 PM, H. S. Teoh wrote:
> On Thu, Mar 14, 2013 at 05:54:05PM -0400, Andrei Alexandrescu wrote:
>> On 3/14/13 2:38 PM, Jonathan M Davis wrote:
>>> On Thursday, March 14, 2013 13:26:18 Andrei Alexandrescu wrote:
>>>> Walter has measured coverage of Phobos unittests a couple of times,
>>>> it's very high. But I agree it would be nice to have it as a target.
>>>
>>> Though this is exactly a case where 100% unit test coverage doesn't
>>> mean much.  All that means is that each path has been instantiated
>>> and run. It doesn't mean that it's covered enough possible
>>> instantiations to properly test the template.
>>
>> Concepts won't help there either.
> [...]
>
> It does help. For example, if the code wrongly assumes mutability for a
> particular template type, then it may work for most test cases (frankly,
> I find const/immutable unittest coverage in Phobos very poor) but fail
> when some daring user passes const(T) instead of T to the template. For
> example, you may have accidentally written the equivalent of:
>
> 	auto func(T)(T t) {
> 		Unqual!T u = t; //<-- spot the bug
> 		...
> 	}
>
> Under a concepts system, this would be caught early because the compiler
> would detect a concept mismatch (Unqual!T != T) when analysing the
> template code.
>
> Currently, though, if there is no unittest that tries instantiating the
> template with const(T), the bug goes undetected, because in all *tested*
> instantiations, Unqual!T == T.

Template constraints start from the most permissive end of the spectrum: by default there's no verification, and constraints add verification.

With typeclasses it's the other way around: by default nothing is allowed, so code needs to add permissions explicitly.

I agree that in that regard typeclasses are better than template constraints. Of course there are many other aspects to be considered when comparing the two.


Andrei

On 03/14/2013 10:48 PM, Andrei Alexandrescu wrote:
> On 3/14/13 4:57 PM, Timon Gehr wrote:
>> ...
>>
>> The following breaks most of std.range, and most of std.algorithm could
>> likely be broken too, but I am too lazy to investigate.
> [snip]
>
> Nice point. The problem is an immutable TrollFace should not qualify
> pass the range tests.

Yes, that is one valid analysis of the situation.

> This could happen if there was a similar bug in a
> concept definition.

Yes. In which case it would be detected while the code is written, which is the point.

> So your argument does not move the ball on concepts
> vs static if.
>...

I'm not intending to move any balls here. The trade-offs are different.