On 8/14/14, 11:26 AM, bearophile wrote:
> Andrei Alexandrescu:
>> Destroy https://issues.dlang.org/show_bug.cgi?id=13291?
>
> I'd like a way to test nested functions:

That's in keep with the turtles principle, but wouldn't practically be too much? -- Andrei

On 8/14/14, 1:25 PM, Walter Bright wrote:
> On 8/13/2014 9:02 PM, H. S. Teoh via Digitalmars-d wrote:
>> 4) How much can you realistically test on a generic type argument T,
>> that you can't already cover with concrete types? I'm finding that the
>> per-instantiation behaviour of unittest blocks inside templated
>> structs/classes is rarely desired, esp. when you write ddoc unittests
>> (because you want code examples in the docs to involve concrete types,
>> not abstract types, otherwise they are of limited use to the reader).
>> Because of this, I often move unittests outside the aggregate or enclose
>> them in static if's. This suggests that perhaps per-instantiation
>> unittests are only of limited utility.
>
> Just to expand on this, one of the great advantages of template
> functions and separate unittests is the unittest can instantiate the
> template function with "mocked up" arguments. Instantiating the unittest
> for every type passed to it would defeat that method.

Clearly unittests that mock up arguments etc. are a useful device for unittesting. But the point of generic unittests is a tad different and has to do with a fundamental difference between C++'s and D's approach to generics.

In C++ it's entirely acceptable to fail to instantiate a template. In fact the C++ idiom SFINAE is literally based on that - "Substitution Failure Is Not An Error". There's a bunch of detail to it but bottom line it's totally fine for a template to fail to instantiate, both technically and socially (e.g. you get syntax errors in the template code etc). There's a good amount of dissatisfaction in the C++ community about that, which has led to a lot of work on concepts.

In D, it's not acceptable to fail to instantiate; a template should either instantiate and work, get filtered out by a template constraint, or fail with information by means of a static assert. Random syntax errors inside the template are considered poor style.

It follows that once a D template gets instantiated, it's supposed to work as expected for the entire range of types it was meant to.


Andrei

On Thu, Aug 14, 2014 at 02:34:02PM -0700, Andrei Alexandrescu via Digitalmars-d wrote: [...]
> Clearly unittests that mock up arguments etc. are a useful device for unittesting. But the point of generic unittests is a tad different and has to do with a fundamental difference between C++'s and D's approach to generics.
> 
> In C++ it's entirely acceptable to fail to instantiate a template. In fact the C++ idiom SFINAE is literally based on that - "Substitution Failure Is Not An Error". There's a bunch of detail to it but bottom line it's totally fine for a template to fail to instantiate, both technically and socially (e.g. you get syntax errors in the template code etc). There's a good amount of dissatisfaction in the C++ community about that, which has led to a lot of work on concepts.
> 
> In D, it's not acceptable to fail to instantiate; a template should either instantiate and work, get filtered out by a template constraint, or fail with information by means of a static assert. Random syntax errors inside the template are considered poor style.
> 
> It follows that once a D template gets instantiated, it's supposed to work as expected for the entire range of types it was meant to.
[...]

How does this relate to writing generic unittests? Since the incoming types are generic, you can't assume anything about them beyond what the function itself already assumes, so how would the unittest test anything beyond what the function already does?

For example, if the function performs x+y on two generic arguments x and y, how would a generic unittest check whether the result is correct, since you can't assume anything about the concrete values of x and y? The only way the unittest can check the result is to see if it equals x+y, which defeats the purpose because it's tautological with what the function already does, and therefore proves nothing at all.


T

-- 
"640K ought to be enough" -- Bill G. (allegedly), 1984. "The Internet is not a primary goal for PC usage" -- Bill G., 1995. "Linux has no impact on Microsoft's strategy" -- Bill G., 1999.

On Thursday, 14 August 2014 at 21:34:00 UTC, Andrei Alexandrescu wrote:
> It follows that once a D template gets instantiated, it's supposed to work as expected for the entire range of types it was meant to.

Yes, but in my experience, it's rarely the case that you can actually test a template generically, because construction is not generic, so the values to use are hard to generate generically, and then creating the values to test the results against are frequently just as hard. Where you can test something generically, it's great, but it's rarely practical from what I've seen. We should support it regardless (and we do currently by separating the template declaration from the function declaration), and adding the shorter syntax for it that you're suggesting may be a good idea, but I really don't see these types of tests as being effective in most cases even though it's fantastic when you can write such tests.

Also, you still have to specifically instantiate functions like this outside of the unittest block that you're proposing - particularly if this is in a library, because in that case, the function may not actually be used outside of its tests, and the unit tests aren't going to run when someone links in the library. And if that's the case, you arguably might as well just write tests for each of the individual types.

So, ultimately, while these sort of tests can be good, I really don't think that they're effective very often.

- Jonathan M Davis

On Thu, Aug 14, 2014 at 10:37:22PM +0000, Jonathan M Davis via Digitalmars-d wrote: [...]
> Also, you still have to specifically instantiate functions like this outside of the unittest block that you're proposing - particularly if this is in a library, because in that case, the function may not actually be used outside of its tests, and the unit tests aren't going to run when someone links in the library. And if that's the case, you arguably might as well just write tests for each of the individual types.
> 
> So, ultimately, while these sort of tests can be good, I really don't think that they're effective very often.
[...]

Yeah, when I first read Andrei's proposal, I actually quite liked it... until I started thinking about use cases where it would be handy, and then I suddenly realized that I can't think of any! All *useful* unittests I can think of involve the use of concrete values of concrete types. Having only .init guaranteed to exist for a generic type T (and sometimes not even that if you have a Voldemort) greatly cripples the utility of the unittest.

So I'd love to see an actual, non-trivial, motivating example of where this kind of per-instantiation unittest is actually useful. That may yet change my mind about this issue.  ;-)


T

-- 
The richest man is not he who has the most, but he who needs the least.

On 8/14/14, 3:34 PM, H. S. Teoh via Digitalmars-d wrote:
> How does this relate to writing generic unittests? Since the incoming
> types are generic, you can't assume anything about them beyond what the
> function itself already assumes, so how would the unittest test anything
> beyond what the function already does?

Check the workings of the function via an alternate algorithm for example. There's plenty of examples, including unittests inside a parameterized struct/class test methods within that class.

> For example, if the function performs x+y on two generic arguments x and
> y, how would a generic unittest check whether the result is correct,
> since you can't assume anything about the concrete values of x and y?
> The only way the unittest can check the result is to see if it equals
> x+y, which defeats the purpose because it's tautological with what the
> function already does, and therefore proves nothing at all.

We could ask the same question about x+y for int in particular: it's a primitive so there's not much to test.

This does bring up the interesting point that we need a way to generate random values of generic types. http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck1 comes to mind.


Andrei

On 8/14/14, 3:37 PM, Jonathan M Davis wrote:
> On Thursday, 14 August 2014 at 21:34:00 UTC, Andrei Alexandrescu wrote:
>> It follows that once a D template gets instantiated, it's supposed to
>> work as expected for the entire range of types it was meant to.
>
> Yes, but in my experience, it's rarely the case that you can actually
> test a template generically, because construction is not generic, so the
> values to use are hard to generate generically, and then creating the
> values to test the results against are frequently just as hard.

We need a way to generate correct but random values generically.

> Where
> you can test something generically, it's great, but it's rarely
> practical from what I've seen. We should support it regardless (and we
> do currently by separating the template declaration from the function
> declaration), and adding the shorter syntax for it that you're
> suggesting may be a good idea, but I really don't see these types of
> tests as being effective in most cases even though it's fantastic when
> you can write such tests.

My thesis here is, if you get to instantiate it you must be able to trigger some testing for it.

> Also, you still have to specifically instantiate functions like this
> outside of the unittest block that you're proposing - particularly if
> this is in a library, because in that case, the function may not
> actually be used outside of its tests, and the unit tests aren't going
> to run when someone links in the library. And if that's the case, you
> arguably might as well just write tests for each of the individual types.

Agreed.

> So, ultimately, while these sort of tests can be good, I really don't
> think that they're effective very often.

I feel this is fertile ground. We should think more about it.


Andrei

On 8/14/14, 3:46 PM, H. S. Teoh via Digitalmars-d wrote:
> Having only .init guaranteed to exist for a generic type T (and
> sometimes not even that if you have a Voldemort) greatly cripples the
> utility of the unittest.

I'm thinking of T.random as a sort of generator of random correct objects of type T...

Andrei

On Friday, 15 August 2014 at 00:52:35 UTC, Andrei Alexandrescu wrote:
> On 8/14/14, 3:46 PM, H. S. Teoh via Digitalmars-d wrote:
>> Having only .init guaranteed to exist for a generic type T (and
>> sometimes not even that if you have a Voldemort) greatly cripples the
>> utility of the unittest.
>
> I'm thinking of T.random as a sort of generator of random correct objects of type T...
>
> Andrei

So now every type will need to declare a way for randomly instances of it?

At any rate, random is bad for unit testing. Unit tests need to be deterministic, because when a unit test fails you want to debug it to see what's wrong. If the unit test generates some random data, in your debug it'll probably generate different data than the one it generated when the test failed, and the problem may or may not repeat.

Of course, you can always use a random function with a predefined seed to solve that, but there are more problems yet. Unit tests usually operate by setting up the data, calling a function, and testing it's result and/or side-effects. If you set up the data by hand, you know what the results and side-effects are supposed to be.

For example, let's take a look at the example unit test for CRC(http://dlang.org/phobos/std_digest_crc.html#CRC32):

    //Simple example, hashing a string using crc32Of helper function
    ubyte[4] hash = crc32Of("abc");
    //Let's get a hash string
    assert(crcHexString(hash) == "352441C2");

Here, we know what the result is supposed to be, because we can calculate it manually(or with external tools that we know they are correct) and write the literal result in the unit test. But How will you check that `crc32Of(string.random())` returns a correct result?

On Friday, 15 August 2014 at 00:51:56 UTC, Andrei Alexandrescu wrote:
>> So, ultimately, while these sort of tests can be good, I really don't
>> think that they're effective very often.
>
> I feel this is fertile ground. We should think more about it.

Oh, I'm not at all opposed to trying to leverage this. If we can, that would be fantastic. I just haven't seen much practical uses of it thus far. And I'm not opposed to adding the proposed feature, but at the moment, it seems like it's just providing a simpler syntax for something that we can already do that hasn't yet shown to be very useful.

- Jonathan M Davis