On Tue, Sep 01, 2015 at 06:14:05PM +0000, deadalnix via Digitalmars-d wrote:
> On Tuesday, 1 September 2015 at 17:35:23 UTC, Jonathan M Davis wrote:
[...]
> >Yeah, any arguments which are basically saying that you need a feature because you shouldn't have to have 100% code coverage is going to fall flat with Walter and Andrei. All code that is released should have 100% test coverage unless it has a really good reason why it can't, and it's egg on the face of the developer who releases the code without that - and no, we don't do a good enough job of that with Phobos, and that _is_ egg on our face. We have good code coverage, but we often don't have 100%, and when we don't, we miss bugs. That test coverage needs to be there, and without it, it's far too easy to release stuff that mostly works but falls apart in the corner cases.
> >
> >So, if there's a feature that can be added which solves the export problems that dicebot is talking about here, but it requires that you unit test your code for it to catch everything, then so be it. I really don't see that as an issue. If it can be done in a simple way that doesn't require unit testing, then fine, but all of that code should be fully unit tested anyway, so arguments based on the premise that you shouldn't need to have full unit test coverage aren't going to convince anyone - especially not the folks who would approve the new feature.
> >
> >- Jonathan M Davis
> 
> No. That is very short sighted.
> 
> A ton of code is not unitestable. GUI as a starter. Random number generator.  Very rare events handling outside the control of the program. Concurent code. Etc...

That would explain why GUI apps tend to be buggy and unreliable, and RNGs tend to have unintended biases.

No, actually, GUIs should be written to be testable (e.g., input should be abstracted via dependency injection so that GUI interactions are scriptable, at least inside a unittest, with expected final states).

RNGs can be unittested for uniform distribution within certain bounds. Unittest does not necessarily mean checking for equality (even though that's the most common usage).

You might be right about concurrent code, though. Unless you make it possible to substitute CPU scheduler via dependency injection, it's probably not unittestable.


T

-- 
May you live all the days of your life. -- Jonathan Swift

On Tuesday, 1 September 2015 at 18:25:06 UTC, H. S. Teoh wrote:
> That would explain why GUI apps tend to be buggy and unreliable, and RNGs tend to have unintended biases.
>
> No, actually, GUIs should be written to be testable (e.g., input should be abstracted via dependency injection so that GUI interactions are scriptable, at least inside a unittest, with expected final states).
>

You can test the framework that way. You can't test the usage of the framework that way, especially if you want rapid iteration.

Change the text on a button, the test is broken. Wait, the translation team changed the text on the button in tagalog ? Why don't these linguist learn of to fix a unittest !

Well, we'd better have an id on the button rather than relying on the text. Now the button is not displaying properly because of an text encoding issue, but nobody knows about it because it has the right id. Damn...

And that is just for one button...

There are solutions, like webdriver tests (or native equivalents) and so on, but they only cover a small portion of the app and tend to be fragile, so using them more is not always a good idea. One tend to only use them for critical flow, like the purchase tunnel on a e commerce, or the registration flow on a social app.

> RNGs can be unittested for uniform distribution within certain bounds. Unittest does not necessarily mean checking for equality (even though that's the most common usage).
>

Sure, and if they work properly, they should fail randomly. That comes very handy with continuous integration. And bisect.

> You might be right about concurrent code, though. Unless you make it possible to substitute CPU scheduler via dependency injection, it's probably not unittestable.
>

It is not unitestable.

On Tuesday, 1 September 2015 at 18:11:02 UTC, deadalnix wrote:
> Ok I get it. I'm not sure that is the right way forward.

I am not sure either but we have lack of ideas in this domain, brainstorming :)

> The same problem arrise for devirtualization and the solution does not cover it. That is a problem.

Can you expand on this problem a bit more please? I know about issue of not knowing if some method is eventually overridden but that seems to be fixed with export within single compilation step (i.e. small static library). Do you mean something different?

> First I'd like to have more compiler checks on template bodies. Andrei and Walter are not super happy with it, but deep down I do think their point is more the result of the clash with the C++ community over static if vs concept than based on any technical merit (the whole thing is a false dichotomy to start with so the clash and its results are nonsensical).

I don't think it is even practical to try pushing in that direction at this point even if I generally agree with statement. Unless you are going to propose to restrict what exported templates can do (0 chance to fly) there still needs to be _some_ solution that works if existing semantics.

> Second, it is most likely a good idea to issue some kind of error when export within template code calls non export code. That most likely needs to be an error. The error would show up with descent test coverage even without the special feature.

Isn't it essentially the same thing I propose, but simply based on convention instead of dedicated feature? It is probably fine but to be honest I think it really does matter to have clear indicator "this is how things are supposed to be done" in this domain. Simply because applying idioms from other languages will be natural course of actions and is unlikely it work in practice.

On Tuesday, 1 September 2015 at 19:42:51 UTC, deadalnix wrote:
> On Tuesday, 1 September 2015 at 18:25:06 UTC, H. S. Teoh wrote:
>> That would explain why GUI apps tend to be buggy and unreliable, and RNGs tend to have unintended biases.
>>
>> No, actually, GUIs should be written to be testable (e.g., input should be abstracted via dependency injection so that GUI interactions are scriptable, at least inside a unittest, with expected final states).
>>
>
> You can test the framework that way. You can't test the usage of the framework that way, especially if you want rapid iteration.
But at least you can unittest all paths in a template so that every called function occures in at least one test - this is all what is needed for the proposed change of "export", or am I wrong here?

> Change the text on a button, the test is broken.
A test for some concrete button?
I would say, thats a little too deep. And of course nothing within a library or exported, I would suspect.

>> RNGs can be unittested for uniform distribution within certain bounds. Unittest does not necessarily mean checking for equality (even though that's the most common usage).
>>
> Sure, and if they work properly, they should fail randomly.
No. Tests for distribution may only sometimes need longer until the statistics come into the valid range. Nevertheless that are useful and valuable tests.

>> You might be right about concurrent code, though. Unless you make it possible to substitute CPU scheduler via dependency injection, it's probably not unittestable.
>
> It is not unitestable.
But we have concepts like "thread local per default" and message passing that should make avoiding race conditions easier. And concurrent code doesn't prevent us from testing all paths of templates.

What I meant is, of course you cannot test everything with unittest. It has a reason that there exist other types of test. But that should not hold anybody back from unit-testing as much as possible. If nothing else this is useful because unit-tests are by far the cheapest tests. And within a library, it should always be possible to reach a 100% code coverage with unit tests alone.
For a program with much user interaction this may not be possible, but that is pretty much out of scope for exported templates.

On Wednesday, 2 September 2015 at 00:00:55 UTC, Dicebot wrote:
> On Tuesday, 1 September 2015 at 18:11:02 UTC, deadalnix wrote:
>> Ok I get it. I'm not sure that is the right way forward.
>
> I am not sure either but we have lack of ideas in this domain, brainstorming :)
>
>> The same problem arrise for devirtualization and the solution does not cover it. That is a problem.
>
> Can you expand on this problem a bit more please? I know about issue of not knowing if some method is eventually overridden but that seems to be fixed with export within single compilation step (i.e. small static library). Do you mean something different?
>

That is pretty much the same problem. You'd like to devirtualize method that are not marked final but that are in fact never overriden. The fact that the compiler is blind when it comes to template means that the compiler have a much harder time to know what method are or can be overriden.

>> First I'd like to have more compiler checks on template bodies. Andrei and Walter are not super happy with it, but deep down I do think their point is more the result of the clash with the C++ community over static if vs concept than based on any technical merit (the whole thing is a false dichotomy to start with so the clash and its results are nonsensical).
>
> I don't think it is even practical to try pushing in that direction at this point even if I generally agree with statement. Unless you are going to propose to restrict what exported templates can do (0 chance to fly) there still needs to be _some_ solution that works if existing semantics.
>

Yes I know that this isn't going to fly with Andrei/Walter. Sad as it is pretty much the only solution that solve the problem at its root, rather than proposing a work around some aspect of the problem.

>> Second, it is most likely a good idea to issue some kind of error when export within template code calls non export code. That most likely needs to be an error. The error would show up with descent test coverage even without the special feature.
>
> Isn't it essentially the same thing I propose, but simply based on convention instead of dedicated feature? It is probably fine but to be honest I think it really does matter to have clear indicator "this is how things are supposed to be done" in this domain. Simply because applying idioms from other languages will be natural course of actions and is unlikely it work in practice.

My understanding was that you got an error if you don't instanciate the template in the export unitest block. But maybe I'm wrong.

I'd be in support of an export unitest block that behaves as an export function, and so would give error if it uses non exported symbols.

On Wednesday, 2 September 2015 at 09:28:49 UTC, Dominikus Dittes Scherkl wrote:
>> It is not unitestable.
> But we have concepts like "thread local per default" and message passing that should make avoiding race conditions easier. And concurrent code doesn't prevent us from testing all paths of templates.

I was responding to the statement that you should unittest everything.

That is simply not true. For some code, this is not possible, for some other code, it gives poor noise to signal ratio.

Still, if a significant portion of the code is not unitested, you have a problem.

On 09/02/2015 05:07 PM, deadalnix wrote:
> On Wednesday, 2 September 2015 at 09:28:49 UTC, Dominikus Dittes Scherkl
> wrote:
>>> It is not unitestable.
>> But we have concepts like "thread local per default" and message
>> passing that should make avoiding race conditions easier. And
>> concurrent code doesn't prevent us from testing all paths of templates.
>
> I was responding to the statement that you should unittest everything.
>
> That is simply not true. For some code, this is not possible, for some
> other code, it gives poor noise to signal ratio.

What would be a good litmus test "this code needs/doesn't need unittesting"? -- Andrei

On Friday, 4 September 2015 at 16:26:15 UTC, Andrei Alexandrescu wrote:
> On 09/02/2015 05:07 PM, deadalnix wrote:
>> On Wednesday, 2 September 2015 at 09:28:49 UTC, Dominikus Dittes Scherkl
>> wrote:
>>>> It is not unitestable.
>>> But we have concepts like "thread local per default" and message
>>> passing that should make avoiding race conditions easier. And
>>> concurrent code doesn't prevent us from testing all paths of templates.
>>
>> I was responding to the statement that you should unittest everything.
>>
>> That is simply not true. For some code, this is not possible, for some
>> other code, it gives poor noise to signal ratio.
>
> What would be a good litmus test "this code needs/doesn't need unittesting"? -- Andrei

IMO, is the test going to be reliable or break easily is probably the most important parameter. You want unitests to have a high noise to signal ratio so they can be part of the build and/or run on every PR.

There is obviously a grey area where reasonable people will disagree on the cost benefit ratio.

A good litmus test would be "is there a lot of parameter that aren't under my control that affect this piece of code ?"

On 09/04/2015 12:54 PM, deadalnix wrote:
> On Friday, 4 September 2015 at 16:26:15 UTC, Andrei Alexandrescu wrote:
>> On 09/02/2015 05:07 PM, deadalnix wrote:
>>> On Wednesday, 2 September 2015 at 09:28:49 UTC, Dominikus Dittes Scherkl
>>> wrote:
>>>>> It is not unitestable.
>>>> But we have concepts like "thread local per default" and message
>>>> passing that should make avoiding race conditions easier. And
>>>> concurrent code doesn't prevent us from testing all paths of templates.
>>>
>>> I was responding to the statement that you should unittest everything.
>>>
>>> That is simply not true. For some code, this is not possible, for some
>>> other code, it gives poor noise to signal ratio.
>>
>> What would be a good litmus test "this code needs/doesn't need
>> unittesting"? -- Andrei
>
> IMO, is the test going to be reliable or break easily is probably the
> most important parameter.

That's sensible, thanks, though I fail to come with many examples besides low-level networking. -- Andrei