On Tuesday, 29 March 2016 at 20:50:57 UTC, Seb wrote:
> Okay I see that for the long run we need a better way to handle the testing infrastructure :/

Actually the idea of achieving "100% coverage" is that we test every line at least once and don't let this testing by done by users.
So to refresh the discussion - there were two general ideas

1) Find "bad", "dangerous" modules, e.g.

/xml.d 64%
/zlib.d 60%
/experimental/allocator/typed.d 54%
/experimental/allocator/building_blocks/segregator.d 50%
/experimental/allocator/building_blocks/bucketizer.d 48%
/encoding.d 66%
/container/binaryheap.d 59%
/digest/digest.d 72%

2) Increase coverage for generic, platform-independent modules like std.algorithm to 100%

I know it's a lot of work, but shouldn't that make our jobs us maintainers easier (=catch the bugs before you have to fix them?). Therefore I am bumping this ;-)

On Wednesday, 25 May 2016 at 02:34:44 UTC, Seb wrote:
> On Tuesday, 29 March 2016 at 20:50:57 UTC, Seb wrote:
>> Okay I see that for the long run we need a better way to handle the testing infrastructure :/
>
> Actually the idea of achieving "100% coverage" is that we test every line at least once and don't let this testing by done by users.
> So to refresh the discussion - there were two general ideas
>
> 1) Find "bad", "dangerous" modules, e.g.
>
> /xml.d 64%
> /zlib.d 60%
> /experimental/allocator/typed.d 54%
> /experimental/allocator/building_blocks/segregator.d 50%
> /experimental/allocator/building_blocks/bucketizer.d 48%
> /encoding.d 66%
> /container/binaryheap.d 59%
> /digest/digest.d 72%
>
> 2) Increase coverage for generic, platform-independent modules like std.algorithm to 100%
>
> I know it's a lot of work, but shouldn't that make our jobs us maintainers easier (=catch the bugs before you have to fix them?). Therefore I am bumping this ;-)

Just a quick follow-up on this. I finally managed to work a bit on the export to Codecov.io - it's pretty neat as it will warn reviewers if the coverage decreases due to changes in a PR.
Moreover according to Codecov.io, Phobos currently has an overall coverage of 88.15% (the actual coverage is a bit higher, because I had to disable a few tests for Travis and missing, "uncovered" lines are also reported for e.g. deprecated lines).

So if someone wants to improve not well-tested modules, at [1] is a list of all modules in Phobos with their coverage. Some highlights include:

regex: 51%
encoding.d: 62%
mathspecial: 57%
mmfile: 61%
process: 69%
socket: 66%
zlib: 60%

Hopefully in a few days this will be part of Phobos (see [2] for details) and thus automatically updated ;-)

[1] https://codecov.io/gh/wilzbach/phobos/tree/5fc9eb90076101c0266fb3491ac68527d3520fba/std
[2] https://github.com/dlang/phobos/pull/4587

On Sunday, 10 July 2016 at 02:38:07 UTC, Seb wrote:
> On Wednesday, 25 May 2016 at 02:34:44 UTC, Seb wrote:
>> On Tuesday, 29 March 2016 at 20:50:57 UTC, Seb wrote:
>>> Okay I see that for the long run we need a better way to handle the testing infrastructure :/
>>
>> Actually the idea of achieving "100% coverage" is that we test every line at least once and don't let this testing by done by users.
>> So to refresh the discussion - there were two general ideas
>>
>> 1) Find "bad", "dangerous" modules, e.g.
>>
>> /xml.d 64%
>> /zlib.d 60%
>> /experimental/allocator/typed.d 54%
>> /experimental/allocator/building_blocks/segregator.d 50%
>> /experimental/allocator/building_blocks/bucketizer.d 48%
>> /encoding.d 66%
>> /container/binaryheap.d 59%
>> /digest/digest.d 72%
>>
>> 2) Increase coverage for generic, platform-independent modules like std.algorithm to 100%
>>
>> I know it's a lot of work, but shouldn't that make our jobs us maintainers easier (=catch the bugs before you have to fix them?). Therefore I am bumping this ;-)
>
> Just a quick follow-up on this. I finally managed to work a bit on the export to Codecov.io - it's pretty neat as it will warn reviewers if the coverage decreases due to changes in a PR.
> Moreover according to Codecov.io, Phobos currently has an overall coverage of 88.15% (the actual coverage is a bit higher, because I had to disable a few tests for Travis and missing, "uncovered" lines are also reported for e.g. deprecated lines).
>
> So if someone wants to improve not well-tested modules, at [1] is a list of all modules in Phobos with their coverage. Some highlights include:
>
> regex: 51%
> encoding.d: 62%
> mathspecial: 57%
> mmfile: 61%
> process: 69%
> socket: 66%
> zlib: 60%
>
> Hopefully in a few days this will be part of Phobos (see [2] for details) and thus automatically updated ;-)
>
> [1] https://codecov.io/gh/wilzbach/phobos/tree/5fc9eb90076101c0266fb3491ac68527d3520fba/std
> [2] https://github.com/dlang/phobos/pull/4587

Short update - the experiment is now live. Please help to kill the uncovered bits in Phobos :)
A short overview of features CodeCov provides:

1) A CodeCov bot will warn the reviewers if (a) the code coverage decreased or (b) there are new, unhit lines added within a PR.

2) For every commit one can browse the current code coverage of all modules
https://codecov.io/gh/dlang/phobos/tree/77bee525787a90759211dfeb7103ca608bb44bf0/std

3) They provide a handy dashboard with an overview of historic information

https://codecov.io/gh/dlang/phobos

(It's a bit empty atm, it might look like this: https://codecov.io/gh/libmir/mir)

4) CodeCov has a bunch of other features, for example browser extensions that integrate the code coverage report directly onto Github diffs:

https://github.com/codecov/browser-extension

That being said the current approach isn't perfect (see the PR for details), but it's hopefully a start to pay more attention to code coverage ;-)

On 7/28/2016 7:16 AM, Seb wrote:
> 4) CodeCov has a bunch of other features, for example browser extensions that
> integrate the code coverage report directly onto Github diffs:
>
> https://github.com/codecov/browser-extension

I installed the browser extension for Chrome, but it says "no coverage" for every Phobos PR I tried it on. Also, the umbrella icon is greyed out. Something's amiss?

On 7/28/2016 7:16 AM, Seb wrote:
> Short update - the experiment is now live. Please help to kill the uncovered
> bits in Phobos :)

Thank you very much for doing this! Raising the visibility of code coverage will push towards much better unittesting of Phobos.

On Thursday, 28 July 2016 at 17:56:49 UTC, Walter Bright wrote:
> On 7/28/2016 7:16 AM, Seb wrote:
>> 4) CodeCov has a bunch of other features, for example browser extensions that
>> integrate the code coverage report directly onto Github diffs:
>>
>> https://github.com/codecov/browser-extension
>
> I installed the browser extension for Chrome, but it says "no coverage" for every Phobos PR I tried it on. Also, the umbrella icon is greyed out. Something's amiss?

Hmm I guess we bumped into this issue: https://github.com/codecov/browser-extension/issues/22 - last time their support team was pretty fast, so I hope we can resolve this issue soon :)

Do you see the Code coverage overlay for this diff?

https://github.com/dlang/phobos/commit/6db08d3dadb007d930a4042a6140ca4fb22ea540

and on this file?

https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d

Older PRs don't show a coverage report, because the report needs to be uploaded & they don't have this in their `.travis.yml` yet. However rebasing should fix this.

On 7/28/2016 11:51 AM, Seb wrote:
> Do you see the Code coverage overlay for this diff?
>
> https://github.com/dlang/phobos/commit/6db08d3dadb007d930a4042a6140ca4fb22ea540

Yes. Took me a moment to realize the green box meant covered.

> and on this file?
>
> https://github.com/dlang/phobos/blob/master/std/algorithm/iteration.d

Yes.

> Older PRs don't show a coverage report, because the report needs to be uploaded
> & they don't have this in their `.travis.yml` yet. However rebasing should fix
> this.

Thanks! Looks like I'll rebase all of my PRs and we'll see how it works!

On 7/28/2016 11:51 AM, Seb wrote:
> Older PRs don't show a coverage report, because the report needs to be uploaded
> & they don't have this in their `.travis.yml` yet. However rebasing should fix
> this.

I rebased https://github.com/dlang/phobos/pull/4657 but it still shows [No coverage]

On Thursday, 28 July 2016 at 20:30:50 UTC, Walter Bright wrote:
> On 7/28/2016 11:51 AM, Seb wrote:
>> Older PRs don't show a coverage report, because the report needs to be uploaded
>> & they don't have this in their `.travis.yml` yet. However rebasing should fix
>> this.
>
> I rebased https://github.com/dlang/phobos/pull/4657 but it still shows [No coverage]

... but it shows that those three lines are never hit ;-)

As far as I understood [1] the problem is that the CodeCov browser extension has troubles handling merge commits. I will ping you once they resolved the issue and the PR view is working.

For now you can click on "Last update 6db08d3...bb94012" to see the diff with overlayed code coverage.

[1] https://github.com/codecov/browser-extension/issues/22

On 7/28/2016 1:46 PM, Seb wrote:
> For now you can click on "Last update 6db08d3...bb94012" to see the diff with
> overlayed code coverage.

I don't see where that link is.