AFAIK dmd supports the `-cov` flag since quite some time, so I would be interested why there's no code coverage bot. There also seems to exist at least two external platforms that allow such code coverage analyzing [1, 2]. Is this already on your roadmap?

[1] https://github.com/codecov/example-d
[2] https://github.com/ColdenCullen/doveralls/issues/16

Anyways it seems pretty easy to hack something quick&dirty together that could be pretty helpful:

```
for f in $(find . -name '*.d' -type f -not -path './c/*'); do ;fe="${f%.*}"; (cd $(dirname "$f") && rdmd -cov -unittest -main "$(basename $f)") && printf "$f " && tail -n1 "$fe".lst | cut -f3 -d ' ' ; done | sort
```

./algorithm/comparison.d 95%
./algorithm/internal.d no
./algorithm/iteration.d 96%
./algorithm/mutation.d 99%
./algorithm/package.d no
./algorithm/searching.d 97%
./algorithm/setops.d 94%
./algorithm/sorting.d 98%
./array.d 95%
./ascii.d 100%
./base64.d 94%
./bigint.d 97%
./bitmanip.d 96%
./compiler.d no
./complex.d 96%
./concurrencybase.d 100%
./container/array.d 94%
./container/binaryheap.d 59%
./container/dlist.d 92%
./container/package.d 1%
./container/rbtree.d 94%
./container/slist.d 95%
./container/util.d 100%
./conv.d 94%
./cstream.d 92%
./datetime.d 94%
./demangle.d 0%
./digest/crc.d 100%
./digest/digest.d 72%
./digest/hmac.d 100%
./digest/md.d 100%
./digest/ripemd.d 100%
./digest/sha.d 80%
./encoding.d 66%
./exception.d 92%
./experimental/allocator/building_blocks/affix_allocator.d 71%
./experimental/allocator/building_blocks/allocator_list.d 81%
./experimental/allocator/building_blocks/bitmapped_block.d 91%
./experimental/allocator/building_blocks/bucketizer.d 48%
./experimental/allocator/building_blocks/fallback_allocator.d 89%
./experimental/allocator/building_blocks/free_list.d 75%
./experimental/allocator/building_blocks/kernighan_ritchie.d 80%
./experimental/allocator/building_blocks/null_allocator.d 56%
./experimental/allocator/building_blocks/package.d no
./experimental/allocator/building_blocks/quantizer.d 69%
./experimental/allocator/building_blocks/region.d 70%
./experimental/allocator/building_blocks/scoped_allocator.d 78%
./experimental/allocator/building_blocks/segregator.d 50%
./experimental/allocator/building_blocks/stats_collector.d 95%
./experimental/allocator/common.d 88%
./experimental/allocator/gc_allocator.d 92%
./experimental/allocator/mmap_allocator.d 100%
./experimental/allocator/showcase.d 100%
./experimental/allocator/typed.d 54%
./experimental/logger/core.d 99%
./experimental/logger/filelogger.d 91%
./experimental/logger/multilogger.d 100%
./experimental/logger/nulllogger.d 100%
./experimental/logger/package.d no
./experimental/ndslice/internal.d 100%
./experimental/ndslice/iteration.d 100%
./experimental/ndslice/package.d 91%
./experimental/ndslice/selection.d 97%
./experimental/ndslice/slice.d 95%
./file.d 93%
./format.d 92%
./functional.d 88%
./internal/cstring.d 93%
./internal/digest/sha_SSSE3.d 0%
./internal/math/biguintcore.d 58%
./internal/math/biguintnoasm.d 75%
./internal/math/biguintx86.d no
./internal/math/errorfunction.d 92%
./internal/math/gammafunction.d 94%
./internal/processinit.d no
./internal/scopebuffer.d 98%
./internal/test/dummyrange.d 100%
./internal/test/uda.d no
./internal/unicode_comp.d 0%
./internal/unicode_decomp.d 0%
./internal/unicode_grapheme.d no
./internal/unicode_norm.d no
./internal/unicode_tables.d 0%
./internal/windows/advapi32.d no
./json.d 93%
./math.d 92%
./mathspecial.d 47%
./meta.d 97%
./mmfile.d 61%
./net/curl.d 68%
./net/isemail.d 84%
./numeric.d 72%
./outbuffer.d 54%
./parallelism.d 88%
./path.d 97%
./random.d 94%
./range/interfaces.d 78%
./range/package.d 92%
./range/primitives.d 90%
./regex/internal/backtracking.d 0%
./regex/internal/generator.d 56%
./regex/internal/kickstart.d 87%
./regex/internal/parser.d 29%
./regex/internal/tests.d 99%
./regex/internal/thompson.d no
./regex/package.d 90%
./signals.d 92%
./socketstream.d 0%
./stdint.d no
./stdio.d 76%
./stdiobase.d 100%
./stream.d 61%
./string.d 99%
./system.d no
./typecons.d 90%
./typetuple.d 80%
./uni.d 91%
./utf.d 90%
./uuid.d 97%
./variant.d 92%
./windows/charset.d no
./windows/iunknown.d no
./windows/registry.d no
./windows/syserror.d no
./xml.d 64%
./zip.d 90%
./zlib.d 60%

On Monday, 28 March 2016 at 01:30:34 UTC, Seb wrote:
> AFAIK dmd supports the `-cov` flag since quite some time, so I would be interested why there's no code coverage bot. There also seems to exist at least two external platforms that allow such code coverage analyzing [1, 2]. Is this already on your roadmap?
>
> [1] https://github.com/codecov/example-d
> [2] https://github.com/ColdenCullen/doveralls/issues/16
>
> Anyways it seems pretty easy to hack something quick&dirty together that could be pretty helpful:

Yep. The difficulties are:

1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.

https://issues.dlang.org/show_bug.cgi?id=14381

2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.

On Monday, 28 March 2016 at 11:29:55 UTC, Vladimir Panteleev wrote:
> Yep. The difficulties are:
>
> 1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.
>
> https://issues.dlang.org/show_bug.cgi?id=14381
>
> 2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.

3. since the coverage is computed at run-time by the application that runs the tests all the static code (+ CTFE) is considered NOT covered. This gives misleading results.

For example std.traits will probably have a bad coverage while actually it's pretty good.

On Monday, 28 March 2016 at 16:10:48 UTC, Basile B. wrote:
> On Monday, 28 March 2016 at 11:29:55 UTC, Vladimir Panteleev wrote:
>> Yep. The difficulties are:
>>
>> 1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.
>>
>> https://issues.dlang.org/show_bug.cgi?id=14381
>>
>> 2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.
>
> 3. since the coverage is computed at run-time by the application that runs the tests all the static code (+ CTFE) is considered NOT covered. This gives misleading results.
>
> For example std.traits will probably have a bad coverage while actually it's pretty good.

Not really an issue since DMD only counts the code that goes into the binary for coverage. Uncovered lines will have seven zeroes before the |, whereas lines with no compiled code will have nothing.

On Monday, 28 March 2016 at 16:44:26 UTC, Vladimir Panteleev wrote:
> On Monday, 28 March 2016 at 16:10:48 UTC, Basile B. wrote:
>> On Monday, 28 March 2016 at 11:29:55 UTC, Vladimir Panteleev wrote:
>>> Yep. The difficulties are:
>>>
>>> 1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.
>>>
>>> https://issues.dlang.org/show_bug.cgi?id=14381
>>>
>>> 2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.
>>
>> 3. since the coverage is computed at run-time by the application that runs the tests all the static code (+ CTFE) is considered NOT covered. This gives misleading results.
>>
>> For example std.traits will probably have a bad coverage while actually it's pretty good.
>
> Not really an issue since DMD only counts the code that goes into the binary for coverage. Uncovered lines will have seven zeroes before the |, whereas lines with no compiled code will have nothing.

Not true. I wouldn't be able to link an example in Phobos but this is a more general issue with coverage. For example take a file with:


template isSomething(T)
{
    bool helper()
    {
        return true; // line 5
    }
    enum isSomething = helper;
}

unittest
{
    bool test;
    static if (isSomething!bool)
    {test = true;}
    assert(test);
}

and compile with -main -unittest -cov then run.
the lst file indicates 00000 for line 5.

Actually with `static if()` and helpers functions executed at compile time, 100% coverage is unreachchable. This is more or less the same that happens with if(__ctfe) branches.

On Monday, 28 March 2016 at 23:45:10 UTC, Basile B. wrote:
> On Monday, 28 March 2016 at 16:44:26 UTC, Vladimir Panteleev wrote:
>> On Monday, 28 March 2016 at 16:10:48 UTC, Basile B. wrote:
>>> On Monday, 28 March 2016 at 11:29:55 UTC, Vladimir Panteleev wrote:
>>>> Yep. The difficulties are:
>>>>
>>>> 1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.
>>>>
>>>> https://issues.dlang.org/show_bug.cgi?id=14381
>>>>
>>>> 2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.
>>>
>>> 3. since the coverage is computed at run-time by the application that runs the tests all the static code (+ CTFE) is considered NOT covered. This gives misleading results.
>>>
>>> For example std.traits will probably have a bad coverage while actually it's pretty good.
>>
>> Not really an issue since DMD only counts the code that goes into the binary for coverage. Uncovered lines will have seven zeroes before the |, whereas lines with no compiled code will have nothing.
>
> Not true. I wouldn't be able to link an example in Phobos but this is a more general issue with coverage. For example take a file with:

Ah, OK. Though, this is something that could be improved in the implementation without changing the language. DMD should not emit code that's not called in the module, and not callable from outside the module due to e.g. being private. Probably would help with template bloat, too. LTO would probably take care of that in theory, I wonder if LDC's or GDC's LTO doesn't have this issue (not that it would help testing Phobos coverage).

On 3/28/2016 9:38 PM, Vladimir Panteleev wrote:
> Ah, OK. Though, this is something that could be improved in the implementation
> without changing the language. DMD should not emit code that's not called in the
> module, and not callable from outside the module due to e.g. being private.
> Probably would help with template bloat, too. LTO would probably take care of
> that in theory, I wonder if LDC's or GDC's LTO doesn't have this issue (not that
> it would help testing Phobos coverage).


Or one could write unit tests specifically for the helper functions. (I've done that.)

On Tuesday, 29 March 2016 at 09:39:49 UTC, Walter Bright wrote:
> On 3/28/2016 9:38 PM, Vladimir Panteleev wrote:
>> Ah, OK. Though, this is something that could be improved in the implementation
>> without changing the language. DMD should not emit code that's not called in the
>> module, and not callable from outside the module due to e.g. being private.
>> Probably would help with template bloat, too. LTO would probably take care of
>> that in theory, I wonder if LDC's or GDC's LTO doesn't have this issue (not that
>> it would help testing Phobos coverage).
>
>
> Or one could write unit tests specifically for the helper functions. (I've done that.)

I've also done so in my user lib. Actually I've particpated to this topic because of this:

https://issues.dlang.org/show_bug.cgi?id=15590

3 monthes ago I had never used D coverage feature. Initially I thought that the coverage file was generated by static analysis (until someone points me https://dlang.org/phobos/core_runtime.html#.dmd_coverDestPath)...

Couldn't the coverage be done at compile time ? (even if I directly see corner cases like functions only called by delegate)

On 3/29/2016 3:33 AM, Basile B. wrote:
> Couldn't the coverage be done at compile time ?

The implementation for that would be utterly different.

On Monday, 28 March 2016 at 11:29:55 UTC, Vladimir Panteleev wrote:
> On Monday, 28 March 2016 at 01:30:34 UTC, Seb wrote:
>> AFAIK dmd supports the `-cov` flag since quite some time, so I would be interested why there's no code coverage bot. There also seems to exist at least two external platforms that allow such code coverage analyzing [1, 2]. Is this already on your roadmap?
>>
>> [1] https://github.com/codecov/example-d
>> [2] https://github.com/ColdenCullen/doveralls/issues/16
>>
>> Anyways it seems pretty easy to hack something quick&dirty together that could be pretty helpful:
>
> Yep. The difficulties are:
>
> 1. Getting data off the autotester. Currently it's nearly impossible, as is making any changes to it.
>
> https://issues.dlang.org/show_bug.cgi?id=14381
>
> 2. Collating data from multiple platforms, as there is a lot of platform-specific code in e.g. std.stdio, std.file, and most of Druntime.

Okay I see that for the long run we need a better way to handle the testing infrastructure :/

Can't we focus for the near future on getting a simple Phobos coverage tester that just compares the coverage of all files changes like your DAutoTest?
With this approach we can still prevent regression and show improvements.