View mode: basic / threaded / horizontal-split · Log in · Help
March 18, 2013
Raising the bar on Phobos unittest coverage
With the new -cov=nnn (not pulled yet) and -main, we can now add to the build 
process a minimum bar for unit test coverage.

The current state looks like this, the percentages are the current coverage 
amounts. It's not bad, but there's a lot of low hanging fruit ready for pull 
requests!

From now on, I'd like such unittest coverage (and justification for low 
coverage) to be part of the minimum standard for all new phobos modules.

        $(DMD) -cov=83 -unittest -main -run std\stdio.d
        $(DMD) -cov=100 -unittest -main -run std\stdiobase.d
        $(DMD) -cov=95 -unittest -main -run std\string.d
        $(DMD) -cov=71 -unittest -main -run std\format.d
        $(DMD) -cov=83 -unittest -main -run std\file.d
        $(DMD) -cov=86 -unittest -main -run std\range.d
        $(DMD) -cov=95 -unittest -main -run std\array.d
        $(DMD) -cov=100 -unittest -main -run std\functional.d
        $(DMD) -cov=96 -unittest -main -run std\path.d
        $(DMD) -cov=41 -unittest -main -run std\outbuffer.d
        $(DMD) -cov=89 -unittest -main -run std\utf.d
        $(DMD) -cov=93 -unittest -main -run std\csv.d
        $(DMD) -cov=91 -unittest -main -run std\math.d
        $(DMD) -cov=95 -unittest -main -run std\complex.d
        $(DMD) -cov=70 -unittest -main -run std\numeric.d
        $(DMD) -cov=94 -unittest -main -run std\bigint.d
        $(DMD) -cov=100 -unittest -main -run std\metastrings.d
        $(DMD) -cov=95 -unittest -main -run std\bitmanip.d
        $(DMD) -cov=82 -unittest -main -run std\typecons.d
        $(DMD) -cov=44 -unittest -main -run std\uni.d
        $(DMD) -cov=91 -unittest -main -run std\base64.d
        $(DMD) -cov=99 -unittest -main -run std\md5.d
        $(DMD) -cov=0  -unittest -main -run std\ctype.d
        $(DMD) -cov=100 -unittest -main -run std\ascii.d
        $(DMD) -cov=0  -unittest -main -run std\demangle.d
        $(DMD) -cov=57 -unittest -main -run std\uri.d
        $(DMD) -cov=51 -unittest -main -run std\mmfile.d
        $(DMD) -cov=95 -unittest -main -run std\getopt.d
        $(DMD) -cov=92 -unittest -main -run std\signals.d
        $(DMD) -cov=100 -unittest -main -run std\typetuple.d
        $(DMD) -cov=85 -unittest -main -run std\traits.d
        $(DMD) -cov=62 -unittest -main -run std\encoding.d
        $(DMD) -cov=61 -unittest -main -run std\xml.d
        $(DMD) -cov=79 -unittest -main -run std\random.d
        $(DMD) -cov=92 -unittest -main -d -run std\exception.d
        $(DMD) -cov=73 -unittest -main -run std\concurrency.d
        $(DMD) -cov=95 -unittest -main -run std\datetime.d
        $(DMD) -cov=96 -unittest -main -run std\uuid.d
        $(DMD) -cov=100 -unittest -main -run std\digest\crc.d
        $(DMD) -cov=55 -unittest -main -run std\digest\sha.d
        $(DMD) -cov=100 -unittest -main -run std\digest\md.d
        $(DMD) -cov=100 -unittest -main -run std\digest\ripemd.d
        $(DMD) -cov=75 -unittest -main -run std\digest\digest.d
        $(DMD) -cov=95 -unittest -main -run std\algorithm.d
        $(DMD) -cov=83 -unittest -main -run std\variant.d
        $(DMD) -cov=0  -unittest -main -run std\syserror.d
        $(DMD) -cov=58 -unittest -main -run std\zlib.d
        $(DMD) -cov=54 -unittest -main -run std\stream.d
        $(DMD) -cov=53 -unittest -main -run std\socket.d
        $(DMD) -cov=0  -unittest -main -run std\socketstream.d
        $(DMD) -cov=88 -unittest -main -run std\container.d
        $(DMD) -cov=90 -unittest -main -d -run std\conv.d
        $(DMD) -cov=0  -unittest -main -run std\zip.d
        $(DMD) -cov=92 -unittest -main -run std\cstream.d
        $(DMD) -cov=77 -unittest -main -run std\regex.d
        $(DMD) -cov=92 -unittest -main -run std\json.d
        $(DMD) -cov=87 -unittest -main -run std\parallelism.d
        $(DMD) -cov=50 -unittest -main -run std\mathspecial.d
        $(DMD) -cov=71 -unittest -main -run std\process.d
        $(DMD) -cov=0  -unittest -main -run crc32.d
        $(DMD) -cov=70 -unittest -main -run std\net\isemail.d
        $(DMD) -cov=2  -unittest -main -run std\net\curl.d
        $(DMD) -cov=60 -unittest -main -run std\windows\registry.d
        $(DMD) -cov=55 -unittest -main -run std\internal\uni.d
        $(DMD) -cov=0  -unittest -main -run std\internal\uni_tab.d
        $(DMD) -cov=0  -unittest -main -run std\internal\digest\sha_SSSE3.d
        $(DMD) -cov=50 -unittest -main -run std\internal\math\biguintcore.d
        $(DMD) -cov=75 -unittest -main -run std\internal\math\biguintnoasm.d
#       $(DMD) -cov -unittest -main -run std\internal\math\biguintx86.d
        $(DMD) -cov=94 -unittest -main -run std\internal\math\gammafunction.d
        $(DMD) -cov=92 -unittest -main -run std\internal\math\errorfunction.d
        $(DMD) -cov=31 -unittest -main -run std\internal\windows\advapi32.d
        $(DMD) -cov=58 -unittest -main -run etc\c\zlib.d
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On 2013-03-18 02:00, Walter Bright wrote:
> With the new -cov=nnn (not pulled yet) and -main, we can now add to the
> build process a minimum bar for unit test coverage.

What are these switches for?

-- 
/Jacob Carlborg
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On Monday, 18 March 2013 at 09:48:11 UTC, Jacob Carlborg wrote:
> On 2013-03-18 02:00, Walter Bright wrote:
>> With the new -cov=nnn (not pulled yet) and -main, we can now 
>> add to the
>> build process a minimum bar for unit test coverage.
>
> What are these switches for?

Executing the program fails (non-zero exit code) if the coverage 
is less than the specified percentage.

David
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On 2013-03-18 11:09, David Nadlinger wrote:

> Executing the program fails (non-zero exit code) if the coverage is less
> than the specified percentage.

Aha, I see. How do you get the percentage, with the standard -cov switch?

-- 
/Jacob Carlborg
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
Jacob Carlborg:

> How do you get the percentage, with the standard -cov switch?

If you use -cov it writes the coverage percentage at the end.

Bye,
bearophile
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On 03/18/2013 02:00 AM, Walter Bright wrote:
>
>  From now on, I'd like such unittest coverage (and justification for low
> coverage) to be part of the minimum standard for all new phobos modules.

Great.

> $(DMD) -cov=83 -unittest -main -run std\file.d

Can someone look into std.file's unittests. They use 60% (32s/50s) of 
the unittest RUNtime on my machine.
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
> On 03/18/2013 02:00 AM, Walter Bright wrote:
> > From now on, I'd like such unittest coverage (and justification for low
> > 
> > coverage) to be part of the minimum standard for all new phobos modules.
> 
> Great.
> 
> > $(DMD) -cov=83 -unittest -main -run std\file.d
> 
> Can someone look into std.file's unittests. They use 60% (32s/50s) of
> the unittest RUNtime on my machine.

My first guess would be that you're running a different OS from Walter and that 
OS-specific code counts as not being run when you run it on a different OS. If 
that's the case, std.file is doomed to have lower code coverage because of the 
larger-than-average percentage of it which is OS-specific. The coverage should 
still be looked into and improved where possible, but Linux code is clearly 
never going to run on Windows or vice versa.

- Jonathan M Davis
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On 3/18/2013 11:12 AM, Jonathan M Davis wrote:
> On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
>> Can someone look into std.file's unittests. They use 60% (32s/50s) of
>> the unittest RUNtime on my machine.
>
> My first guess would be that you're running a different OS from Walter and that
> OS-specific code counts as not being run when you run it on a different OS.

Code that is statically compiled out is not counted as executable code by the 
coverage analyzer.

As for looking into it, examine the report generated - std-file.lst - it'll tell 
you which lines were executed and which were not.
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
On Monday, March 18, 2013 12:09:47 Walter Bright wrote:
> On 3/18/2013 11:12 AM, Jonathan M Davis wrote:
> > On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
> >> Can someone look into std.file's unittests. They use 60% (32s/50s) of
> >> the unittest RUNtime on my machine.
> > 
> > My first guess would be that you're running a different OS from Walter and
> > that OS-specific code counts as not being run when you run it on a
> > different OS.
> Code that is statically compiled out is not counted as executable code by
> the coverage analyzer.

Then it was a bad guess. Good to know (and better behavior that way really).

> As for looking into it, examine the report generated - std-file.lst - it'll
> tell you which lines were executed and which were not.

Yeah, looking into it should be straigtforward. It just means taking the time 
to do it.

- Jonathan M Davis
March 18, 2013
Re: Raising the bar on Phobos unittest coverage
Please note that std.net.curl needs to have PHOBOS_TEST_ALLOW_NET 
env variable defined or else it will early out on most unittests. 
I guess that will increase the coverage from 2 to something more 
sane.

-Jonas

>         $(DMD) -cov=2  -unittest -main -run std\net\curl.d
« First   ‹ Prev
1 2 3 4
Top | Discussion index | About this forum | D home