Jump to page: 1 24  
Page
Thread overview
Raising the bar on Phobos unittest coverage
Mar 18, 2013
Walter Bright
Mar 18, 2013
Jacob Carlborg
Mar 18, 2013
David Nadlinger
Mar 18, 2013
Jacob Carlborg
Mar 18, 2013
bearophile
Mar 18, 2013
Martin Nowak
Mar 18, 2013
Jonathan M Davis
Mar 18, 2013
Walter Bright
Mar 18, 2013
Jonathan M Davis
Mar 18, 2013
monarch_dodra
Mar 18, 2013
Jonathan M Davis
Mar 18, 2013
Martin Nowak
Mar 18, 2013
Jonathan M Davis
Mar 18, 2013
Jonas Drewsen
Mar 18, 2013
Walter Bright
Mar 18, 2013
Jonas Drewsen
Mar 18, 2013
Jonathan M Davis
Mar 18, 2013
Walter Bright
Mar 19, 2013
Brad Roberts
Mar 19, 2013
Walter Bright
Jun 20, 2013
Jonathan M Davis
Jun 21, 2013
Jacob Carlborg
Jun 21, 2013
Jacob Carlborg
Jun 21, 2013
Walter Bright
Jun 22, 2013
Jacob Carlborg
Jun 22, 2013
Jacob Carlborg
Jun 20, 2013
Jonathan M Davis
Jun 21, 2013
qznc
Jun 21, 2013
Jacob Carlborg
Jun 21, 2013
Vladimir Panteleev
March 18, 2013
With the new -cov=nnn (not pulled yet) and -main, we can now add to the build process a minimum bar for unit test coverage.

The current state looks like this, the percentages are the current coverage amounts. It's not bad, but there's a lot of low hanging fruit ready for pull requests!

From now on, I'd like such unittest coverage (and justification for low coverage) to be part of the minimum standard for all new phobos modules.

        $(DMD) -cov=83 -unittest -main -run std\stdio.d
        $(DMD) -cov=100 -unittest -main -run std\stdiobase.d
        $(DMD) -cov=95 -unittest -main -run std\string.d
        $(DMD) -cov=71 -unittest -main -run std\format.d
        $(DMD) -cov=83 -unittest -main -run std\file.d
        $(DMD) -cov=86 -unittest -main -run std\range.d
        $(DMD) -cov=95 -unittest -main -run std\array.d
        $(DMD) -cov=100 -unittest -main -run std\functional.d
        $(DMD) -cov=96 -unittest -main -run std\path.d
        $(DMD) -cov=41 -unittest -main -run std\outbuffer.d
        $(DMD) -cov=89 -unittest -main -run std\utf.d
        $(DMD) -cov=93 -unittest -main -run std\csv.d
        $(DMD) -cov=91 -unittest -main -run std\math.d
        $(DMD) -cov=95 -unittest -main -run std\complex.d
        $(DMD) -cov=70 -unittest -main -run std\numeric.d
        $(DMD) -cov=94 -unittest -main -run std\bigint.d
        $(DMD) -cov=100 -unittest -main -run std\metastrings.d
        $(DMD) -cov=95 -unittest -main -run std\bitmanip.d
        $(DMD) -cov=82 -unittest -main -run std\typecons.d
        $(DMD) -cov=44 -unittest -main -run std\uni.d
        $(DMD) -cov=91 -unittest -main -run std\base64.d
        $(DMD) -cov=99 -unittest -main -run std\md5.d
        $(DMD) -cov=0  -unittest -main -run std\ctype.d
        $(DMD) -cov=100 -unittest -main -run std\ascii.d
        $(DMD) -cov=0  -unittest -main -run std\demangle.d
        $(DMD) -cov=57 -unittest -main -run std\uri.d
        $(DMD) -cov=51 -unittest -main -run std\mmfile.d
        $(DMD) -cov=95 -unittest -main -run std\getopt.d
        $(DMD) -cov=92 -unittest -main -run std\signals.d
        $(DMD) -cov=100 -unittest -main -run std\typetuple.d
        $(DMD) -cov=85 -unittest -main -run std\traits.d
        $(DMD) -cov=62 -unittest -main -run std\encoding.d
        $(DMD) -cov=61 -unittest -main -run std\xml.d
        $(DMD) -cov=79 -unittest -main -run std\random.d
        $(DMD) -cov=92 -unittest -main -d -run std\exception.d
        $(DMD) -cov=73 -unittest -main -run std\concurrency.d
        $(DMD) -cov=95 -unittest -main -run std\datetime.d
        $(DMD) -cov=96 -unittest -main -run std\uuid.d
        $(DMD) -cov=100 -unittest -main -run std\digest\crc.d
        $(DMD) -cov=55 -unittest -main -run std\digest\sha.d
        $(DMD) -cov=100 -unittest -main -run std\digest\md.d
        $(DMD) -cov=100 -unittest -main -run std\digest\ripemd.d
        $(DMD) -cov=75 -unittest -main -run std\digest\digest.d
        $(DMD) -cov=95 -unittest -main -run std\algorithm.d
        $(DMD) -cov=83 -unittest -main -run std\variant.d
        $(DMD) -cov=0  -unittest -main -run std\syserror.d
        $(DMD) -cov=58 -unittest -main -run std\zlib.d
        $(DMD) -cov=54 -unittest -main -run std\stream.d
        $(DMD) -cov=53 -unittest -main -run std\socket.d
        $(DMD) -cov=0  -unittest -main -run std\socketstream.d
        $(DMD) -cov=88 -unittest -main -run std\container.d
        $(DMD) -cov=90 -unittest -main -d -run std\conv.d
        $(DMD) -cov=0  -unittest -main -run std\zip.d
        $(DMD) -cov=92 -unittest -main -run std\cstream.d
        $(DMD) -cov=77 -unittest -main -run std\regex.d
        $(DMD) -cov=92 -unittest -main -run std\json.d
        $(DMD) -cov=87 -unittest -main -run std\parallelism.d
        $(DMD) -cov=50 -unittest -main -run std\mathspecial.d
        $(DMD) -cov=71 -unittest -main -run std\process.d
        $(DMD) -cov=0  -unittest -main -run crc32.d
        $(DMD) -cov=70 -unittest -main -run std\net\isemail.d
        $(DMD) -cov=2  -unittest -main -run std\net\curl.d
        $(DMD) -cov=60 -unittest -main -run std\windows\registry.d
        $(DMD) -cov=55 -unittest -main -run std\internal\uni.d
        $(DMD) -cov=0  -unittest -main -run std\internal\uni_tab.d
        $(DMD) -cov=0  -unittest -main -run std\internal\digest\sha_SSSE3.d
        $(DMD) -cov=50 -unittest -main -run std\internal\math\biguintcore.d
        $(DMD) -cov=75 -unittest -main -run std\internal\math\biguintnoasm.d
#       $(DMD) -cov -unittest -main -run std\internal\math\biguintx86.d
        $(DMD) -cov=94 -unittest -main -run std\internal\math\gammafunction.d
        $(DMD) -cov=92 -unittest -main -run std\internal\math\errorfunction.d
        $(DMD) -cov=31 -unittest -main -run std\internal\windows\advapi32.d
        $(DMD) -cov=58 -unittest -main -run etc\c\zlib.d
March 18, 2013
On 2013-03-18 02:00, Walter Bright wrote:
> With the new -cov=nnn (not pulled yet) and -main, we can now add to the
> build process a minimum bar for unit test coverage.

What are these switches for?

-- 
/Jacob Carlborg
March 18, 2013
On Monday, 18 March 2013 at 09:48:11 UTC, Jacob Carlborg wrote:
> On 2013-03-18 02:00, Walter Bright wrote:
>> With the new -cov=nnn (not pulled yet) and -main, we can now add to the
>> build process a minimum bar for unit test coverage.
>
> What are these switches for?

Executing the program fails (non-zero exit code) if the coverage is less than the specified percentage.

David
March 18, 2013
On 2013-03-18 11:09, David Nadlinger wrote:

> Executing the program fails (non-zero exit code) if the coverage is less
> than the specified percentage.

Aha, I see. How do you get the percentage, with the standard -cov switch?

-- 
/Jacob Carlborg
March 18, 2013
Jacob Carlborg:

> How do you get the percentage, with the standard -cov switch?

If you use -cov it writes the coverage percentage at the end.

Bye,
bearophile
March 18, 2013
On 03/18/2013 02:00 AM, Walter Bright wrote:
>
>  From now on, I'd like such unittest coverage (and justification for low
> coverage) to be part of the minimum standard for all new phobos modules.

Great.

> $(DMD) -cov=83 -unittest -main -run std\file.d

Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine.

March 18, 2013
On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
> On 03/18/2013 02:00 AM, Walter Bright wrote:
> > From now on, I'd like such unittest coverage (and justification for low
> > 
> > coverage) to be part of the minimum standard for all new phobos modules.
> 
> Great.
> 
> > $(DMD) -cov=83 -unittest -main -run std\file.d
> 
> Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine.

My first guess would be that you're running a different OS from Walter and that OS-specific code counts as not being run when you run it on a different OS. If that's the case, std.file is doomed to have lower code coverage because of the larger-than-average percentage of it which is OS-specific. The coverage should still be looked into and improved where possible, but Linux code is clearly never going to run on Windows or vice versa.

- Jonathan M Davis
March 18, 2013
On 3/18/2013 11:12 AM, Jonathan M Davis wrote:
> On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
>> Can someone look into std.file's unittests. They use 60% (32s/50s) of
>> the unittest RUNtime on my machine.
>
> My first guess would be that you're running a different OS from Walter and that
> OS-specific code counts as not being run when you run it on a different OS.

Code that is statically compiled out is not counted as executable code by the coverage analyzer.

As for looking into it, examine the report generated - std-file.lst - it'll tell you which lines were executed and which were not.
March 18, 2013
On Monday, March 18, 2013 12:09:47 Walter Bright wrote:
> On 3/18/2013 11:12 AM, Jonathan M Davis wrote:
> > On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
> >> Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine.
> > 
> > My first guess would be that you're running a different OS from Walter and that OS-specific code counts as not being run when you run it on a different OS.
> Code that is statically compiled out is not counted as executable code by the coverage analyzer.

Then it was a bad guess. Good to know (and better behavior that way really).

> As for looking into it, examine the report generated - std-file.lst - it'll tell you which lines were executed and which were not.

Yeah, looking into it should be straigtforward. It just means taking the time to do it.

- Jonathan M Davis
March 18, 2013
Please note that std.net.curl needs to have PHOBOS_TEST_ALLOW_NET env variable defined or else it will early out on most unittests. I guess that will increase the coverage from 2 to something more sane.

-Jonas

>         $(DMD) -cov=2  -unittest -main -run std\net\curl.d
« First   ‹ Prev
1 2 3 4