March 18, 2013 Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
With the new -cov=nnn (not pulled yet) and -main, we can now add to the build process a minimum bar for unit test coverage. The current state looks like this, the percentages are the current coverage amounts. It's not bad, but there's a lot of low hanging fruit ready for pull requests! From now on, I'd like such unittest coverage (and justification for low coverage) to be part of the minimum standard for all new phobos modules. $(DMD) -cov=83 -unittest -main -run std\stdio.d $(DMD) -cov=100 -unittest -main -run std\stdiobase.d $(DMD) -cov=95 -unittest -main -run std\string.d $(DMD) -cov=71 -unittest -main -run std\format.d $(DMD) -cov=83 -unittest -main -run std\file.d $(DMD) -cov=86 -unittest -main -run std\range.d $(DMD) -cov=95 -unittest -main -run std\array.d $(DMD) -cov=100 -unittest -main -run std\functional.d $(DMD) -cov=96 -unittest -main -run std\path.d $(DMD) -cov=41 -unittest -main -run std\outbuffer.d $(DMD) -cov=89 -unittest -main -run std\utf.d $(DMD) -cov=93 -unittest -main -run std\csv.d $(DMD) -cov=91 -unittest -main -run std\math.d $(DMD) -cov=95 -unittest -main -run std\complex.d $(DMD) -cov=70 -unittest -main -run std\numeric.d $(DMD) -cov=94 -unittest -main -run std\bigint.d $(DMD) -cov=100 -unittest -main -run std\metastrings.d $(DMD) -cov=95 -unittest -main -run std\bitmanip.d $(DMD) -cov=82 -unittest -main -run std\typecons.d $(DMD) -cov=44 -unittest -main -run std\uni.d $(DMD) -cov=91 -unittest -main -run std\base64.d $(DMD) -cov=99 -unittest -main -run std\md5.d $(DMD) -cov=0 -unittest -main -run std\ctype.d $(DMD) -cov=100 -unittest -main -run std\ascii.d $(DMD) -cov=0 -unittest -main -run std\demangle.d $(DMD) -cov=57 -unittest -main -run std\uri.d $(DMD) -cov=51 -unittest -main -run std\mmfile.d $(DMD) -cov=95 -unittest -main -run std\getopt.d $(DMD) -cov=92 -unittest -main -run std\signals.d $(DMD) -cov=100 -unittest -main -run std\typetuple.d $(DMD) -cov=85 -unittest -main -run std\traits.d $(DMD) -cov=62 -unittest -main -run std\encoding.d $(DMD) -cov=61 -unittest -main -run std\xml.d $(DMD) -cov=79 -unittest -main -run std\random.d $(DMD) -cov=92 -unittest -main -d -run std\exception.d $(DMD) -cov=73 -unittest -main -run std\concurrency.d $(DMD) -cov=95 -unittest -main -run std\datetime.d $(DMD) -cov=96 -unittest -main -run std\uuid.d $(DMD) -cov=100 -unittest -main -run std\digest\crc.d $(DMD) -cov=55 -unittest -main -run std\digest\sha.d $(DMD) -cov=100 -unittest -main -run std\digest\md.d $(DMD) -cov=100 -unittest -main -run std\digest\ripemd.d $(DMD) -cov=75 -unittest -main -run std\digest\digest.d $(DMD) -cov=95 -unittest -main -run std\algorithm.d $(DMD) -cov=83 -unittest -main -run std\variant.d $(DMD) -cov=0 -unittest -main -run std\syserror.d $(DMD) -cov=58 -unittest -main -run std\zlib.d $(DMD) -cov=54 -unittest -main -run std\stream.d $(DMD) -cov=53 -unittest -main -run std\socket.d $(DMD) -cov=0 -unittest -main -run std\socketstream.d $(DMD) -cov=88 -unittest -main -run std\container.d $(DMD) -cov=90 -unittest -main -d -run std\conv.d $(DMD) -cov=0 -unittest -main -run std\zip.d $(DMD) -cov=92 -unittest -main -run std\cstream.d $(DMD) -cov=77 -unittest -main -run std\regex.d $(DMD) -cov=92 -unittest -main -run std\json.d $(DMD) -cov=87 -unittest -main -run std\parallelism.d $(DMD) -cov=50 -unittest -main -run std\mathspecial.d $(DMD) -cov=71 -unittest -main -run std\process.d $(DMD) -cov=0 -unittest -main -run crc32.d $(DMD) -cov=70 -unittest -main -run std\net\isemail.d $(DMD) -cov=2 -unittest -main -run std\net\curl.d $(DMD) -cov=60 -unittest -main -run std\windows\registry.d $(DMD) -cov=55 -unittest -main -run std\internal\uni.d $(DMD) -cov=0 -unittest -main -run std\internal\uni_tab.d $(DMD) -cov=0 -unittest -main -run std\internal\digest\sha_SSSE3.d $(DMD) -cov=50 -unittest -main -run std\internal\math\biguintcore.d $(DMD) -cov=75 -unittest -main -run std\internal\math\biguintnoasm.d # $(DMD) -cov -unittest -main -run std\internal\math\biguintx86.d $(DMD) -cov=94 -unittest -main -run std\internal\math\gammafunction.d $(DMD) -cov=92 -unittest -main -run std\internal\math\errorfunction.d $(DMD) -cov=31 -unittest -main -run std\internal\windows\advapi32.d $(DMD) -cov=58 -unittest -main -run etc\c\zlib.d |
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On 2013-03-18 02:00, Walter Bright wrote: > With the new -cov=nnn (not pulled yet) and -main, we can now add to the > build process a minimum bar for unit test coverage. What are these switches for? -- /Jacob Carlborg |
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | On Monday, 18 March 2013 at 09:48:11 UTC, Jacob Carlborg wrote:
> On 2013-03-18 02:00, Walter Bright wrote:
>> With the new -cov=nnn (not pulled yet) and -main, we can now add to the
>> build process a minimum bar for unit test coverage.
>
> What are these switches for?
Executing the program fails (non-zero exit code) if the coverage is less than the specified percentage.
David
|
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to David Nadlinger | On 2013-03-18 11:09, David Nadlinger wrote: > Executing the program fails (non-zero exit code) if the coverage is less > than the specified percentage. Aha, I see. How do you get the percentage, with the standard -cov switch? -- /Jacob Carlborg |
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | Jacob Carlborg:
> How do you get the percentage, with the standard -cov switch?
If you use -cov it writes the coverage percentage at the end.
Bye,
bearophile
|
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On 03/18/2013 02:00 AM, Walter Bright wrote: > > From now on, I'd like such unittest coverage (and justification for low > coverage) to be part of the minimum standard for all new phobos modules. Great. > $(DMD) -cov=83 -unittest -main -run std\file.d Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine. |
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Martin Nowak | On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
> On 03/18/2013 02:00 AM, Walter Bright wrote:
> > From now on, I'd like such unittest coverage (and justification for low
> >
> > coverage) to be part of the minimum standard for all new phobos modules.
>
> Great.
>
> > $(DMD) -cov=83 -unittest -main -run std\file.d
>
> Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine.
My first guess would be that you're running a different OS from Walter and that OS-specific code counts as not being run when you run it on a different OS. If that's the case, std.file is doomed to have lower code coverage because of the larger-than-average percentage of it which is OS-specific. The coverage should still be looked into and improved where possible, but Linux code is clearly never going to run on Windows or vice versa.
- Jonathan M Davis
|
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jonathan M Davis | On 3/18/2013 11:12 AM, Jonathan M Davis wrote:
> On Monday, March 18, 2013 17:34:12 Martin Nowak wrote:
>> Can someone look into std.file's unittests. They use 60% (32s/50s) of
>> the unittest RUNtime on my machine.
>
> My first guess would be that you're running a different OS from Walter and that
> OS-specific code counts as not being run when you run it on a different OS.
Code that is statically compiled out is not counted as executable code by the coverage analyzer.
As for looking into it, examine the report generated - std-file.lst - it'll tell you which lines were executed and which were not.
|
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On Monday, March 18, 2013 12:09:47 Walter Bright wrote: > On 3/18/2013 11:12 AM, Jonathan M Davis wrote: > > On Monday, March 18, 2013 17:34:12 Martin Nowak wrote: > >> Can someone look into std.file's unittests. They use 60% (32s/50s) of the unittest RUNtime on my machine. > > > > My first guess would be that you're running a different OS from Walter and that OS-specific code counts as not being run when you run it on a different OS. > Code that is statically compiled out is not counted as executable code by the coverage analyzer. Then it was a bad guess. Good to know (and better behavior that way really). > As for looking into it, examine the report generated - std-file.lst - it'll tell you which lines were executed and which were not. Yeah, looking into it should be straigtforward. It just means taking the time to do it. - Jonathan M Davis |
March 18, 2013 Re: Raising the bar on Phobos unittest coverage | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | Please note that std.net.curl needs to have PHOBOS_TEST_ALLOW_NET env variable defined or else it will early out on most unittests. I guess that will increase the coverage from 2 to something more sane.
-Jonas
> $(DMD) -cov=2 -unittest -main -run std\net\curl.d
|
Copyright © 1999-2021 by the D Language Foundation