Thread overview
[Issue 18619] documented unittests extracted by test_extractor should be run by auto-tester
Mar 15, 2018
Timothee Cour
Mar 15, 2018
greenify
Mar 15, 2018
Timothee Cour
Mar 15, 2018
greenify
March 15, 2018
https://issues.dlang.org/show_bug.cgi?id=18619

Timothee Cour <timothee.cour2@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |timothee.cour2@gmail.com
                 OS|Mac OS X                    |All

--
March 15, 2018
https://issues.dlang.org/show_bug.cgi?id=18619

greenify <greeenify@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |greeenify@gmail.com
         Resolution|---                         |WONTFIX

--- Comment #1 from greenify <greeenify@gmail.com> ---
I think you misunderstood me. They are already run on CircleCi and there is
almost no value in running them on auto-tester as they are already run there,
just not extracted. So the problems only amount to accessibility.
Also there are severe disadvantages about the auto-tester: it's a horrible pain
to get stuff running there + the pipeline currently takes ~3 minutes which
would be added to all ten host runs.

--
March 15, 2018
https://issues.dlang.org/show_bug.cgi?id=18619

--- Comment #2 from Timothee Cour <timothee.cour2@gmail.com> ---
so if i understand correctly circleci runs the (unextracted) unittests blocks (regardless of documented or not) and at the moment nothing (except end user on dlang.org) runs the extracted ones; the things that can go wrong in 2nd step are cases of documented unittest relying on private symbols, ie accessibility issues.

These should be caught in code reviews as they're not caught automatically.

> Also there are severe disadvantages about the auto-tester: it's a horrible pain to get stuff running there + the pipeline currently takes ~3 minutes which would be added to all ten host runs.

ok, I understand the tradeoff, that's a good explanation for closing it, thanks!

--
March 15, 2018
https://issues.dlang.org/show_bug.cgi?id=18619

--- Comment #3 from greenify <greeenify@gmail.com> ---
(In reply to Timothee Cour from comment #2)
> so if i understand correctly circleci runs the (unextracted) unittests blocks (regardless of documented or not) and at the moment nothing (except end user on dlang.org) runs the extracted ones; the things that can go wrong in 2nd step are cases of documented unittest relying on private symbols, ie accessibility issues.
> 
> These should be caught in code reviews as they're not caught automatically.

CircleCi runs `make -f publictest -j4`

This is the extraction and run of the public examples.
The normal testsuite with all unittests is run in all CIs is run on auto-tester
and CircleCi - the latter is needed for reporting the coverage to CodeCov.

There might be bugs in the extraction step, but apart from that it's almost guaranteed that the examples are runnable via run.dlang.io on dlang.org

--