On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe wrote:
> So true. Then I will do PR's first.

I finally got around implementing running dmd/druntime/phobos pull requests against all dub packages. Thank you digger, for making it so easy.

I did a run of master + druntime#1602. Then I ran a comparison with a batch from 2.071.2-b2. 108 packages had a different build result. I have no nice stats or pictures, but a quick glance over the raw data:

50 of them went from green unittests to a dmd exit code 1.
16 went from unknown build results to dmd exit code 1.
10 went from dmd exit code 255 to 1
9 of them are now green.
8 of them went from linker errors to a dmd exit code 1.
6 of then went from non-zero exit code during unittest run to a dmd exit code 1.
3 previous ran out of memory but now resulted in dmd exit code 1.
etc.

All in all I think +/- 96 package are affected. A little over 11%.

On Monday, 22 August 2016 at 20:44:05 UTC, Sebastiaan Koppe wrote:
> On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe wrote:
>> So true. Then I will do PR's first.
>
> I finally got around implementing running dmd/druntime/phobos pull requests against all dub packages. Thank you digger, for making it so easy.
>
> I did a run of master + druntime#1602. Then I ran a comparison with a batch from 2.071.2-b2. 108 packages had a different build result. I have no nice stats or pictures, but a quick glance over the raw data:
>
> 50 of them went from green unittests to a dmd exit code 1.
> 16 went from unknown build results to dmd exit code 1.
> 10 went from dmd exit code 255 to 1
> 9 of them are now green.
> 8 of them went from linker errors to a dmd exit code 1.
> 6 of then went from non-zero exit code during unittest run to a dmd exit code 1.
> 3 previous ran out of memory but now resulted in dmd exit code 1.
> etc.
>
> All in all I think +/- 96 package are affected. A little over 11%.

That's awesome to know!

How difficult would it be to integrate it with the dlang GitHub PR workflow?
I am just shooting an idea that popped into my head:

We already use CircleCi and Travis for the dlang repos, so if we lock the packages to a fixed version (to prevent failures caused by the package authors), we might be able create a simple file like:

```
vibe.d==0.7.29
mir==0.16.3
...
```

We could select a subset (e.g. 50-100), s.t. the runtime doesn't get exorbitant.
Following we could then do enable the checking in CircleCi with sth. similar to:

```
wget https://raw.githubusercontent.com/dlang/community-list/master/dlang-packages.master # just an example, idea is to host the file somewhere on GitHub s.t. editing it is easy
dub fetch your-fancy-tool --version="x.y.y"
dub run your-fancy-tool --config dlang-stable.packages
```

Of course CircleCi doesn't have the access rights to post back to the hook API, but you could send a notification to dlang-bot [1] which has the permissions or let the CI error / fail.

Otherwise you could of course look into setting up your own job queue (or hack with the code from the auto-tester [2]), which might be fun too.

[1] https://github.com/MartinNowak/dlang-bot
[2] https://github.com/braddr/d-tester

On Friday, 26 August 2016 at 18:52:17 UTC, Seb wrote:
> That's awesome to know!
>
> How difficult would it be to integrate it with the dlang GitHub PR workflow?

Not at all. Just need an api key from someone with administration access. But let's not get ahead of ourselves. Right now I am just planning to contact the github api.

Still, there is some work to be done first for purging and updating the job queue when prs are updated and probably some other cases.

Also there are some choices left regarding the interpretation of the results. Right now, for pull requests I do a diff with the latest dmd release and collect all the packages that have a different outcome. It would be better to run the comparison against the commit the pull request was based on, although that requires building twice as much.

Currently I am focused on a simple frontend to give people a view into the results. It is coming along quite nicely.

> I am just shooting an idea that popped into my head:
>
> We already use CircleCi and Travis for the dlang repos, so if we lock the packages to a fixed version (to prevent failures caused by the package authors), we might be able create a simple file like:
>
> ```
> vibe.d==0.7.29
> mir==0.16.3
> ...
> ```
>
> We could select a subset (e.g. 50-100), s.t. the runtime doesn't get exorbitant.
> Following we could then do enable the checking in CircleCi with sth. similar to:
>
> ```
> wget https://raw.githubusercontent.com/dlang/community-list/master/dlang-packages.master # just an example, idea is to host the file somewhere on GitHub s.t. editing it is easy
> dub fetch your-fancy-tool --version="x.y.y"
> dub run your-fancy-tool --config dlang-stable.packages
> ```
>
> Of course CircleCi doesn't have the access rights to post back to the hook API, but you could send a notification to dlang-bot [1] which has the permissions or let the CI error / fail.
>
> Otherwise you could of course look into setting up your own job queue (or hack with the code from the auto-tester [2]), which might be fun.
I already have my own queue. The important part though is a place to keep the results and running queries against it. Currently I use various regexes against dub test's output and categorise accordingly.

What I am really happy about is the aggressive caching. It allows me to build and unit tests a package in 10sec on average.