On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe wrote:
> I have just finished a first iteration of dubster, a test runner that runs `dub test` on each package for each dmd release.
>
> see https://github.com/skoppe/dubster
>
> Please provide feedback as it will determine the direction/life of this tester.
>
> I am planning on adding a web ui/api next to look around in the data.

That are excellent news!
Some random ideas:

1) Send the packages a notification about the build error (e.g. Github comment) - this should probably be tweaked a bit, s.t. it doesn't spam too often for still broken packages
2) Allow easy, manual build of special branches for the core team, e.g. let's say Walter develops the new scoped pointers feature ( https://github.com/dlang/DIPs/pull/24), than it would be great to know how many packages would break by pulling in the branch (in comparison to the last release or current nightly). A similar "breakage by shipping" test might be very interesting for critical changes to druntime or phobos too.
3) Once you have the API
a) (try to) get a shield badge (-> http://shields.io/)
b) Make the data available to the dub-registry (-> https://github.com/dlang/dub-registry)
4) Assess the quality of the unittests. Probably the easiest is `dub test -b unittest-cov`, and then summing up the total coverage of all generated .lst files, but running with coverage might increase your build times, though I would argue that it's worth it ;-)
5) Log your daily "broken" statistics - could be a good indicator of whether your hard work gets acknowledged.
6) Regarding linker errors - I can only redirect you to the open DUB issue (https://github.com/dlang/dub/issues/852) and DEP 5 (https://github.com/dlang/dub/wiki/DEP5).

On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe wrote:
> - code.dlang.org has an api but doesn't provide an endpoint to retrieve all packages/version. Now I just scrape the site instead (thanks Adam for your dom implementation).

No endpoint but still possible in two steps. For example test this script:

°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
import std.stdio;
import std.net.curl;
import std.json;
import std.format;

void main()
{
    auto allRaw = get(`https://code.dlang.org/packages/index.json`);
    auto allJson = parseJSON(allRaw);
    enum latestFmtSpec = `https://code.dlang.org/api/packages/%s/latest`;
    foreach(p; allJson.array)
    {
        auto ver = get(latestFmtSpec.format(p.str));
        writeln(p.str, " ", ver);
    }
}
°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°

To get the whole list with the json is faster than scrapping. To get the version will certainly be slower (even in paralell because of the get() for each package but it's cleaner since it uses the API.

On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe wrote:
> - code.dlang.org has an api but doesn't provide an endpoint to retrieve all packages/version. Now I just scrape the site instead (thanks Adam for your dom implementation).

Why don't you make a PR to the dub registry (https://github.com/dlang/dub-registry) to get such an endpoint? Or at least open an issue ;-)

On Saturday, 6 August 2016 at 19:46:52 UTC, Seb wrote:
> That are excellent news!
Thanks.

> 1) Send the packages a notification about the build error (e.g. Github comment) - this should probably be tweaked a bit, s.t. it doesn't spam too often for still broken packages
I was thinking about having people register for notifications themselves.

> 2) Allow easy, manual build of special branches for the core team.
I need something similar for dev/testing purposes as well. Since I am using digger it is really easy to build whatever dmd + pull request is needed. Problem is controlling access.

> 3) Once you have the API
> a) (try to) get a shield badge (-> http://shields.io/)
Nice find. Will use.

> b) Make the data available to the dub-registry (-> https://github.com/dlang/dub-registry)
Sure.

> 4) Assess the quality of the unittests. Probably the easiest is `dub test -b unittest-cov`, and then summing up the total coverage of all generated .lst files
I am not sure this is a good idea. Besides the fact that coverage doesn't correlate with quality, it is outside of the purpose for this tool (identifying dmd regressions and identifying broken packages).

> 5) Log your daily "broken" statistics - could be a good indicator of whether your hard work gets acknowledged.
I rather hear it from people than seeing it in the stats :)

> 6) Regarding linker errors - I can only redirect you to the open DUB issue (https://github.com/dlang/dub/issues/852) and DEP 5 (https://github.com/dlang/dub/wiki/DEP5).
It is an open problem and I don't wont to solve it. For now I think I will just install the most important ones and just accept that not all packages will be build.

On another note, I do think the dub package definition could use some extra fields. Like compatible platforms and compatible dmd versions. Take vibe.d for instance, it is specifically build for certain dmd versions and it makes no sense for dubster to try to compile it with an unsupported version. Also, it allows dub itself to notify you of incompatible packages w.r.t. the installed compiler. Same idea for platform.

On Saturday, 6 August 2016 at 20:00:53 UTC, Basile B. wrote:
> On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe wrote:
>> - code.dlang.org has an api but doesn't provide an endpoint to retrieve all packages/version. Now I just scrape the site instead (thanks Adam for your dom implementation).
>
> No endpoint but still possible in two steps.
Yeah, I considered something like that myself. I find scraping to be better for both sides though, and simpler. A good api would certainly be better.

On Saturday, 6 August 2016 at 20:08:47 UTC, Seb wrote:
> Why don't you make a PR to the dub registry (https://github.com/dlang/dub-registry) to get such an endpoint? Or at least open an issue ;-)
https://github.com/dlang/dub-registry/issues/171

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
> I am aiming really low at first, but will eventually add things like memory usage, history, notifications, etc.

I actually don't think this makes sense. You're not in the position to maintain 1K+ packages, it's the library owners that need to test their code.
Just this short list I'm using for the project tester is hardly maintainable.
https://github.com/MartinNowak/project_tester (uses Jenkins, no need to write yet another CI).

I've already thought about many different aspects of this and here are the 2 things that are useful and might work out.

- Implement a tester that runs for every PR (just the other testers) and tests the most popular/important dub packages. Once a day is not enough b/c will feel responsible for breakages, we really need feedback before merging.

- Show test results of various CIs on code.dlang.org. Testing a dub package on Travis-CI is already a no-brainer. For example the following .travis.yml would test a package against all dmd release channels.

```yaml
language: d
d: [dmd, dmd-beta, dmd-nightly]
```

On Sunday, 7 August 2016 at 23:08:34 UTC, Martin Nowak wrote:
> I actually don't think this makes sense. You're not in the position to maintain 1K+ packages, it's the library owners that need to test their code.
Thanks for taking the time to respond.

I agree with you. Library owners should test their code themselves. But they don't. 24% of the packages don't build.

> Just this short list I'm using for the project tester is hardly maintainable.
I don't need to maintain anything besides linker errors. It is quite simple, I just run `dub test` and see what happens. If that doesn't work I consider it a failed build.

> https://github.com/MartinNowak/project_tester (uses Jenkins, no need to write yet another CI).
I would argue mine is simpler to deploy and have nodes join.

> I've already thought about many different aspects of this and here are the 2 things that are useful and might work out.
> - Implement a tester that runs for every PR (just the other testers) and tests the most popular/important dub packages.
> Once a day is not enough b/c will feel responsible for breakages, we really need feedback before merging.
It is just a matter of resources. I choose nightly since it seemed doable using just my own resources.

> - Show test results of various CIs on code.dlang.org. Testing a dub package on Travis-CI is already a no-brainer. For example the following .travis.yml would test a package against all dmd release channels.
>
> ```yaml
> language: d
> d: [dmd, dmd-beta, dmd-nightly]
> ```
Yes, that is quite nice. But that only gets triggered when the repo is updated.

All in all I understand your reservations, and I highly appreciate your feedback. I understand I won't bring the end-all solution to testing, but I do hope to reach the goals that I have set forth for myself. 1) catching (some) regressions, 2) giving insights into bit rot on code.dlang.org, 3) have fun.

It might take a couple of months before I reach them, or I might not at all.

On 08/08/2016 09:54 AM, Sebastiaan Koppe wrote:
> On Sunday, 7 August 2016 at 23:08:34 UTC, Martin Nowak wrote:
>> I actually don't think this makes sense. You're not in the position to maintain 1K+ packages, it's the library owners that need to test their code.
> Thanks for taking the time to respond.

You're welcome. This is an important topic for us.

> I agree with you. Library owners should test their code themselves. But they don't. 24% of the packages don't build.

We want better ranking of dub packages (mostly by download, but for sure also showing CI results [¹]). It's rather trivial to filter out low-quality packages b/c they're hardly used.

[¹]: https://trello.com/c/CaYJwtBV/60-integrate-ci-results-with-dub-registry

>> Just this short list I'm using for the project tester is hardly maintainable.
> I don't need to maintain anything besides linker errors. It is quite simple, I just run `dub test` and see what happens. If that doesn't work I consider it a failed build.
> 
>> https://github.com/MartinNowak/project_tester (uses Jenkins, no need
>> to write yet another CI).
> I would argue mine is simpler to deploy and have nodes join.

Is it already usable? How to deploy then? I need to test
https://github.com/dlang/druntime/pull/1602 and otherwise have to
resetup my project tester for that.
Adding more servers to Jenkins is trivial as well.

>> Once a day is not enough b/c will feel responsible for breakages, we really need feedback before merging.
> It is just a matter of resources. I choose nightly since it seemed doable using just my own resources.

Yes, but from past experience we know that people don't look at results, if you don't make it part of PR acceptance.

>> - Show test results of various CIs on code.dlang.org. Testing a dub package on Travis-CI is already a no-brainer. For example the following .travis.yml would test a package against all dmd release channels.
>>
>> ```yaml
>> language: d
>> d: [dmd, dmd-beta, dmd-nightly]
>> ```
> Yes, that is quite nice. But that only gets triggered when the repo is updated.

Travis now allows cron scheduling, you still have to ask their support to unlock that.

-Martin

On Wednesday, 10 August 2016 at 10:32:24 UTC, Martin Nowak wrote:
> We want better ranking of dub packages (mostly by download, but for sure also showing CI results [¹]).

I was also thinking about integrating results from CI builds that packages do themselves. But there is some 'impedance mismatch': those CI build are done on the master branch, not on the latest release that is on code.dlang.org.

> Is it already usable?

Short answer: No.

I am currently test running it on all packages against the 10 latest dmd releases (I have done 6k packages on and off since 2 days ago). But I am running into vibe.d issues/missing features. Things like not being able to use gzip with requestHttp (let alone with a RestInterfaceClient), invalid internal state with the http client pool on interrupting requests, and some other things.

Also, I am writing a PR for vibe.d to send http request to unix sockets.

> How to deploy then?

For the worker it's just a docker container. But until the unix sockets PR is done you do have to setup the docker daemon to listen on the docker0 interface.

> I need to test
> https://github.com/dlang/druntime/pull/1602 and otherwise have to resetup my project tester for that.

I am using digger to build dmd, so adding in the pull request is trivial. I do need to adjust internals to properly handle it though.

But alas, family is coming over so don't expect anything anytime soon.

> Yes, but from past experience we know that people don't look at results, if you don't make it part of PR acceptance.

So true. Then I will do PR's first.

On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe wrote:
>> Yes, but from past experience we know that people don't look at results, if you don't make it part of PR acceptance.
>
> So true. Then I will do PR's first.

Thinking about it, you could also opt for integrating it with the dmd PR flow - in a similar manner to the autotester or coverage bot:
Select a subset (depending on the runtime) of packages and run your dub autotester for every commit and thus save for every commit a list of passing packages.
Now for a new PR, search for the master commit hash in your DB of runs and run the dub autotester with those packages. The workflow of the AutoTester [1] is a bit more complicated, because it is throwing away results as soon as the master HEAD changes (to avoid any inconsistencies) and there are often rebases and additional pushes happening, but you could just opt for a simple 80% solution.
I imagine shouting at Walter with a Github comment "Hey this PR will break 10% of all packages [of the subset]" could be quite helpful ;-)

[1] https://auto-tester.puremagic.com