On Sunday, 17 July 2016 at 13:17:45 UTC, Jacob Carlborg wrote:
> On 2016-07-16 22:34, Sebastiaan Koppe wrote:
> Why not using something existing, like GitLab? Although GitLab is a source code hosting system its CI is excellent. It uses a master-worker architecture as well, GitLab being the master and one or more runners. Runners are available for all major operating systems: macOS, Linux and Windows. For Linux a Docker runner is available. This way it could run tests on multiple platforms. The installation is straight forward with native packages.

I don't have a good answer for this question. It might very well be the case that going with Gitlab (or similar) would be the better option.

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
> Just to let you guys know - and to be sure no one is doing the same - I decided to go ahead and *start* writing an autotester that will fetch dmd nightly and unittest each dub package.
>
> It will be using a classic master-worker architecture and will leverage docker containers.
>
> I am aiming really low at first, but will eventually add things like memory usage, history, notifications, etc.

Great! Maybe I can help you? Do you have a repository somewhere already?

I don't think nightlies are that important. Older releases, alpha/beta versions, LDC, and GDC seem more important for me. For example, I would like to know if a dub package which was last updated two years ago is still working with the current dmd.

The hardest part is probably the work distribution. It should work across platforms, so we can (eventually) test Windows, Android, Raspberry Pi, etc.

I don't believe GitLab would be a good idea. It is not built for this and I find the CI parts quite minimal. I have some buildbot experience, it would fit, but is a bitch to maintain. Maybe v0.9 is better when it is finished, but development happened at glacial speed in the last years (!).

If we build something custom, the question is: Dogfooding or not? With (e.g.) Python we would have something working much quicker.

On Monday, 18 July 2016 at 07:22:07 UTC, qznc wrote:
> Great! Maybe I can help you? Do you have a repository somewhere already?

Not yet. Let me first do some groundwork. It could take month though.

> I don't think nightlies are that important. Older releases, alpha/beta versions, LDC, and GDC seem more important for me. For example, I would like to know if a dub package which was last updated two years ago is still working with the current dmd.

Once infrastructure is in place everything is just a git commit hash. Well, kind of.

> The hardest part is probably the work distribution. It should work across platforms, so we can (eventually) test Windows, Android, Raspberry Pi, etc.

Like I said I am aiming really low. On purpose. I have a wife and two kids and I need to keep the scope limited. The first step is writing something that works in under 1kloc in 80% of the cases. If it would catch one regression per week/month before they end up in release I would be quite happy.

All the fancy stuff comes after that.

> If we build something custom, the question is: Dogfooding or not? With (e.g.) Python we would have something working much quicker.

It should be written in D, that way everybody is a potential contributor.

On Monday, 18 July 2016 at 09:55:04 UTC, Sebastiaan Koppe wrote:
> Not yet. Let me first do some groundwork. It could take month though.
>
> I have a wife and two kids and I need to keep the scope limited.

Hey, me too. Slow and steady wins the race. ;)

On 2016-07-18 09:22, qznc wrote:

> The hardest part is probably the work distribution. It should work
> across platforms, so we can (eventually) test Windows, Android,
> Raspberry Pi, etc.

GitLab can handle this really easy.

> I don't believe GitLab would be a good idea. It is not built for this
> and I find the CI parts quite minimal.

No, it's not built for this but I don't see a reason why it wouldn't work. I use GitLab extensively at work and it works great.

> I have some buildbot experience, it would fit, but is a bitch to maintain.

GitLab is not :)

-- 
/Jacob Carlborg

On 2016-07-18 11:55, Sebastiaan Koppe wrote:

> Like I said I am aiming really low. On purpose. I have a wife and two
> kids and I need to keep the scope limited.

In that case, go with something that already exists.

-- 
/Jacob Carlborg

On Monday, 18 July 2016 at 18:47:28 UTC, Jacob Carlborg wrote:
> On 2016-07-18 11:55, Sebastiaan Koppe wrote:
>
>> Like I said I am aiming really low. On purpose. I have a wife and two
>> kids and I need to keep the scope limited.
>
> In that case, go with something that already exists.

I think Martin Nowak has some sort of automated setup for testing a limited number of dub packages against each release, but I can't find the relevant post at the moment.

On 2016-07-16 22:34, Sebastiaan Koppe wrote:
> Just to let you guys know - and to be sure no one is doing the same - I
> decided to go ahead and *start* writing an autotester that will fetch
> dmd nightly and unittest each dub package.
>
> It will be using a classic master-worker architecture and will leverage
> docker containers.
>
> I am aiming really low at first, but will eventually add things like
> memory usage, history, notifications, etc.

Just as a test I setup a project on GitLab.com [1]. This is an example of a build [2]. The config for building that looks like:

linux:
  script:
    - curl -L -o dvm https://github.com/jacob-carlborg/dvm/releases/download/v0.4.4/dvm-0.4.4-linux-debian7-x86_64
    - chmod +x dvm
    - ./dvm install dvm
    - source ~/.bashrc
    - dvm install 2.071.1
    - dvm use 2.071.1 -d
    - curl -o dub.tar.gz 'http://code.dlang.org/files/dub-1.0.0-linux-x86_64.tar.gz'
    - tar -xzf dub.tar.gz
    - ./dub test

"linux" is the name of the job and "script" is the commands that should be executed.

The setup I was thinking about would work something like this:

1. Create one or more runners for each supported platform
2. Tag those in GitLab with the name of the platform
3. Create one job per platform that should be tested
4. Use the tags to specify which runner should be used, something like this [3]:

linux_x86_64:
  tags:
    - linux_x86_64
  script:
    - ./gitlab.sh

windows:
  tag:
    - windows
  script: gitlab.bat

osx:
  tag:
    - osx
  script: ./gitlab.sh

5. Update code.dlang.org to mirror the repository to GitLab when it finds an update in GitHub
6. Trigger a build using the GitLab API [4]:

curl -X POST \
  -F token=TOKEN \
  -F ref=master \
  https://gitlab.example.com/api/v3/projects/9/trigger/builds

One issue is to get the .gitlab-ci.yml file in the repository.

Since it's possible to host GitLab and the runners you get full control of everything.

[1] https://gitlab.com/Carlborg/orange
[2] https://gitlab.com/Carlborg/orange/builds/2450044
[3] https://gitlab.com/help/ci/yaml/README.md
[4] https://gitlab.com/help/ci/triggers/README.md

-- 
/Jacob Carlborg

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
> Just to let you guys know - and to be sure no one is doing the same - I decided to go ahead and *start* writing an autotester that will fetch dmd nightly and unittest each dub package.
>
> It will be using a classic master-worker architecture and will leverage docker containers.
>
> I am aiming really low at first, but will eventually add things like memory usage, history, notifications, etc.

Nice :)

I have just finished a first iteration of dubster, a test runner that runs `dub test` on each package for each dmd release.

see https://github.com/skoppe/dubster

Please provide feedback as it will determine the direction/life of this tester.

I am planning on adding a web ui/api next to look around in the data.

Today I gave it a spin and let it run on 488 packages on dub (about half). The component that runs `dub test` was done on a 2gb 2vcpu cloud instance. It compiled the packages with dmd 2.071.2-b2 and it took about 20 sec per package on average.

59 packages didn't build because of missing libraries. See the end for a total list of missing libaries.
13 packages caused dmd to use too much memory for the instance I was running it on.
112 packages had build errors, almost all of them with exitcode 1, no segfaults.
35 packages had unittests that retured non zero exitcodes (due to exceptions and failing assertions).
213 packages passed their unittests.

The remaining 56 packages I couldn't categorize automatically so easily. Would have to take a deeper look at them.

Some issues along the way:

- code.dlang.org has an api but doesn't provide an endpoint to retrieve all packages/version. Now I just scrape the site instead (thanks Adam for your dom implementation).

- Originally I was planning on running with nightlies, but the ones on the download section don't have a git commit hash associated with them. For now I just use digger to build the latest dmd releases on the worker nodes.

- Some packages when running `dub test` didn't terminate on their own.

- Linker errors (a lot of them windows)

aclui, advapi32, asound, blas, bzip2, comctl32, comdlg32, ev, fcgi, fdb_c, fmod, ftgl, gccjit, gdi32, git2, GL, gsl, gslcblas, gumbo, imm32, iup, iupcontrols, jack, Judy, kernel32, lapack, lapacke, leveldb, libco, libshp, lz32, mad, miniupnpc, mpr, mrss, mysqlclient, nanomsg, nanovg, netapi32, Netapi32, netcdf, nlopt, ole32, oleacc, oleaut32, OpenCL, powrprof, pq, rasapi32, rdkafka, rpcns4, Rpcrt4, rpcrt4, sapnwrfc, scrypt, secur32, setupapi, shell32, shlwapi, snappy, sodium, tarsnap, tcc, tcl, tcmalloc, tk, udis86, usb, user32, version, vfw32, wayland, webp, winhttp, wininet, winmm, winspool, Ws2_32, wtsapi32, X11, xcb, xkbcommon, zlib, zmq, zookeeper_mt