Hi,

After some lamenting I've finally gotten round to setting up a buildbot for gdc.

https://buildbot.dgnu.org

Fitted with a smorgasbord of cross compiler builds, all of them are broken in one way or another, given the definition of "not broken" being:

 - Can build a usable compiler: all should be green here.
 - Can build druntime and libphobos: I think only ARM holds this title.
 - Passes the testsuite with zero failures: There exist tests that ICE on certain targets.

It it my hope this encourages fixing broken targets, and flags up regressions in build immediately, not months down the line later.

Build configuration is available on github.  Using docker so that anyone can set up and debug failing builds locally.

https://github.com/D-Programming-GDC/buildbot-gdc


Regards,
Iain.

Am Sun, 02 Jul 2017 19:53:05 +0000
schrieb Iain Buclaw <ibuclaw@gdcproject.org>:

> Hi,
> 
> After some lamenting I've finally gotten round to setting up a buildbot for gdc.
> 
> https://buildbot.dgnu.org
> 
> Fitted with a smorgasbord of cross compiler builds, all of them are broken in one way or another, given the definition of "not broken" being:
> 
>   - Can build a usable compiler: all should be green here.
>   - Can build druntime and libphobos: I think only ARM holds this
> title.
>   - Passes the testsuite with zero failures: There exist tests
> that ICE on certain targets.
> 
> It it my hope this encourages fixing broken targets, and flags up regressions in build immediately, not months down the line later.
> 
> Build configuration is available on github.  Using docker so that anyone can set up and debug failing builds locally.
> 
> https://github.com/D-Programming-GDC/buildbot-gdc
> 
> 
> Regards,
> Iain.

Nice. I had a similar idea but only managed to read through the first 120 pages of the buildbot manual last weekend ;-) Buildbot sounds really interesting as it seems to be highly customizable. The most complicated but also useful part seems to somehow use multiple workers for one build step. This is required for advanced cross compiler testing, though in a simple setup we can just dedicate one ARM machine to a X86 machine and simply keep the ARM machine out of the buildbot configuration. The X86 can then use standard dejagnu+ssh to connect to the ARM machine.

Another interesting thing is support of libvirt, so we should be able to test in virtualized QEMU VMs. For some architectures hardware for testing seems hard to come by (ARM and MIPS is easy, but SH4, PPC, ...). One thin buildbot doesn't seem to support out of the box is a docker latent worker hosted in a libvirt latent worker, but I guess this shouldn't be difficult to add.


Regarding failing builds, seems like the new ARM arm-d.c file is just missing an import. AArch64, s390, hppa, ppc, sh4, mips and alpha are OK (as in fail in druntime). For gnuspe and sparc64 seems like the new config mechanism is broken.

BTW: Do you know if there's any way to cluster builds by branch on the buildbot main page? I haven't gotten that far in the docs yet ;-)

-- Johannes

On 4 July 2017 at 12:10, Johannes Pfau via D.gnu <d.gnu@puremagic.com> wrote:
>
> Nice. I had a similar idea but only managed to read through the first 120 pages of the buildbot manual last weekend ;-) Buildbot sounds really interesting as it seems to be highly customizable. The most complicated but also useful part seems to somehow use multiple workers for one build step. This is required for advanced cross compiler testing, though in a simple setup we can just dedicate one ARM machine to a X86 machine and simply keep the ARM machine out of the buildbot configuration. The X86 can then use standard dejagnu+ssh to connect to the ARM machine.
>

Yeah, I think just using dejagnu + remote testing would be the way to go.  This can be part of the buildci.sh script (which should be merged with semaphoreci.sh).

In any case, there should probably be a way to disable/skip runnable tests if host != target.

https://gcc.gnu.org/onlinedocs/gccint/Directives.html#Directives

{ dg-do run  [{ target $host_triplet }] }

Probably not the wisest thing to do though.


> Another interesting thing is support of libvirt, so we should be able to test in virtualized QEMU VMs. For some architectures hardware for testing seems hard to come by (ARM and MIPS is easy, but SH4, PPC, ...). One thin buildbot doesn't seem to support out of the box is a docker latent worker hosted in a libvirt latent worker, but I guess this shouldn't be difficult to add.
>

How do these look?

https://www.scaleway.com/armv8-cloud-servers/

They also have ARMv7's too, I'm currently paying around 18€ a month for the current linode box - 50G SSD, 4G memory, 2x cores.  For the same price, could order the following kit for use as build workers.

ARMv8: 50G SSD, 2G memory, 4x cores.  (aarch64 native)
ARMv7: 50G SSD, 2G memory, 4x cores.  (arm native)
x86_64: 100G SSD, 4G memory, 4x cores. (all cross compilers)

Then move all hosted sites + database stuff to a cheap box.

x86_64: 50G SSD, 2G memory, 2x cores.

Should all come up to 14€ a month, unless there are hidden costs. :-)

Could even add another cheap x86 box and have that do x86_64 native builds also - but semaphoreCI is doing well enough for that purpose so far...

>
> Regarding failing builds, seems like the new ARM arm-d.c file is just missing an import. AArch64, s390, hppa, ppc, sh4, mips and alpha are OK (as in fail in druntime). For gnuspe and sparc64 seems like the new config mechanism is broken.
>

Yep, as this is dockerized, anyone can build + debug locally.  This the patches should be fixed asap.  Also, if you look at the runtests (skipping over the unresolved bits), you'll see that a few of them ICE.  I've noticed this for aarch64, maybe others do the same.  This is another thing that should be investigated and fixed.

In other words, I would like to get all of them green, but only have so much time. I better start by filtering out the noise first. :-)

> BTW: Do you know if there's any way to cluster builds by branch on the buildbot main page? I haven't gotten that far in the docs yet ;-)
>

Doesn't look like it.

https://github.com/buildbot/buildbot/blob/0d44f0344ff82b707d02c75871df23c5f6b9cb8f/www/base/src/app/home/home.tpl.jade#L18-L24


Regards,
Iain.

Am Tue, 4 Jul 2017 20:42:52 +0200
schrieb "Iain Buclaw via D.gnu" <d.gnu@puremagic.com>:

> 
> How do these look?
> 
> https://www.scaleway.com/armv8-cloud-servers/
> 
> They also have ARMv7's too, I'm currently paying around 18€ a month for the current linode box - 50G SSD, 4G memory, 2x cores.  For the same price, could order the following kit for use as build workers.
> 
> ARMv8: 50G SSD, 2G memory, 4x cores.  (aarch64 native)
> ARMv7: 50G SSD, 2G memory, 4x cores.  (arm native)
> x86_64: 100G SSD, 4G memory, 4x cores. (all cross compilers)

The ARMs look good. I'm not sure about the X86_64: The 'VPS' are
'shared' but I haven't found any information what exactly they share
(shared cores?). Additionally some 2016 reports say the VPS use
Intel Atom cores and only the more expensive 'Bare Metal' plans use Xeon
Cores. I haven't found any official information on the homepage though
so we might have to ask or just try and check available CPU
resources / build speed.

> 
> Yep, as this is dockerized, anyone can build + debug locally.  This the patches should be fixed asap.  Also, if you look at the runtests (skipping over the unresolved bits), you'll see that a few of them ICE.  I've noticed this for aarch64, maybe others do the same.  This is another thing that should be investigated and fixed.
> 
> In other words, I would like to get all of them green, but only have so much time. I better start by filtering out the noise first. :-)

I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.

> > BTW: Do you know if there's any way to cluster builds by branch on
> > the buildbot main page? I haven't gotten that far in the docs
> > yet ;-)
> 
> Doesn't look like it.
> 
> https://github.com/buildbot/buildbot/blob/0d44f0344ff82b707d02c75871df23c5f6b9cb8f/www/base/src/app/home/home.tpl.jade#L18-L24

OK, then this is something to look into (a lot) later. I guess buildbot should allow setting up custom sub pages so there's likely some way to implement a per-branch overview.

-- Johannes

On 4 July 2017 at 23:05, Johannes Pfau via D.gnu <d.gnu@puremagic.com> wrote:
> Am Tue, 4 Jul 2017 20:42:52 +0200
> schrieb "Iain Buclaw via D.gnu" <d.gnu@puremagic.com>:
>
>>
>> How do these look?
>>
>> https://www.scaleway.com/armv8-cloud-servers/
>>
>> They also have ARMv7's too, I'm currently paying around 18€ a month for the current linode box - 50G SSD, 4G memory, 2x cores.  For the same price, could order the following kit for use as build workers.
>>
>> ARMv8: 50G SSD, 2G memory, 4x cores.  (aarch64 native)
>> ARMv7: 50G SSD, 2G memory, 4x cores.  (arm native)
>> x86_64: 100G SSD, 4G memory, 4x cores. (all cross compilers)
>
> The ARMs look good. I'm not sure about the X86_64: The 'VPS' are
> 'shared' but I haven't found any information what exactly they share
> (shared cores?). Additionally some 2016 reports say the VPS use
> Intel Atom cores and only the more expensive 'Bare Metal' plans use Xeon
> Cores. I haven't found any official information on the homepage though
> so we might have to ask or just try and check available CPU
> resources / build speed.
>

Well, the linode is running on a shared server as well (it's been given 2 cores of a 12 core Xeon), so there's not much difference there.  It seems that scaleway use specifically Avotons for their low-end bare metal servers  (I did a quick look up for comparison: http://ark.intel.com/compare/81908,77982).

I guess for the cross compilers, speed doesn't really bother me all that much.  I'll probably just leave them to build master only anyway, I don't see it necessary to have instant feedback from PRs with them, an FYI for a recent change breaking something is enough - remember, there are 17 configurations to test! (though I turned off half of them due to lack of disk space).

For native builds, having a policy of must never break seems more reasonable.


>>
>> Yep, as this is dockerized, anyone can build + debug locally.  This the patches should be fixed asap.  Also, if you look at the runtests (skipping over the unresolved bits), you'll see that a few of them ICE.  I've noticed this for aarch64, maybe others do the same.  This is another thing that should be investigated and fixed.
>>
>> In other words, I would like to get all of them green, but only have so much time. I better start by filtering out the noise first. :-)
>
> I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.
>

I hope that it should be super easy - I used docker 17.06 and docker-compose 1.8.  Should just be `docker-compose up worker-cross` to fire up the build environment.  :-)


>> > BTW: Do you know if there's any way to cluster builds by branch on
>> > the buildbot main page? I haven't gotten that far in the docs
>> > yet ;-)
>>
>> Doesn't look like it.
>>
>> https://github.com/buildbot/buildbot/blob/0d44f0344ff82b707d02c75871df23c5f6b9cb8f/www/base/src/app/home/home.tpl.jade#L18-L24
>
> OK, then this is something to look into (a lot) later. I guess buildbot should allow setting up custom sub pages so there's likely some way to implement a per-branch overview.
>

I have a quick look to see if gdb or anyone else does this - answer, it seems like they don't, it's just bundled in together.

https://gdb-build.sergiodj.net/grid http://buildbot.python.org/all/grid http://buildbot.nektar.info/grid

What you see are the build results for the last X changes, irrespective of branch.

Regards,
Iain.

On Tuesday, 4 July 2017 at 22:45:45 UTC, Iain Buclaw wrote:
>> The ARMs look good. I'm not sure about the X86_64: The 'VPS' are
>> 'shared' but I haven't found any information what exactly they share
>> (shared cores?). Additionally some 2016 reports say the VPS use
>> Intel Atom cores and only the more expensive 'Bare Metal' plans use Xeon
>> Cores.

Yes, it's Avotons, furthermore many of the virtualized Scaleway x86_64 share a network connected storage controller with many CPU sockets (called LSSD). While this allows migration, it also adds some latency.
I'm running the nightly build server on a bare-metal Avoton, about 3x slower single-core than a Xeon, but you get 8 of 'em and full virtualization support https://www.online.net/en/dedicated-server/dedibox-xc.
The Jenkins at ci.dlang.org is running on a https://www.online.net/en/dedicated-server/dedibox-lt though it idles so much that I consider to use EC2 spot instances as workers https://github.com/dlang/ci/issues/37.

Am Tue, 4 Jul 2017 23:05:23 +0200
schrieb Johannes Pfau <nospam@example.com>:

> I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.

The ARM config patch problem can be fixed by including arm-protos.h:

#include "config.h"
#include "system.h"
#include "coretypes.h"
#include "target.h"
#include "arm-protos.h"
#include "d/d-target.h"
#include "d/d-target-def.h"

/* Implement TARGET_D_CPU_VERSIONS for ARM targets.  */


-- Johannes

Am Wed, 5 Jul 2017 12:18:28 +0200
schrieb Johannes Pfau <nospam@example.com>:

> Am Tue, 4 Jul 2017 23:05:23 +0200
> schrieb Johannes Pfau <nospam@example.com>:
> 
> > I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.
> 
> The ARM config patch problem can be fixed by including arm-protos.h:
> 

SPARC needs #include "memmodel.h" in glibc-d.c (Or probably sparc-protos.h should be fixed to include memmodel instead, as the sparc-protos.h file actually depends on memmodel).

powerpc-linux-gnuspe needs a #include "target.h" in config/powerpcspe/powerpcspe-d.c

This should fix all gdc buildbot builds. Next step after fixing these build errors is probably adding a gdc -v call to the buildci script after make all-gcc so we can easily check whether predefined versions are defined correctly.


-- Johannes

On 5 July 2017 at 12:18, Johannes Pfau via D.gnu <d.gnu@puremagic.com> wrote:
> Am Tue, 4 Jul 2017 23:05:23 +0200
> schrieb Johannes Pfau <nospam@example.com>:
>
>> I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.
>
> The ARM config patch problem can be fixed by including arm-protos.h:
>
> #include "config.h"
> #include "system.h"
> #include "coretypes.h"
> #include "target.h"
> #include "arm-protos.h"
> #include "d/d-target.h"
> #include "d/d-target-def.h"
>
> /* Implement TARGET_D_CPU_VERSIONS for ARM targets.  */
>
>
> -- Johannes
>

Hmm, ok.  That should probably be tm_p.h then.  I've s/target.h/tm.h/ in all patched sources.  That's all we need to pull in for CPU related information.

Iain.

On 5 July 2017 at 15:33, Johannes Pfau via D.gnu <d.gnu@puremagic.com> wrote:
> Am Wed, 5 Jul 2017 12:18:28 +0200
> schrieb Johannes Pfau <nospam@example.com>:
>
>> Am Tue, 4 Jul 2017 23:05:23 +0200
>> schrieb Johannes Pfau <nospam@example.com>:
>>
>> > I'll try to setup a local builder for debugging later this week or next weekend and see if I can reduce some bugs.
>>
>> The ARM config patch problem can be fixed by including arm-protos.h:
>>
>
> SPARC needs #include "memmodel.h" in glibc-d.c (Or probably sparc-protos.h should be fixed to include memmodel instead, as the sparc-protos.h file actually depends on memmodel).
>

Actually, including memmodel.h sounds correct, have done it explicitly for d-target.cc a couple times.


> powerpc-linux-gnuspe needs a #include "target.h" in config/powerpcspe/powerpcspe-d.c
>

Added "tm.h".

> This should fix all gdc buildbot builds. Next step after fixing these build errors is probably adding a gdc -v call to the buildci script after make all-gcc so we can easily check whether predefined versions are defined correctly.
>

OK.

Iain.