September 16, 2020
On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
> However, on a closer look I can spot multiple failures or files/directories which can't be found.
> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
> - The build package step failed with: cd ./libcc1: no such file or directory.

I had a look on the most recent build, and they are all timeout failures.  This has on some rare occasions happened locally.  It is related to building phobos unittests with optimizations enabled.

The default timeout value is 300, maybe it should be increased to 600.

Should be as simple as adding:

    global tool_timeout
    set tool_timeout 600

Inside this proc https://github.com/D-Programming-GDC/gcc/blob/8177cfa01e10aabb29bc8496657ff0b847e9ceb9/libphobos/testsuite/lib/libphobos.exp#L96-L100
September 16, 2020
On Wednesday, 16 September 2020 at 10:57:46 UTC, Iain Buclaw wrote:
> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>> [...]
>
> I had a look on the most recent build, and they are all timeout failures.  This has on some rare occasions happened locally.  It is related to building phobos unittests with optimizations enabled.
>
> The default timeout value is 300, maybe it should be increased to 600.
>
> Should be as simple as adding:
>
>     global tool_timeout
>     set tool_timeout 600
>
> Inside this proc https://github.com/D-Programming-GDC/gcc/blob/8177cfa01e10aabb29bc8496657ff0b847e9ceb9/libphobos/testsuite/lib/libphobos.exp#L96-L100

Done.
Is it supposed to not error out on time out failures ?
September 16, 2020
On Wednesday, 16 September 2020 at 00:49:52 UTC, Seb wrote:
>
> If you manage to build tarballs of the binaries, I think a lot of people would greatly appreciate if they are made available (e.g. can be done directly on GitHub via "Releases").
> For example, see https://github.com/dlang/installer/pull/251, https://forum.dlang.org/thread/xktompypwvaabwebnjol@forum.dlang.org, or https://forum.dlang.org/thread/bnkbldsifjhsseswiceq@forum.dlang.org .

Cirrus CI doesn't support to make releases available in the GitHub repository (at least no at the moment). But they provide an example script of how that can be achieved.

I think such a publish script is even better because it's possible to push the downloads to external mirrors as well.

September 16, 2020
On Wednesday, 16 September 2020 at 10:42:27 UTC, Iain Buclaw wrote:
> On Wednesday, 16 September 2020 at 09:55:22 UTC, wjoe wrote:
>>
>> The way it's being done right now is that 'make install' installs to the /usr prefix. After that a tarball of this prefix is created (via tar cJf gdc-triplet.txz /usr). I'm not sure if that's suitable as a release as is because tar omits the root / so the result will be extracted as usr/
>> There isn't a lot of time budget left in that task but it should be possible to run some more scripts.
>> If the time limit won't suffice it should be possible to cache /usr and move the tar ball script into a new task.
>>
>
> If it follows the convention of the existing packages, it should be fine.  e.g: tar extracts gdc into 'x86_64-unknown-linux-gnu/bin/gdc'

The tar ball is 443MiB. That's because it includes half the docker container :)

The buildci script [1] uses hard coded --prefix=/usr and lib-dirs=/usr/lib.
Is there a particular reason for that ?
Or, rather, could I just change it or introduce a variable prefix in order to be able to use an isolated directory ?

[1] https://github.com/W-joe/gcc/blob/master-ci/buildci.sh#L274-L280
September 16, 2020
On Wednesday, 16 September 2020 at 11:17:41 UTC, wjoe wrote:
> On Wednesday, 16 September 2020 at 10:57:46 UTC, Iain Buclaw wrote:
>> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>>> [...]
>>
>> I had a look on the most recent build, and they are all timeout failures.  This has on some rare occasions happened locally.  It is related to building phobos unittests with optimizations enabled.
>>
>> The default timeout value is 300, maybe it should be increased to 600.
>>
>> Should be as simple as adding:
>>
>>     global tool_timeout
>>     set tool_timeout 600
>>
>> Inside this proc https://github.com/D-Programming-GDC/gcc/blob/8177cfa01e10aabb29bc8496657ff0b847e9ceb9/libphobos/testsuite/lib/libphobos.exp#L96-L100
>
> Done.
> Is it supposed to not error out on time out failures ?

The testsuite runs until completion, it doesn't exit after the first failure occurred.

The buildci script can be tweaked to return failure if there's any FAIL tests in the summary log as part of the final step.
September 16, 2020
On Wednesday, 16 September 2020 at 12:50:57 UTC, wjoe wrote:
> On Wednesday, 16 September 2020 at 10:42:27 UTC, Iain Buclaw wrote:
>> On Wednesday, 16 September 2020 at 09:55:22 UTC, wjoe wrote:
>>>
>>> The way it's being done right now is that 'make install' installs to the /usr prefix. After that a tarball of this prefix is created (via tar cJf gdc-triplet.txz /usr). I'm not sure if that's suitable as a release as is because tar omits the root / so the result will be extracted as usr/
>>> There isn't a lot of time budget left in that task but it should be possible to run some more scripts.
>>> If the time limit won't suffice it should be possible to cache /usr and move the tar ball script into a new task.
>>>
>>
>> If it follows the convention of the existing packages, it should be fine.  e.g: tar extracts gdc into 'x86_64-unknown-linux-gnu/bin/gdc'
>
> The tar ball is 443MiB. That's because it includes half the docker container :)
>

You should be able to get yourself down to about ~30MB. ;-)

I'll see if there's any miscellaneous tweaks you can do, but the most obvious one is `strip --strip-debug`.

> The buildci script [1] uses hard coded --prefix=/usr and lib-dirs=/usr/lib.
> Is there a particular reason for that ?
> Or, rather, could I just change it or introduce a variable prefix in order to be able to use an isolated directory ?
>
> [1] https://github.com/W-joe/gcc/blob/master-ci/buildci.sh#L274-L280

IIRC, that top line just matches Debian/Ubuntu built gcc (in the hope that no weirdness would happen when running testsuite).

Seems reasonable to break it out into a variable that can be overridden by the CI.

Just looking at an old binary, the builder used `--prefix=/home/build/share/cache/install/x86_64-unknown-linux-gnu`.  Not saying that you should do the same, but the last part being the target triplet is the key.

...

It may only be a marginal gain, but I find that --disable-libstdcxx-pch helps with speeding up incremental builds (a long time is spent compiling headers in libstdc++).



September 16, 2020
On Wednesday, 16 September 2020 at 10:47:01 UTC, wjoe wrote:
> On Wednesday, 16 September 2020 at 10:23:16 UTC, Iain Buclaw wrote:
>> On Wednesday, 16 September 2020 at 09:33:05 UTC, wjoe wrote:
>>>[...]
>>
>> build_bootstrap=enabled should be OK.  Even better, it configures with --enable-checking=release.
>>
>>
>>
>>>[...]
>>
>> All tasks could be ran in parallel as there's no dependencies between them.  The Package task though I'd imagine should be reserved for the release branches, so it's only ran infrequently.
>
> Configuration wise that's not a problem at all. My reasoning for the dependency was that there's no point in making a release tar ball if the Unittests task fails.

It's not really the end of the world if a test fails.  While tagged releases should ideally all pass, some failures can occur that are not our fault (i.e: x32 has a libc bug that causes some syscalls to fail and trigger asserts in a couple libphobos tests).

For non-tagged builds, I imagine we'd just be replacing the previously built tarball based on the given branch it was built off, so if something really is broken, in the worst case we just wait until a fix goes in and retrigger CI.  Or if some downstream is affected, we'd have some sort of versioning in place (such as syncthing) to do a quick restore.
September 16, 2020
On Wednesday, 16 September 2020 at 21:05:27 UTC, Iain Buclaw wrote:
> On Wednesday, 16 September 2020 at 12:50:57 UTC, wjoe wrote:
>>[...]
>
> You should be able to get yourself down to about ~30MB. ;-)
>
> I'll see if there's any miscellaneous tweaks you can do, but the most obvious one is `strip --strip-debug`.
>
> [...]

I guess the /usr prefix makes sense if it's built for the .deb package.

>
> [...]

A triplet prefix it is :)

>
> [...]

I think this may come in handy.
September 16, 2020
On Wednesday, 16 September 2020 at 21:05:27 UTC, Iain Buclaw wrote:
>
> You should be able to get yourself down to about ~30MB. ;-)
>
> I'll see if there's any miscellaneous tweaks you can do, but the most obvious one is `strip --strip-debug`.

`make install-strip` should do the trick.

Also, the final installation docs [1] advise against stripping the GNAT runtime, as this would break certain features of the debugger that depend on this debugging information (catching Ada exceptions for instance). Is that relevant ?

[1] https://gcc.gnu.org/install/finalinstall.html
September 16, 2020
On Wednesday, 16 September 2020 at 21:25:19 UTC, Iain Buclaw wrote:
> On Wednesday, 16 September 2020 at 10:47:01 UTC, wjoe wrote:
>> On Wednesday, 16 September 2020 at 10:23:16 UTC, Iain Buclaw wrote:
>>> [...]
>>
>> Configuration wise that's not a problem at all. My reasoning for the dependency was that there's no point in making a release tar ball if the Unittests task fails.
>
> It's not really the end of the world if a test fails.  While tagged releases should ideally all pass, some failures can occur that are not our fault (i.e: x32 has a libc bug that causes some syscalls to fail and trigger asserts in a couple libphobos tests).
>
> For non-tagged builds, I imagine we'd just be replacing the previously built tarball based on the given branch it was built off, so if something really is broken, in the worst case we just wait until a fix goes in and retrigger CI.  Or if some downstream is affected, we'd have some sort of versioning in place (such as syncthing) to do a quick restore.

I see. So I'll just remove the dependency and let all the tasks run in parallel. It may still be delayed due to scheduling but time wise it won't be worse that with a dependency on Unittest.