On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
> However, on a closer look I can spot multiple failures or files/directories which can't be found.
> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
> - The build package step failed with: cd ./libcc1: no such file or directory.

This is the package function for reference:

build_package() {
    cd build || exit 1
    make install || exit 1
    tar -cJf gdc-${build_host}.txz /usr || exit 1
}

Links for your convenience.
[1] https://github.com/w-joe/gcc/blob/master-ci/buildci.sh
[2] https://github.com/w-joe/gcc/blob/master-ci/.cirrus.yml

On Tuesday, 15 September 2020 at 16:05:53 UTC, wjoe wrote:
> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
>> However, on a closer look I can spot multiple failures or files/directories which can't be found.
>> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
>> - The build package step failed with: cd ./libcc1: no such file or directory.
>
> This is the package function for reference:
>
> build_package() {
>     cd build || exit 1
>     make install || exit 1
>     tar -cJf gdc-${build_host}.txz /usr || exit 1
> }
>
> Links for your convenience.
> [1] https://github.com/w-joe/gcc/blob/master-ci/buildci.sh
> [2] https://github.com/w-joe/gcc/blob/master-ci/.cirrus.yml

Note the comments in the build function, only the dependencies of the C++ and D libraries are built. So you'll need to do `make all` in order to cover anything that was deliberately skipped.

On Tuesday, 15 September 2020 at 16:05:53 UTC, wjoe wrote:
> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
>> However, on a closer look I can spot multiple failures or files/directories which can't be found.
>> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
>> - The build package step failed with: cd ./libcc1: no such file or directory.
>
> This is the package function for reference:
>
> build_package() {
>     cd build || exit 1
>     make install || exit 1
>     tar -cJf gdc-${build_host}.txz /usr || exit 1
> }
>
> Links for your convenience.
> [1] https://github.com/w-joe/gcc/blob/master-ci/buildci.sh
> [2] https://github.com/w-joe/gcc/blob/master-ci/.cirrus.yml

That's great work and progress!


I just wanted to a side note:

If you manage to build tarballs of the binaries, I think a lot of people would greatly appreciate if they are made available (e.g. can be done directly on GitHub via "Releases").
For example, see https://github.com/dlang/installer/pull/251, https://forum.dlang.org/thread/xktompypwvaabwebnjol@forum.dlang.org, or https://forum.dlang.org/thread/bnkbldsifjhsseswiceq@forum.dlang.org .

If the download links are in the official install.sh script, then it will auto-magically be available on Travis CI and others.

I'm happy to help with getting such releases shipped.

On Tuesday, 15 September 2020 at 18:49:48 UTC, Iain Buclaw wrote:
> On Tuesday, 15 September 2020 at 16:05:53 UTC, wjoe wrote:
>> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>>> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
>>> However, on a closer look I can spot multiple failures or files/directories which can't be found.
>>> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
>>> - The build package step failed with: cd ./libcc1: no such file or directory.
>>
>> This is the package function for reference:
>>
>> build_package() {
>>     cd build || exit 1
>>     make install || exit 1
>>     tar -cJf gdc-${build_host}.txz /usr || exit 1
>> }
>>
>> Links for your convenience.
>> [1] https://github.com/w-joe/gcc/blob/master-ci/buildci.sh
>> [2] https://github.com/w-joe/gcc/blob/master-ci/.cirrus.yml
>
> Note the comments in the build function, only the dependencies of the C++ and D libraries are built. So you'll need to do `make all` in order to cover anything that was deliberately skipped.

Does that mean that building with build_bootstrap="enabled" is insufficient ?
Because I exported that variable in the Cirrus CI configuration and set it to 'disabled' for GDC and Unittest tasks and to 'enabled' for the Package task.

Packaging in the Unittest task didn't work unfortunately because building that with bootstrap enabled exceeded the 2h time limit.

So now there are 3 tasks.
- Build GDC testsuite
- Build Unittests
- Build Package task

At the moment I made the Package task depend on the Unittest task, but GDC and Unittest run in parallel.
I guess best practices would ask to make Unittest depend on GDC to go easy on resources.
However, that would blow up the build cycle to something over 3.5h.

The stats right now are:
- Docker container builds in less than 5 minutes
  It's automatically cached by CirrusCI i.e. this only applies if the Dockerfile changed.
- GDC testsuite takes about 45 minutes with build_bootstrap=disabled
- Unittests take a little more than 1 hour with build_bootstrap=disabled
- Package task took 1:45h with build_bootstrap=enabled
  That was 1:15h for building, and
  a little less than half an hour for the package step (make install && tar)

For a total of 2:46h

On Wednesday, 16 September 2020 at 00:49:52 UTC, Seb wrote:
> On Tuesday, 15 September 2020 at 16:05:53 UTC, wjoe wrote:
>> On Tuesday, 15 September 2020 at 16:00:56 UTC, wjoe wrote:
>>> I've added the above tasks and they are reported to have been completed successfully in the Cirrus summary.
>>> However, on a closer look I can spot multiple failures or files/directories which can't be found.
>>> - Some 12 failed tests as well as 10 unresolved test cases in the unittest step.
>>> - The build package step failed with: cd ./libcc1: no such file or directory.
>>
>> This is the package function for reference:
>>
>> build_package() {
>>     cd build || exit 1
>>     make install || exit 1
>>     tar -cJf gdc-${build_host}.txz /usr || exit 1
>> }
>>
>> Links for your convenience.
>> [1] https://github.com/w-joe/gcc/blob/master-ci/buildci.sh
>> [2] https://github.com/w-joe/gcc/blob/master-ci/.cirrus.yml
>
> That's great work and progress!
>
>
> I just wanted to a side note:
>
> If you manage to build tarballs of the binaries, I think a lot of people would greatly appreciate if they are made available (e.g. can be done directly on GitHub via "Releases").
> For example, see https://github.com/dlang/installer/pull/251, https://forum.dlang.org/thread/xktompypwvaabwebnjol@forum.dlang.org, or https://forum.dlang.org/thread/bnkbldsifjhsseswiceq@forum.dlang.org .
>

Thank you for your kind words :)

I do but I didn't preserve it in the last run. I've just added the Artifacts step to the cirrus configuration.
Artifacts is the terminology they use to make files from the build environment available for download after the task completed.

The way it's being done right now is that 'make install' installs to the /usr prefix. After that a tarball of this prefix is created (via tar cJf gdc-triplet.txz /usr). I'm not sure if that's suitable as a release as is because tar omits the root / so the result will be extracted as usr/
There isn't a lot of time budget left in that task but it should be possible to run some more scripts.
If the time limit won't suffice it should be possible to cache /usr and move the tar ball script into a new task.

Also, all of that is Linux only at the moment.
I've created a matrix for the Dockerfiles so all platforms that can run bash should be easy to add - simply adding the Dockerfile to the matrix and it-should-work(TM).

> If the download links are in the official install.sh script, then it will auto-magically be available on Travis CI and others.
>
> I'm happy to help with getting such releases shipped.

I couldn't find the install script in my 1 minute search in the download section. Could I inconvenience you to copy/paste the link, please ?
Thanks :)

On Wednesday, 16 September 2020 at 09:55:22 UTC, wjoe wrote:
> I couldn't find the install script in my 1 minute search in the download section. Could I inconvenience you to copy/paste the link, please ?
> Thanks :)

Found it. It was just one more click away :)

On Wednesday, 16 September 2020 at 09:33:05 UTC, wjoe wrote:
> On Tuesday, 15 September 2020 at 18:49:48 UTC, Iain Buclaw wrote:
>> On Tuesday, 15 September 2020 at 16:05:53 UTC, wjoe wrote:
>>> [...]
>>
>> Note the comments in the build function, only the dependencies of the C++ and D libraries are built. So you'll need to do `make all` in order to cover anything that was deliberately skipped.
>
> Does that mean that building with build_bootstrap="enabled" is insufficient ?
> Because I exported that variable in the Cirrus CI configuration and set it to 'disabled' for GDC and Unittest tasks and to 'enabled' for the Package task.
>

build_bootstrap=enabled should be OK.  Even better, it configures with --enable-checking=release.



> Packaging in the Unittest task didn't work unfortunately because building that with bootstrap enabled exceeded the 2h time limit.
>
> So now there are 3 tasks.
> - Build GDC testsuite
> - Build Unittests
> - Build Package task
>
> At the moment I made the Package task depend on the Unittest task, but GDC and Unittest run in parallel.
> I guess best practices would ask to make Unittest depend on GDC to go easy on resources.
> However, that would blow up the build cycle to something over 3.5h.
>

All tasks could be ran in parallel as there's no dependencies between them.  The Package task though I'd imagine should be reserved for the release branches, so it's only ran infrequently.

On Wednesday, 16 September 2020 at 09:57:21 UTC, wjoe wrote:
> On Wednesday, 16 September 2020 at 09:55:22 UTC, wjoe wrote:
>> I couldn't find the install script in my 1 minute search in the download section. Could I inconvenience you to copy/paste the link, please ?
>> Thanks :)
>
> Found it. It was just one more click away :)

It should be pulling it from the gdc site.  If we need the capacity, I have no problem with ordering a storage box to host all downloads (it would be nice to move the existing tarballs off the tiny VM anyway).

On Wednesday, 16 September 2020 at 09:55:22 UTC, wjoe wrote:
>
> The way it's being done right now is that 'make install' installs to the /usr prefix. After that a tarball of this prefix is created (via tar cJf gdc-triplet.txz /usr). I'm not sure if that's suitable as a release as is because tar omits the root / so the result will be extracted as usr/
> There isn't a lot of time budget left in that task but it should be possible to run some more scripts.
> If the time limit won't suffice it should be possible to cache /usr and move the tar ball script into a new task.
>

If it follows the convention of the existing packages, it should be fine.  e.g: tar extracts gdc into 'x86_64-unknown-linux-gnu/bin/gdc'

On Wednesday, 16 September 2020 at 10:23:16 UTC, Iain Buclaw wrote:
> On Wednesday, 16 September 2020 at 09:33:05 UTC, wjoe wrote:
>>[...]
>
> build_bootstrap=enabled should be OK.  Even better, it configures with --enable-checking=release.
>
>
>
>>[...]
>
> All tasks could be ran in parallel as there's no dependencies between them.  The Package task though I'd imagine should be reserved for the release branches, so it's only ran infrequently.

Configuration wise that's not a problem at all. My reasoning for the dependency was that there's no point in making a release tar ball if the Unittests task fails.