On Sunday, 11 February 2018 at 15:39:18 UTC, Pjotr Prins wrote:
> On Friday, 6 January 2017 at 13:39:31 UTC, fredmanglis wrote:
>> On Thursday, 5 January 2017 at 23:18:36 UTC, Joakim wrote:
>>> Seem fairly small, but I could see why you wouldn't want to maintain them yourself.  If you're up for it, we could probably add a version flag to the test suite to disable these sections for packagers.
>
> Similar to StdUnittest in https://github.com/dlang/phobos/pull/5927, I propose we
> add version(StdNetworkunittest) and/or version(StdEnvUnittest) for those tests that require internet access or access to a directory outside the source tree (e.g. $TMPDIR) as these may not be available in containers and other restrictive build systems. Does that make sense?

How about the reverse, s.t. not all build scripts need to be updated?

On Sunday, 11 February 2018 at 19:40:56 UTC, Seb wrote:
> How about the reverse, s.t. not all build scripts need to be updated?

The following would only be for tests that access the internet:

   version(StdEnvUnittest) {
       do something with internet enabled
   }

Not that many tests.

On Sunday, 11 February 2018 at 19:40:56 UTC, Seb wrote:
> How about the reverse, s.t. not all build scripts need to be updated?

Can we do something like

!version(ConstrainedEnvUnittest) {
   networking....
}

i.e., a negative? Or do we need to write

version(ConstrainedEnvUnittest) {
   // empty
} else {
   networking...
}

everywhere.

Maybe version is not the best way to do this.

On Wednesday, 4 January 2017 at 18:40:57 UTC, fredmanglis wrote:
> Hi.
>
> I'm packaging ldc-1.1.0-beta6 for guix, which does not allow for network access when building the package, leading to errors, and timeouts when the tests are run.
>
> Is there a way to deactivate such tests, say a flag passed to the build system or such? I would be grateful if you'd point me to one.
>
> In the meantime, I have had to make use of huge patches to deactivate the tests to make the builds pass, which does not seem sustainable

I am packaging ldc for NixOS and I was having the same problem there.
I solved it by putting the tests with special needs into a fixed output derivation.
That's how it is called in Nix world. Fixed output derivation's always produce the same output and are therefore allowed to access the network. They are also used to implement all kinds of fetchers to get the source archives from GitHub and the like.
guix sure has something equal because it basically is the same system. They even use parts of Nix.

Here you can take a look into the current package: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/ldc/default.nix

On Monday, 12 February 2018 at 21:19:35 UTC, Thomas Mader wrote:
> I am packaging ldc for NixOS and I was having the same problem there. I solved it by putting the tests with special needs into a fixed output derivation.
> That's how it is called in Nix world. Fixed output derivation's always produce the same output and are therefore allowed to access the network. They are also used to implement all kinds of fetchers to get the source archives from GitHub and the like.

That is an interesting approach and I will discuss it with others. So an output derivation is an output that gets generated and allows for running it after the installation phase - giving it an opportunity to run tests. It is an interesting approach and works because the store is 'immutable'.

Two downsides: (1) is you can not assume there is a network on the build machine. We need to be able to build and test packages in isolation on restricted computers to make sure no one can tamper. (2) it allows pulling software, as you state, which changes the state of test system and makes it potentially non-reproducible. I don't think the Guix folks thing that this is the way forward. Impurity is a big no.

> guix sure has something equal because it basically is the same system. They even use parts of Nix.

Currently Nix and Guix share the build daemon. Otherwise Nix and Guix are different animals altogether. I moved from Nix to Guix a few years back and know both systems intimately.

> Here you can take a look into the current package: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/ldc/default.nix

here is mine (almost done, I need to remove some lines):

    https://gitlab.com/genenetwork/guix/blob/dlang/gnu/packages/ldc.scm#L183

to me the difference is that (if you think away the LISP braces) Guix looks a lot more readable and the logic is clear from the expression (some of it is probably a matter of taste). I also note that the Nix package does not to build shared phobos libs and is not running all ldc tests.

But, admittedly, both package descriptions are short and clear.

On Friday, 16 February 2018 at 06:13:05 UTC, Pjotr Prins wrote:
> Two downsides: (1) is you can not assume there is a network on the build machine. We need to be able to build and test packages in isolation on restricted computers to make sure no one can tamper. (2) it allows pulling software, as you state, which changes the state of test system and makes it potentially non-reproducible. I don't think the Guix folks thing that this is the way forward. Impurity is a big no.

Do you mean phobos curl tests? They use a self-contained test server, should be pure.

On Friday, 16 February 2018 at 12:00:18 UTC, Kagamin wrote:
> On Friday, 16 February 2018 at 06:13:05 UTC, Pjotr Prins wrote:
>> Two downsides: (1) is you can not assume there is a network on the build machine. We need to be able to build and test packages in isolation on restricted computers to make sure no one can tamper. (2) it allows pulling software, as you state, which changes the state of test system and makes it potentially non-reproducible. I don't think the Guix folks thing that this is the way forward. Impurity is a big no.
>
> Do you mean phobos curl tests? They use a self-contained test server, should be pure.

Those are fine. Current patches to make the build work are

  https://gitlab.com/genenetwork/guix/blob/dlang/gnu/packages/patches/ldc-1.7.0-disable-phobos-tests.patch

Not too many!

Remaining problems are with assumptions about socket and root path. I'll submit some patches if I can make it work. The root path can probably be avoided by faking it into the build dir.

On Friday, 16 February 2018 at 06:13:05 UTC, Pjotr Prins wrote:
> That is an interesting approach and I will discuss it with others. So an output derivation is an output that gets generated and allows for running it after the installation phase - giving it an opportunity to run tests. It is an interesting approach and works because the store is 'immutable'.
>
> Two downsides: (1) is you can not assume there is a network on the build machine. We need to be able to build and test packages in isolation on restricted computers to make sure no one can tamper.

As Kagamin already noted, Phobos isn't running any network tests but dub is in it's tests.
But I don't see that as a problem. You always depend on the network for all package builds because you need to get the source from somewhere.
If you want to be able to build without Internet access you need to have it cached somewhere. Either on your machine or on a proxy. That's how it works. Is that different with guix?

> (2) it allows pulling software, as you state, which changes the state of test system and makes it potentially non-reproducible. I don't think the Guix folks thing that this is the way forward. Impurity is a big no.

As you already stated a fixed output derivation isn't changing the derivation which uses it because the output is the same all the time. What happens inside the derivation is of no concern to the using derivation as long as it runs through.

>> guix sure has something equal because it basically is the same system. They even use parts of Nix.
>
> Currently Nix and Guix share the build daemon. Otherwise Nix and Guix are different animals altogether. I moved from Nix to Guix a few years back and know both systems intimately.

I decided to go with Nix because the community seems bigger and it was the project with the initial idea for such a system.
I don't see me moving but would still be interested in your reasons.

> here is mine (almost done, I need to remove some lines):
>
>     https://gitlab.com/genenetwork/guix/blob/dlang/gnu/packages/ldc.scm#L183
>
> to me the difference is that (if you think away the LISP braces) Guix looks a lot more readable and the logic is clear from the expression (some of it is probably a matter of taste). I also note that the Nix package does not to build shared phobos libs and is not running all ldc tests.
>
> But, admittedly, both package descriptions are short and clear.

Sorry I can't see that. :-)
I knew that guix comes with it's own language and if I remember correct I read somewhere that the Nix project lead is not against something like that for Nix too but I am not sure if it is currently an active topic. But I don't think it is.
I don't care, for me it would be more important to have such a system for Windows too to be able to have one package manager for the major three OSes.

Does guix come with a nice command line interface? Nix used to have multiple commands but with Nix 2.0 all is unified via 'nix <subcommand>'.

On Friday, 16 February 2018 at 13:14:22 UTC, Pjotr Prins wrote:
> Those are fine. Current patches to make the build work are
>
>   https://gitlab.com/genenetwork/guix/blob/dlang/gnu/packages/patches/ldc-1.7.0-disable-phobos-tests.patch
>
> Not too many!
>
> Remaining problems are with assumptions about socket and root path. I'll submit some patches if I can make it work. The root path can probably be avoided by faking it into the build dir.

Solutions to phobos problems already went upstream.
Some of them should be getting out with the next major version release for dmd.
You could go through the package script for dmd, some of them have comments.

The ones I remember which are fixed: root/homedirectory, datetime problems with the path to the timezone db, curl path problems.

The remaining problems I am having seem to be Nix related.
So you shouldn't have other problems with guix than me with Nix. It might be a good idea if we follow each other on Github or something to prevent double work.

On Friday, 16 February 2018 at 06:13:05 UTC, Pjotr Prins wrote:
> I also note that the Nix package does not to build shared phobos libs and is not running all ldc tests.

Which ones are missing?

Why are the shared libs not built?