Hello all,

I decided that it was time to try creating snap packages of various core D programs, and I thought I'd start with LDC since its very standard cmake-based build and install makes for super-easy integration with the snapcraft package-creation system.

The TL;DR is that creating the _package_ is super-easy, but there are a handful of interesting integration issues that it would be good to have your advice on.

For cross-reference, here's the discussion on the snapcraft mailing list where I raise the issues I encountered:
https://lists.ubuntu.com/archives/snapcraft/2016-August/000860.html

... and I'm writing the current email in the wake of feedback received there.  Anyway, first let me describe what I've done, before raising any questions.

The basics of how to create an LDC snap package are super easy.  Simply create a file called `snapcraft.yaml` in an empty directory, containing the following contents:

-----------------------------------------
name: ldc2
version: "1.0.0"
summary: D compiler with LLVM backend
description: LDC is a compiler for the D programming Language.
             It is based on the latest DMD frontend and uses LLVM as backend.
confinement: strict

apps:
  ldc2:
    command: ldc2
    plugs: [home]
  ldmd2:
    command: ldmd2
    plugs: [home]

parts:
  ldc:
    plugin: cmake
    source: git://github.com/ldc-developers/ldc.git
    source-tag: v1.0.0
    build-packages:
    - ldc
    - llvm-dev
    - libconfig++-dev
    - libcurl4-gnutls-dev
    - libedit-dev
    - zlib1g-dev
-----------------------------------------

As you can see, this metadata describes the apps to be included in the snap package, together with the build plugin required (cmake is supported out of the box), and where to get the source (git is again out-of-the-box supported and you simply need to specify the version tag to check out).

Note the listing of `ldc` as a build dependency: this will use the Ubuntu-packaged ldc 0.17.1.  Note also the ["home"] plugs: this ensures that the applications in the containerized snap will still be able to access the user's home directory (pretty important if you want to build stuff).

Given all this, on any Ubuntu 16.04 system, from the same directory as the `snapcraft.yaml` file, just run:

    snapcraft

The snapcraft program will install any necessary build dependencies, check out the code, build it, and install the results into a `stage` directory.  This is then further processed to create a `prime` directory creating the actual contents that will be in the snap package; and finally, the snap package itself is generated.  Assuming you're running this on a 64-bit system, it will be named something like:

    ldc2_1.0.0_amd64.snap

This can then be installed (on Ubuntu 16.04) with:

    sudo snap install ldc2_1.0.0_amd64.snap

Snaps are installed (read-only) into the `/snap/` top-level folder; in this case, you can find the contents in `/snap/ldc2/current/` (`current` is a symlink to the particular snap that is currently in use).  Everything you'd expect to find in a normal LDC install will be in there: the `bin/` directory containing `ldc2` and `ldmd2`; the `etc/` directory containing `bash_completion.d` and `ldc2.conf`; the `include/` dir with the usual contents; the `lib/` dir containing static libraries of druntime and phobos (with and without debug symbols); and so on.

You should now be able to run `ldc2 --version` and see the appropriate info.

So far so good!  Unfortunately, now we have to start dealing with the integration issues.

One issue is to do with how the commands are exposed: the actual commands you can run will be called `ldc2` and `ldc2.ldmd2` (as snappy effectively 'namespaces' apps that do not share a name with the snap package).  This is solvable but will require some updates to `snapd` itself, so no need to discuss, just to be aware that this is the current situation.

Next, the `ldc2.conf` file's -I and -L path definitions are problematic, possibly because of the containerized way that snapcraft builds the snap:

    switches = [
        "-I/include/d/ldc",
        "-I/include/d",
        "-L-L/lib",
        "-defaultlib=phobos2-ldc,druntime-ldc",
        "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
    ];

This leads me to my first question: is it possible to specify the paths here relative to the location of the directory where the `ldc2` binary resides?  I tried using the dmd-style:

    -I%@P%/../include/d/ldc

... etc., but that didn't seem to work (I don't know if is %@P% in any way standard, or whether it's something dmd-specific...?).

Manually specifying the exact path on the command line (or manually editing the generated files before finalizing the snap package) can be used to work around this.

Finally, the snap-packaged LDC will currently fail at the linker stage, because (as I understand it) the snap package can't access regular system resources, only stuff that is in other snaps, and the core system snap doesn't include stuff like gcc.  So, building falls over with an error:

    Error: failed to locate gcc

There are a few long-term potential fixes here, including creating a snap that exposes GCC or a linker, but in the short term, I've been advised that probably the easiest way to create a reliable standalone LDC package is to bundle the linker into the snap.

So, my question is, what would be the minimum requirements for a linker to accompany ldc2, and how would I ensure that the built ldc2 tries to use _that_ linker rather than some other one?

Thanks & best wishes,

     -- Joe

On Sunday, 28 August 2016 at 11:58:00 UTC, Joseph Rushton Wakeling wrote:
> This leads me to my first question: is it possible to specify the paths here relative to the location of the directory where the `ldc2` binary resides?

AFAIK contents of config file "switches" variable in ldc is simply prepended to command-line flag list with no special processing.

On Sunday, 28 August 2016 at 12:37:17 UTC, Dicebot wrote:
> On Sunday, 28 August 2016 at 11:58:00 UTC, Joseph Rushton Wakeling wrote:
>> This leads me to my first question: is it possible to specify the paths here relative to the location of the directory where the `ldc2` binary resides?
>
> AFAIK contents of config file "switches" variable in ldc is simply prepended to command-line flag list with no special processing.

Yes, that I understand.  The problem is that the contents of that file are wrong ;-)  I can edit it part way through the snap packaging process, but what I'd really like is an automatable solution.

The issue here is that, as I understand it, a snap package can wind up being installed in

/snap/snapname/current/

or /home/<user>/snap/snapname/current/

... so the precise path can't be known at package creation time but only at install time.

However, if the paths in `ldc2.conf` could be specified relative to the path where the `ldc2` binary lives, that might provide an easy workaround.

On Sunday, 28 August 2016 at 12:41:44 UTC, Joseph Rushton Wakeling wrote:
> On Sunday, 28 August 2016 at 12:37:17 UTC, Dicebot wrote:
>> On Sunday, 28 August 2016 at 11:58:00 UTC, Joseph Rushton Wakeling wrote:
>>> This leads me to my first question: is it possible to specify the paths here relative to the location of the directory where the `ldc2` binary resides?
>>
>> AFAIK contents of config file "switches" variable in ldc is simply prepended to command-line flag list with no special processing.
>
> Yes, that I understand.  The problem is that the contents of that file are wrong ;-)  I can edit it part way through the snap packaging process, but what I'd really like is an automatable solution.

Providing correct ldc2.conf is normally a responsibility of a packager for the target system thus it doesn't seem wrong to me. You may want to check how the config file is defined in tarball provided by github releases though as I suppose it is intended to be usable without any installation.

On Sunday, 28 August 2016 at 12:41:44 UTC, Joseph Rushton Wakeling wrote:
> However, if the paths in `ldc2.conf` could be specified relative to the path where the `ldc2` binary lives, that might provide an easy workaround.

To be more precise: this is exactly what dmd uses in its dmd.conf file, e.g.:

[Environment32]
DFLAGS=-I%@P%/../../src/phobos -I%@P%/../../src/druntime/import -L-L%@P%/../lib32 -L--export-dynamic

[Environment64]
DFLAGS=-I%@P%/../../src/phobos -I%@P%/../../src/druntime/import -L-L%@P%/../lib64 -L--export-dynamic

However, would I be right in assuming that %@P% is a dmd-specific wildcard that the compiler knows to replace with the compiler's path location?

I ask because I've tried using

  -I%@P%/../include/d/ldc -I%@P%/../include/d

manually on the command line with ldc2 and ldmd2 and it doesn't work, whereas manually specifying the _full_ path does.

Is there any way of providing a similarly relative-to-compiler-path -I option to ldc2 and/or ldmd2 ... ?

On Sunday, 28 August 2016 at 12:48:55 UTC, Dicebot wrote:
> Providing correct ldc2.conf is normally a responsibility of a packager for the target system thus it doesn't seem wrong to me.

Well, yes... and hence these questions, because as the packager I'm trying to understand my options, so that I _can_ take responsibility for what's in the package being correct. ;-)

Note the constraint that while the path relative to the snap install location is well defined, the absolute install path isn't.

> You may want to check how the config file is defined in tarball provided by github releases though as I suppose it is intended to be usable without any installation.

Hmm, interesting. The ldc2.conf file includes what looks like a wildcard for the compiler location, but passing that wildcard manually via -I flags doesn't work. I'll re-check later (had to step out for a bit) but it looks like it might be intended to be manually replaced.

On Sunday, 28 August 2016 at 12:56:17 UTC, Joseph Rushton Wakeling wrote:
> Is there any way of providing a similarly relative-to-compiler-path -I option to ldc2 and/or ldmd2 ... ?

Of course; the Windows package is already portable.
`%%ldcbinarypath%%` is replaced by the directory containing the ldc2 executable. So `-I%%ldcbinarypath%%/../include/d` etc. is the way to go.

On Sunday, 28 August 2016 at 13:28:07 UTC, Joseph Rushton Wakeling wrote:
> Hmm, interesting. The ldc2.conf file includes what looks like a wildcard for the compiler location, but passing that wildcard manually via -I flags doesn't work. I'll re-check later (had to step out for a bit) but it looks like it might be intended to be manually replaced.

I think we only replace that special placeholder in the ldc2.conf contents, not generally in the command line.

On Sunday, 28 August 2016 at 13:31:42 UTC, kinke wrote:
> On Sunday, 28 August 2016 at 13:28:07 UTC, Joseph Rushton Wakeling wrote:
>> Hmm, interesting. The ldc2.conf file includes what looks like a wildcard for the compiler location, but passing that wildcard manually via -I flags doesn't work. I'll re-check later (had to step out for a bit) but it looks like it might be intended to be manually replaced.
>
> I think we only replace that special placeholder in the ldc2.conf contents, not generally in the command line.

OK, cool. I'm out for the rest of the afternoon but I'll try to tweak the package this way later this evening & report back how it goes.

Meanwhile, thanks for the help!

On 28 Aug 2016, at 14:28, kinke via digitalmars-d-ldc wrote:
> On Sunday, 28 August 2016 at 12:56:17 UTC, Joseph Rushton Wakeling wrote:
>> Is there any way of providing a similarly relative-to-compiler-path -I option to ldc2 and/or ldmd2 ... ?
>
> Of course; the Windows package is already portable.

Not only the Windows packages, by the way – binary releases were portable pretty much ever since I first set them up in their current form.

 — David