What is the motivation?

D package management on Gentoo doesn't exist de facto. There are spread out ebuilds (package manager scripts) for dmd and ldc and that's it. Orthogonal to that are projects like Jacob Carlborg's "dvm", which only handles dmd and Sönke Ludwig's "dub" which don't integrate with the official package managers.

When I decide to install a D application on Gentoo, I want it to just work. For Gentoo that includes the choice of compiler to build it and the used compilation flags.

The package manager should handle installing LLVM, the latest stable LDC and required libraries on top of that and finally the application. Just like it is for most other main stream languages right now. If you think this is a good thing to have, please read on...


---(The following is rather Gentoo specific)---

I tried to plot in my head what D support in Gentoo would ideally look like. Some corners I have found are:

- The makefiles are still not typical, although I don't have
  to patch them anymore since 2.062, which is positive.
- Compiler and library are closely coupled and depend on each
  other.
- Each new D point release brings a new, generally
  incompatible standard library.
- All three major compiler backends are in good shape and use.
  But their generated code is not exchangeable (as far as
  I've heard due to ABI differences).
- Tools tend to expect that "dmd" is available as a command.

To deal with the complexity of package dependency management in this scenario, we need to be able to install multiple versions of D compilers in parallel (similar to what dvm does). This can be accomplished by so called "slots" for each point release. We can do this for all three compilers and will end up with binaries like "dmd2.064" and "gdc2.063".

Next a symlink should be established to the current
implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
This symlink could be managed with an eselect plugin as it is
already done for other languages like Python or the Java
Runtime.

Since libraries for D depend not only on druntime/Phobos, but also on the compiler the Gentoo user should be able to setup globally and for each library which compilers it should work with. Again there is prior work on Gentoo for supporting a list of things (Apache modules, graphics cards, cameras, ...). The environment variables that are used for that usually go into /etc/make.conf: D_COMPILERS="dmd gdc ldc" and are mapped to regular "Use-Flags" for packages, i.e. d_compilers_dmd, d_compilers_gdc and d_compilers_ldc. (This step is already automated.)

The libraries could depend directly on an installation of the
respective compilers, but I think it would be cleaner to have
a "virtual/phobos" package, that is sensitive to the list of D
compilers as mentioned above.
One question I haven't finally answered yet is how to handle
libraries when there are multiple versions of D available.
For some programming languages this is solved by installing
libraries into <language_name>-<version>/ prefixes.
This certainly requires a more thoughtful dmd configuration
process. E.g. multiple dmd.conf for each installed version, to
point to the correct library directory. (A small patch to the
compiler sources during installation.)

Thoughs?

-- 
Marco

I think you're headed the right way, and I'd recommend studying how Ruby/rubygems are handled in Gentoo.  A similar pattern, with eselect, a set of symlinks, and versioned package directories, would do an awful lot.  One may end up with a mass of installed slots, but that sort of problem is usually resolved over time.

One thing of benefit, assuming it's being applied properly by all vendors, is the availability of std.compiler and a few predefined version identifiers so that D programs can already check what compiler and version they are being processed by.  This saves the trouble of trying to establish a USE flag convention for that.

In light of trying to support different compilers, it might be a good idea to have a virtual/dmd (or virtual/dc? or...?) package to provide dependency genericity.  I know you mentioned having a virtual/phobos package, but I'm not sure that's safe -- yet.  Once we are at a stage where dynamic libraries work well enough that phobos is normally used as such, then it would make more sense.

I'm falling asleep here, but before I go...

On Saturday, 9 November 2013 at 00:38:16 UTC, Marco Leise wrote:
>
> "dub" which don't integrate with the official package
> managers.
>
With the right functionality, we could do this similarly to how we hook CPAN.pm for g-cpan.  If nothing else, it could be done a damn sight more reliably than Diego's gem eclass if we get this squared away sooner than later.  (This is a weird space, though.)

> When I decide to install a D application on Gentoo, I want it
> to just work. For Gentoo that includes the choice of compiler
> to build it and the used compilation flags.
>
> The package manager should handle installing LLVM, the latest
> stable LDC and required libraries on top of that and finally
> the application.
>
For a long time, GDC was just a USE flag on GCC.  Not sure what the current state is because I've been trying to ignore the sad state of our D packaging in hopes that it would go away.  (How many D overlays have we killed at this point anyway?)

> - Compiler and library are closely coupled and depend on each
>   other.

So we'll need a virtual for "the D implementation"?  cf. virtual/package-manager and dozens of others.

> - Each new D point release brings a new, generally
>   incompatible standard library.

Unless I've misunderstood completely, the compiler, runtime, and standard library are ostensibly separable, but for the foreseeable future they need to be revbumped in lockstep for this reason.  No biggie.  cf. kde-base/*

> - All three major compiler backends are in good shape and use.
>   But their generated code is not exchangeable (as far as
>   I've heard due to ABI differences).

Do they? I thought this had been worked out.

> - Tools tend to expect that "dmd" is available as a command.
>
Which tools?  Quantify the problem, and we may be able to fix it. (We have like all of upstream right here...;)

> To deal with the complexity of package dependency management
> in this scenario, we need to be able to install multiple
> versions of D compilers in parallel (similar to what dvm does).

Do we really?  Not necessarily disagreeing, but I think this needs substantiation.

> This can be accomplished by so called "slots" for each point
> release. We can do this for all three compilers and will end
> up with binaries like "dmd2.064" and "gdc2.063".
>
See, I'd like to avoid the awful version creep we see with GCC-- that only exists because we're trying to provide an upgrade path for lazy people who never upgrade (never mind that it's never really worked well).  Thankfully RAP may finally be fixing that, now that Benda is joining the toolchain herd.

> Next a symlink should be established to the current
> implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
> This symlink could be managed with an eselect plugin as it is
> already done for other languages like Python or the Java
> Runtime.
>
I'm so not fond of those, but it looks like we may not have a choice.  Do we have any use for an env.d entry here?

> Since libraries for D depend not only on druntime/Phobos, but
> also on the compiler the Gentoo user should be able to setup
> globally and for each library which compilers it should work
> with.

It's way too early for this shade of bikeshedding.  Let's see about actually getting packages in the tree, first.  (The approach may or may not be a good fit.  I think we should give it some time to settle first, though.)

-Wyatt

Some comments on general packaging topic.

On Saturday, 9 November 2013 at 00:38:16 UTC, Marco Leise wrote:
> - Compiler and library are closely coupled and depend on each
>   other.

Not really true, it is straightforward to write an alternative druntime / stdlib substitution to use with existing compiler package.

> - Each new D point release brings a new, generally
>   incompatible standard library.
> - All three major compiler backends are in good shape and use.
>   But their generated code is not exchangeable (as far as
>   I've heard due to ABI differences).

Yes, ABI is the key issue that makes 3 major compiler ecosystems not interchangeable in any way. And it is not likely to be fixed any time soon.

> - Tools tend to expect that "dmd" is available as a command.

Actually I don't think this is true for newer ones. There are plenty that tend to either use `rdmd` or detect compiler present in the system.

> To deal with the complexity of package dependency management
> in this scenario, we need to be able to install multiple
> versions of D compilers in parallel (similar to what dvm does).
> This can be accomplished by so called "slots" for each point
> release. We can do this for all three compilers and will end
> up with binaries like "dmd2.064" and "gdc2.063".

I have been thinking about it when doing Arch packaging but discarded that approach as not KISS. All maintained applications support either latest compiler version or previous one. Rare cases when legacy version is needed are covered by dvm. In that sense pure bleeding edge model suits D realities most and only separation between dmd/ldc/gdc is needed.

> Next a symlink should be established to the current
> implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
> This symlink could be managed with an eselect plugin as it is
> already done for other languages like Python or the Java
> Runtime.

And what if one wants to have all 3 simultaneously?

> ...

(rest is a bit too Gentoo-specific for me to make any reasonable input :))

Am Sat, 09 Nov 2013 08:42:59 +0100
schrieb "Chris Nicholson-Sauls" <ibisbasenji@gmail.com>:

> I think you're headed the right way, and I'd recommend studying how Ruby/rubygems are handled in Gentoo.  A similar pattern, with eselect, a set of symlinks, and versioned package directories, would do an awful lot.  One may end up with a mass of installed slots, but that sort of problem is usually resolved over time.

Quite a few languages use similar schemes. Even "equery f gcc"
can server as a guide. But reusing the gcc ebuild is not that
simple. I can't easily remove the other language USE flags from
gcc for gdc and the build process takes an awful long time.
On the positive side, it was relatively easy to "mixin" the
GDC code from Github though from a specific branch and commit.

> One thing of benefit, assuming it's being applied properly by all vendors, is the availability of std.compiler and a few predefined version identifiers so that D programs can already check what compiler and version they are being processed by.  This saves the trouble of trying to establish a USE flag convention for that.

The .d files can, but the build tools need an environment variable (lets call it DC) to find the active D compiler. You can change the C compiler for Clang or ICC on Gentoo. This scheme should be adopted for D, too.

This is what really worries me:
Assuming my current compiler (through eselect or other means)
is dmd-2.063. Now I emerge "apps-office/cool-d-application"
which uses GtkD. That means that I have to establish a
dependency on GtkD compiled with dmd-2.063. How would I do
that without naively introducing one use flag per compiler and
release?
You said, look at Ruby and it ebuilds do exactly that:
USE_RUBY="ruby18 ruby19 ree18 jruby"
A list of language implementations and versions.

> In light of trying to support different compilers, it might be a good idea to have a virtual/dmd (or virtual/dc? or...?) package to provide dependency genericity.  I know you mentioned having a virtual/phobos package, but I'm not sure that's safe -- yet. Once we are at a stage where dynamic libraries work well enough that phobos is normally used as such, then it would make more sense.

It actually is a virtual/dc under the hood. The idea was that it is the Phobos API that changes often, not the compiler ABI. I would name it virtual/dlang for what it's worth. :D

-- 
Marco

Am Sat, 09 Nov 2013 08:46:35 +0100
schrieb "Wyatt" <wyatt.epp@gmail.com>:

> I'm falling asleep here, but before I go...
> 
> On Saturday, 9 November 2013 at 00:38:16 UTC, Marco Leise wrote:
> >
> > "dub" which don't integrate with the official package managers.
> >
> With the right functionality, we could do this similarly to how we hook CPAN.pm for g-cpan.  If nothing else, it could be done a damn sight more reliably than Diego's gem eclass if we get this squared away sooner than later.  (This is a weird space, though.)

That sounds like an option. It might require some extensions to dub though so we can have a library installed multiple times for different compilers and D versions.

> > When I decide to install a D application on Gentoo, I want it to just work. For Gentoo that includes the choice of compiler to build it and the used compilation flags.
> >
> > The package manager should handle installing LLVM, the latest stable LDC and required libraries on top of that and finally the application.
> >
> For a long time, GDC was just a USE flag on GCC.  Not sure what the current state is because I've been trying to ignore the sad state of our D packaging in hopes that it would go away.  (How many D overlays have we killed at this point anyway?)

There is a toolchain.eclass that creates the complete list of IUSE for GCC, excluding 'd' for now. But without the front-end source code in the source package it doesn't do anything useful anyway.

> > - Compiler and library are closely coupled and depend on each
> >   other.
> 
> So we'll need a virtual for "the D implementation"?  cf. virtual/package-manager and dozens of others.

Yes, I had virtual/phobos in mind, but you guys a probably right that it should be more general like virtual/dlang.

> > - Each new D point release brings a new, generally
> >   incompatible standard library.
> 
> Unless I've misunderstood completely, the compiler, runtime, and standard library are ostensibly separable, but for the foreseeable future they need to be revbumped in lockstep for this reason.  No biggie.  cf. kde-base/*

I wouldn't separate Phobos out just yet. One package per D implementation will do the job.

> > - All three major compiler backends are in good shape and use.
> >   But their generated code is not exchangeable (as far as
> >   I've heard due to ABI differences).
> 
> Do they? I thought this had been worked out.

I asked Mr. Buclaw recently and he told me that is was still an issue, at least when it comes to dmd.

> > - Tools tend to expect that "dmd" is available as a command.
> >
> Which tools?  Quantify the problem, and we may be able to fix it. (We have like all of upstream right here...;)

Hehe, I can't really quantify it, but assumed that "dmd" is the default for rdmd, dub, Mono-D and other tools. For many you can override the compiler name, but it could still be helpful to have /bin/dmd point to /bin/ldmd2-0.12.0 if LDC is the only installed D compiler.

> > To deal with the complexity of package dependency management in this scenario, we need to be able to install multiple versions of D compilers in parallel (similar to what dvm does).
> 
> Do we really?  Not necessarily disagreeing, but I think this needs substantiation.

Ok, fair enough... We have D compilers that excel at different
things. A big selling point is the rapid turn around time of
code changes when compiling with dmd, but its optimizer is
lacking on modern CPUs. My choice is to setup my debug builds
to use dmd and the release builds to use gdc/ldc. Also
sometimes one compiler has a corner case with some code and it
helps to try another one till a bug fix is available. (So much
for the value of multiple compilers, on to multiple
versions...)
Some projects are still written in D1. And with the Phobos
library as an .so now, we have to see how we can keep multiple
versions of Phobos in case some application isn't updated for
half a year and happens to need that last Phobos version.

> > This can be accomplished by so called "slots" for each point release. We can do this for all three compilers and will end up with binaries like "dmd2.064" and "gdc2.063".
> >
> See, I'd like to avoid the awful version creep we see with GCC-- that only exists because we're trying to provide an upgrade path for lazy people who never upgrade (never mind that it's never really worked well).  Thankfully RAP may finally be fixing that, now that Benda is joining the toolchain herd.

Some time you need to give me a who-is-who in Gentoo. Ok, just what do you think is tolerable? gcc has quite a few versions, but it doesn't really affect me. I have only 4.7.3-r1 installed. For D the story is different since the compiler and standard library ship together and - from a packager's point of view - it is all new every 2 to 5 months. If you have some D programs installed that haven't been updated in a while you have to keep old versions of phobos.so and other dependencies around.

> > Next a symlink should be established to the current
> > implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
> > This symlink could be managed with an eselect plugin as it is
> > already done for other languages like Python or the Java
> > Runtime.
> >
> I'm so not fond of those, but it looks like we may not have a choice.  Do we have any use for an env.d entry here?

I have never looked at those in depth. Possibly yes, since there is Python and GCC already.

> > Since libraries for D depend not only on druntime/Phobos, but also on the compiler the Gentoo user should be able to setup globally and for each library which compilers it should work with.
> 
> It's way too early for this shade of bikeshedding.  Let's see about actually getting packages in the tree, first.  (The approach may or may not be a good fit.  I think we should give it some time to settle first, though.)

Hmm. This is actually my main motivation: Getting packages like Derelict or GtkD working with both dmd for debug builds and gdc for release builds without manual recompilation, changing lib paths and library names, just by setting D_COMPILERS="dmd gdc" in make.conf.

> -Wyatt

Thanks for your input. It is nice to talk about our options here and to hear from few more D users on Gentoo that way.

-- 
Marco

Am Sat, 09 Nov 2013 12:21:30 +0100
schrieb "Dicebot" <public@dicebot.lv>:

> Some comments on general packaging topic.
> 
> On Saturday, 9 November 2013 at 00:38:16 UTC, Marco Leise wrote:
> > - Compiler and library are closely coupled and depend on each
> >   other.
> 
> Not really true, it is straightforward to write an alternative druntime / stdlib substitution to use with existing compiler package.

Yeah, I was pointing at the lock-step upgrade nature of D development. You can't really update one without the other when it comes to every day package management.

> > - Each new D point release brings a new, generally
> >   incompatible standard library.
> > - All three major compiler backends are in good shape and use.
> >   But their generated code is not exchangeable (as far as
> >   I've heard due to ABI differences).
> 
> Yes, ABI is the key issue that makes 3 major compiler ecosystems not interchangeable in any way. And it is not likely to be fixed any time soon.
> 
> > - Tools tend to expect that "dmd" is available as a command.
> 
> Actually I don't think this is true for newer ones. There are plenty that tend to either use `rdmd` or detect compiler present in the system.

Good, the only thing that might go wrong then is alternate binary names or install locations.

> > To deal with the complexity of package dependency management in this scenario, we need to be able to install multiple versions of D compilers in parallel (similar to what dvm does). This can be accomplished by so called "slots" for each point release. We can do this for all three compilers and will end up with binaries like "dmd2.064" and "gdc2.063".
> 
> I have been thinking about it when doing Arch packaging but discarded that approach as not KISS. All maintained applications support either latest compiler version or previous one. Rare cases when legacy version is needed are covered by dvm. In that sense pure bleeding edge model suits D realities most and only separation between dmd/ldc/gdc is needed.

Your experience is welcome! Does D1 have a place in that model? And what does really happen when you work with e.g. dmd-2.065 and found a D application in the tree that hasn't been updated for a few weeks, so it still requires 2.064? It would more or less mean that the whole D package system would have to be updated in one go, when all packages are updated to the latest D version, which can take a while.

> > Next a symlink should be established to the current
> > implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
> > This symlink could be managed with an eselect plugin as it is
> > already done for other languages like Python or the Java
> > Runtime.
> 
> And what if one wants to have all 3 simultaneously?

You call them by their real names. :)

-- 
Marco

On 2013-11-09 01:38, Marco Leise wrote:

> Next a symlink should be established to the current
> implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.

Wouldn't it be better to have a symlink called "dc", similar to "cc" for the C compiler. But that would require the DMD release to included a "dc" symlink and have it available on all platforms. And also people start using that instead of directly using dmd.

On the other hand, it would be better to hide these things behind a tool like dub.

-- 
/Jacob Carlborg

Am Sat, 09 Nov 2013 14:38:48 +0100
schrieb Jacob Carlborg <doob@me.com>:

> On 2013-11-09 01:38, Marco Leise wrote:
> 
> > Next a symlink should be established to the current implementation of dmd, e.g.: /bin/dmd -> /bin/ldmd2.
> 
> Wouldn't it be better to have a symlink called "dc", similar to "cc" for the C compiler. But that would require the DMD release to included a "dc" symlink and have it available on all platforms. And also people start using that instead of directly using dmd.
> 
> On the other hand, it would be better to hide these things behind a tool like dub.

"dmd" and it's ilk (gdmd, ldmd2) have a known command line
syntax, "dc" doesn't. Symlinks are most useful when the
semantic stays the same. For example you can have "sh" be a
link to "bash" since bash recognizes being invoked as "sh" and
emulates it.
FWIW, DC as an environment variable across systems might be a
good thing. You can have that on Windows, too. Looking at the
file part of the path you could tell if it is ldc2, dmd, gdc
or rdmd...

-- 
Marco

On 2013-11-09 14:52, Marco Leise wrote:

> "dmd" and it's ilk (gdmd, ldmd2) have a known command line
> syntax, "dc" doesn't. Symlinks are most useful when the
> semantic stays the same.

I might have been a bit unclear. I was thinking we had a symlink called "dc" that would point to dmd, gdmd or ldmd2. I don't like that you can invoke "dmd" and it will compile using GDC.

-- 
/Jacob Carlborg