Hello!
Me bringing dub to Debian (and subsequently Ubuntu) has sparked quite some interest in getting more D applications shipped in Linux distributions.
Since I think D is a great language, I would welcome that - in order to get more D code into distributions though, it would be awesome to sort out a few issues which currently (seem to) exist.
So I created this list, first to raise a bit of awareness on the issues that I see, and second - because I am very new to D - to allow others to point out solutions to those problems, or explain a different view on why some things are (not) done in a certain way (yet).
While I am thinking from a Debian point of view, quite some stuff is relevant for other distros as well.
So, let's get started:

## How complete are the free compilers?
This is an important question, because we would need to know whether we can expect D code to be compiled by any compiler, or whether there are tradeoffs that must be made.
This question is asking manly how complete LDC and GDC are compared to each other, but also whether both are implementing the D2 specification completely.
The question here is also, which compiler should be the default (which IMHO would be the most complete, most bug-free actively maintained one ^^).

## Why is every D compiler shipping an own version of Phobos?
If one assumes that the D2 language specification is stable, and D compilers implement it completely, the question is why every compiler is shipping an own copy of Phobos. This is a major pain, since e.g. GDCs Phobos is behind what is documented on dlang.org, but also because the compilers sometimes accidentally use the "wrong" Phobos version (e.g. GDC trying to use the one of LDC), leading to breakage.
Also, distributors hate code duplication, so deduplicating this would be awesome (druntime being compiler-specific makes sense to me, Phobos not so much...).
It's not an essential thing, but would be quite nice...

## Where should D source-code / D interfaces be put?
If I install a D library or source-only module as a distribution package, where should the sources be put? So far, I have seen:
 * /usr/include/d
 * /usr/include/dlang/
 * /usr/include/d/(dmd|gdc|ldc)
 * /usr/share/dlang
Having one canonical path would be awesome ;-)

## dub: Where should dub modules be installed?
DUB currently only searches online for it's packages, but if we install them as distributor, we want dub to also look locally in some path where the packages can be found, and - if possible - use a prebuilt shared/static library.
(reported as https://github.com/dlang/dub/issues/811 )

## dub: How to install modules?
Ideally, dub would also have a "dub install" command, to install the binary, "headers (.di)"/sources and data into standard directories on Linux.
(reported as https://github.com/dlang/dub/issues/811 )

++++++
Aside from these things, there are also some other things which would be very useful:

## Shared library support in all compilers
In distributions, we hate duplicating binary code copies. At time, D links everything statically, which means that when a bug is discovered in a library used by many other tools (worst case: the standard library), we will need to recompile all depending software.
This is really annoying for the security team, while not being as bad as having actual duplicate source-code copies which need to be patched individually.

## Stable ABI among compilers
Simple: Compile your library with GDC, have it used by code compiled with LDC. Otherwise, shared libraries would be of very limited use.

Cheers,
    Matthias

On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
> ## How complete are the free compilers?
> ## Why is every D compiler shipping an own version of Phobos?

These two can be answered at once. LDC and GDC share the same frontend code as DMD, but not the glue layer and backend (don't worry, it's only the DMD backend that has the more restrictive licence). Language and library development happens in DMD and with regard to DMD's current capabilities. Changes to DMD and druntime often require effort to port to LDC and GDC, due to the different backends. So an LDC or GDC release is complete w.r.t. a given past DMD version, but new features may have been added in more recent DMDs.

The constraints on shipping a shared phobos between them:
ABI compatibility: we don't have it, even across compiler versions
It would hold back phobos development (e.g. "you can't do that, because GDC doesn't support it yet")
You would still need to ship a separate runtime for each compiler, so you don't really gain anything.

On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
> 
> The question here is also, which compiler should be the default (which IMHO would be the most complete, most bug-free actively maintained one ^^).

Is performance of the outputted code a criterium? Or the number of supported architectures?

On Thursday, 21 April 2016 at 08:30:59 UTC, John Colvin wrote:
> On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
>> ## How complete are the free compilers?
>> ## Why is every D compiler shipping an own version of Phobos?
> The constraints on shipping a shared phobos between them:
> ABI compatibility: we don't have it, even across compiler versions
> It would hold back phobos development (e.g. "you can't do that, because GDC doesn't support it yet")
> You would still need to ship a separate runtime for each compiler, so you don't really gain anything.

Once we have full ABI compatibility we need to make sure we can work with a shared druntime: If an application compiled with compiler A(requiring runtime A) should be able to link with a shared library compiled by compiler B (requiring runtime B) both 'modules' need to use the same runtime => runtime A==B. This is because we can't have two GCs running concurrently (and there's some more low-level stuff).

What we need is a defined ABI for druntime. Then one druntime implementation is installed system wide. The implementation can differ (e.g. GDC might use GCC builtins) but the ABI needs to be fixed to make sure every compiler can use the 'system' druntime. Think of libc or libc++ for example: You can't reliably link modules from compilers using different libcs (or you end up with GC proxies and similar hacks).

Compiler specific extensions (gcc.builtins, etc) then must be moved to extra libraries (libgdc) (only if they affect the ABI though. gcc.builtins is actually a bad example as it doesn't contain non-extern functions).

On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
> ## Where should D source-code / D interfaces be put?
> If I install a D library or source-only module as a distribution package, where should the sources be put? So far, I have seen:
>  * /usr/include/d
>  * /usr/include/dlang/
>  * /usr/include/d/(dmd|gdc|ldc)
>  * /usr/share/dlang
> Having one canonical path would be awesome ;-)

/usr/include/d or /usr/include/dlang. I prefer /usr/include/d as it's shorter, but the important point is that we need to decide on a common standard.

You currently can't install druntime or phobos headers in this directory, as each compiler will have slightly modified versions and then you end up with /usr/include/d/(dmd|gdc|ldc). But all other library headers should be shareable between compilers and can be placed in /usr/include/d. (This might even work for phobos, I don't think we have many compiler specific changes there. But then you need to use the same frontend version for all compilers.) You can only install headers for one library version with this approach! A versioned approach is nicer /usr/include/d/libfoo/1.0.0 but requires explicit compiler support and it's unlikely this will happen (or explicit dub support and you compile everything through dub).

> ## dub: Where should dub modules be installed?

What does FHS recommend in this case? If you only keep headers/sources you could install into /usr/include. Otherwise you probably need /var/cache or /var/lib or somehting like that. If you split packages you could use the standard /usr/lib* folders but then you need to keep versioned subdirectories to support installing multiple versions.


> ## dub: How to install modules?
> Ideally, dub would also have a "dub install" command, to install the binary, "headers (.di)"/sources and data into standard directories on Linux.
> (reported as https://github.com/dlang/dub/issues/811 )

See above. The main question is: Do you want to have a C style install of one version of a library and have gdc find the includes and library when running gdc main.a -libfoo or do you want dub-style support for multiple library versions which means you need to compile everything through dub.

I'd love to have some extended compiler support (so you could simply do gdc -use=libfoo:1.0.0 and this would pick up the correct headers and linker flags). But as some DMD maintainers are opposed to this idea it won't happen. You'll probably always need dub for a 'nice' user interface.

> ++++++
> Aside from these things, there are also some other things which would be very useful:
>
> ## Shared library support in all compilers

This is mainly a GDC issue. DMD and LDC support shared libs, although I don't think these libraries are ABI compatible.

On Thursday, 21 April 2016 at 11:49:13 UTC, Johannes Pfau wrote:
> I'd love to have some extended compiler support (so you could simply do gdc -use=libfoo:1.0.0 and this would pick up the correct headers and linker flags). But as some DMD maintainers are opposed to this idea it won't happen. You'll probably always need dub for a 'nice' user interface.
>

We could probably also provide a small wrapper tool for this:
pkgdc gdc -use=libfoo:1.0.0 -other -gdc-args
pkgdc dmd -use=libfoo:1.0.0 -other -dmd-args

looks up package flags and calls the real compiler with appropriate args. We'd need to define standard for this to be really useful though.

On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
> ## How complete are the free compilers?
> This is an important question, because we would need to know whether we can expect D code to be compiled by any compiler, or whether there are tradeoffs that must be made.
> This question is asking manly how complete LDC and GDC are compared to each other, but also whether both are implementing the D2 specification completely.
> The question here is also, which compiler should be the default (which IMHO would be the most complete, most bug-free actively maintained one ^^).

Many D users are enthusiasts and push the compiler to its limits, they are usually stuck with DMD (even DMD HEAD sometimes) as it provides the latest fixes. It depends on the coding style, AppStream generator is an example of old good plain business logic one is unlikely to need recent frontend for, but for people writing black magic code a couple of versions lag of free compilers behind DMD is usually a blocker, but whoever uses the free compilers already considers them complete enough for their tasks. Currently LDC looks like the most actively maintained one.

Hi, and thanks for your detailed explanations!

On Thursday, 21 April 2016 at 11:49:13 UTC, Johannes Pfau wrote:
> On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
>> [...]
>
> You currently can't install druntime or phobos headers in this directory, as each compiler will have slightly modified versions and then you end up with /usr/include/d/(dmd|gdc|ldc).

That doesn't seem to be the case for LDC on Debian... It installs Phobos into /usr/include/d/std, which makes GDC go crazy as soon as LDC is installed too.
I suspect this is a packaging bug, if so, I'll report a bug against LDC.

> But all other library headers should be shareable between compilers and can be placed in /usr/include/d. (This might even work for phobos, I don't think we have many compiler specific changes there. But then you need to use the same frontend version for all compilers.)

This is probably a very naive question, but: Isn't the D specification finished? If the language is done, why would there be a dependence of Phobos on a specific compiler?
Or is this because the new code in Phobos might expose bugs in the compiler itself, which causes these incompatibilities?

> You can only install headers for one library version with this approach! A versioned approach is nicer /usr/include/d/libfoo/1.0.0 but requires explicit compiler support and it's unlikely this will happen (or explicit dub support and you compile everything through dub).

Would be nice, but since we - most of the time - only allow one version of a specific software package to be present in distributions, it isn't a huge issue.

>> ## dub: Where should dub modules be installed?
>
> What does FHS recommend in this case? If you only keep headers/sources you could install into /usr/include.

By the FHS, /usr/include/d should be okay for headers (well, the spec explicitly says C programming language headers, but well.... :P).

> Otherwise you probably need /var/cache or /var/lib or somehting like that.

/var/cache and /var/lib are forbidden for distributors, since they contain state information which shouldn't be touched (as always, there are exceptions...).

If the dub packages contain architecture-independent stuff only, they need to go to /usr/share/dlang or /usr/include/d, and the shared or static library must go to /usr/lib/<triplet>/.

> If you split packages you could use the standard /usr/lib* folders but then you need to keep versioned subdirectories to support installing multiple versions.

Multiple versions would be a rare event, since no distro package management system allows that (excluding Nix here, which is kind of special).

Installing arbitrary arch-specific content into a subdirectory in /usr/lib is fine too, but I doubt that will be necessary...

>> ## dub: How to install modules?
>> Ideally, dub would also have a "dub install" command, to install the binary, "headers (.di)"/sources and data into standard directories on Linux.
>> (reported as https://github.com/dlang/dub/issues/811 )
>
> See above. The main question is: Do you want to have a C style install of one version of a library and have gdc find the includes and library when running gdc main.a -libfoo

For plain gdc/ldc, I'd say: yes

> or do you want dub-style support for multiple library versions which means you need to compile everything through dub.

For stuff using dub, I'd say yes to that too ;-)

> I'd love to have some extended compiler support (so you could simply do gdc -use=libfoo:1.0.0 and this would pick up the correct headers and linker flags). But as some DMD maintainers are opposed to this idea it won't happen. You'll probably always need dub for a 'nice' user interface.
>
>> ++++++
>> Aside from these things, there are also some other things which would be very useful:
>>
>> ## Shared library support in all compilers
>
> This is mainly a GDC issue. DMD and LDC support shared libs, although I don't think these libraries are ABI compatible.

Sounds more and more like LDC is - at time - the better choice over GDC...

On Thursday, 21 April 2016 at 11:58:23 UTC, Kagamin wrote:
> On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
>> [...]
>
> Many D users are enthusiasts and push the compiler to its limits, they are usually stuck with DMD (even DMD HEAD sometimes) as it provides the latest fixes. It depends on the coding style, AppStream generator is an example of old good plain business logic one is unlikely to need recent frontend for, but for people writing black magic code a couple of versions lag of free compilers behind DMD is usually a blocker, but whoever uses the free compilers already considers them complete enough for their tasks. Currently LDC looks like the most actively maintained one.

Asgen is super-boring code ;-) Mainly because the task it performs can be represented without using much black magic. But still, in order to make it work, I needed to embed a copy of std.concurrency.Generator in my code, because GDCs Phobos didn't contain that thing yet (and it's *so* useful!). Basically, the huge differences in the standard versions is what annoyed me the most in D, since it also means you can't trust the documentation much, depending on the compiler you use. For someone new to D, this is annoying.

LDC also fails to compile asgen:
```
/usr/include/d/std/parallelism.d-mixin-3811(3837): Error: template core.atomic.atomicOp cannot deduce function from argument types !("+=")(shared(ulong), int), candidates are:
/usr/include/d/core/atomic.d(178):        core.atomic.atomicOp(string op, T, V1)(ref shared T val, V1 mod) if (__traits(compiles, mixin("val" ~ op ~ "mod")))
/usr/include/d/std/parallelism.d-mixin-3811(3837): Error: template core.atomic.atomicOp cannot deduce function from argument types !("+=")(shared(ulong), int), candidates are:
/usr/include/d/core/atomic.d(178):        core.atomic.atomicOp(string op, T, V1)(ref shared T val, V1 mod) if (__traits(compiles, mixin("val" ~ op ~ "mod")))
/usr/include/d/std/parallelism.d-mixin-3823(3849): Error: template core.atomic.atomicOp cannot deduce function from argument types !("+=")(shared(ulong), int), candidates are:
/usr/include/d/core/atomic.d(178):        core.atomic.atomicOp(string op, T, V1)(ref shared T val, V1 mod) if (__traits(compiles, mixin("val" ~ op ~ "mod")))
/usr/include/d/std/parallelism.d(3344): Error: template instance std.parallelism.ParallelForeach!(Package[]) error instantiating
source/engine.d(90):        instantiated from here: parallel!(Package[])
```
while GDC compiles the code flawlessly (LDC previously even crashed, but that was with the beta version). I will investigate why this happens now, but it's basically these small things which make working with D less fun, at least when you want to use a system without proprietary components.

On Thursday, 21 April 2016 at 09:07:57 UTC, Johan Engelen wrote:
> On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
>> 
>> The question here is also, which compiler should be the default (which IMHO would be the most complete, most bug-free actively maintained one ^^).
>
> Is performance of the outputted code a criterium? Or the number of supported architectures?

The number of supported architectures is at least a criterium for Debian (speed is also important, obviously ^^) - for arch support, one could set a different compiler on different architectures though...