Hi there!
These are probably questions directed mostly at Walter and others shaping D's goals, but this could be of general interest for many people, so to the forum it goes :-)

DMD is completely free software now and we can legally distribute it in Debian main - yay! This is an awesome achievement and will make D adoption in Linux distributions much easier (in fact, Red Hat is working on getting good support into Fedora too already).

There are a two issues though that we will be facing in Debian soon, and I would like to get some opinion and maybe perspective on from the D community on them.

Naturally, when the reference compiler is available in Debian, we would compile everything with that, as it is the development focus and the thing many people test with.
We do, however, have quite a bit of bioinformatics and other D software in the archive where performance matters - so our users and the developers of that software (like BioD, potentially Mir, maybe even Vibe.d) will want the fastest performance and will ask us to compile the libraries with LDC or GDC.

If we do that, we will run into the D ABI trap: Libraries compiled with compiler X can not be used from software compiled with D compiler Y. There is actually no ABI stability guarantee even between DMD releases.
This will make integrating D a huge pain. Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy. Having static libraries in the dependencies doesn't solve the issue. Compiling each library with all D compilers is highly impractical and not really feasible.
So, how should we proceed here? We could make it "DMD is the only thing on the highway" compiling everything with DMD with zero exceptions, which would leave us with only DMD-internal ABI breakage and bad D code performance for some libraries. We could also continue using LDC for everything, but that comes with it's own issues (I am hitting quite a bunch of LDC bugs and upstream projects usually test with DMD or use features which are only in the latest DMD).

The other issue is architecture support. In Debian we are strongly encouraged to support as many architectures as possible, to the point of having to justify why arch X is not supported.
LDC runs on at least armhf and ppc64el additionally to amd64/ia32, while DMD says it's specifically only for ia32/amd64.
This means we might end up compiling stuff with different D compilers on different architectures, or we will need to drop architectures and request arch-specific package removals for things that currently build on architectures not supported by DMD, which will trigger resistance.

So, in summary:
  1) Is there some perspective on D getting a defined ABI that works with all major D compilers?
  2) What would the D community recommend on how to deal with the ABI issues currently? A Linux distribution is a bunch of tightly integrated software, and changing one piece in an incompatible way (e.g. by building it with LDC instead of DMD) will have consequences.
  3) Will DMD support more architectures in the near future? How should the architecture issue be handled?

I am interested in some feedback here, since I currently can't see a good way to address these issues.

Also: If you want to help out Debian's D team, feel free to ping me or any other D team member (we are very short handed and are only two active people right now). See https://wiki.debian.org/D (caution, wiki page is very outdated, last touched in 2012)

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
> Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy.

This is the part that I do not understand.

Who came up with those policies and decided that they apply to D? Because I really don't think they should.

A lot of D code is generic. Very extremely few exceptions, it is a given that any time a library is used, at least some part of the library's code is going to be compiled not when the library is "built", but when programs using the library are. The notion of D libraries preserving ABI compatibility of built shared libraries is pretty much unheard of (it would have to be an explicit design goal), with Phobos and everything else going for source-compatibility only. Heck, there are packages out there that do not even produce useful object files (such as header-only C++ libraries).

As far as I can see, the main obstacle of D in Debian is the extreme dogmatism of the distribution's policies. D is not C (where these rules make much more sense), so I really don't think the same rules should apply. Can we treat it more like an interpreted language instead?

On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>> Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy.
>
> This is the part that I do not understand.
>
> Who came up with those policies and decided that they apply to D? Because I really don't think they should.

They are the result of years of experience in building complex systems and keeping them secure.
If you have a dependency chain "X -> Y -> Z" (-> meaning "depends on"), and you find a security bug in Z, you the security team will just need to fix the bug in Z to resolve it in the whole distribution.
But if the code which has this issue is compiled into all of the packages that depend on them, you will need to rebuild the full dependency chain to actually fix the security issue, which is not only time intensive but also a huge maintenance effort.
In this simple example it doesn't look like much, but those dependency chains can grow massively large and complicated, and the only way to keep the large software stack maintainable and secure is by splitting pieces cleanly.

Embedded code copies are allowed in rare events, but in these cases the security team needs to be aware of them.
Sometimes, the licenses also explicitly prevent embedded code copies.

Aside from these issues, splitting things cleanly also makes general package maintenance much easier, and adds flexibility for our users who can mix and match parts of the distribution as they like and combine them with their own code.

You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system. The technical policies work for everything else, so there is nothing that really justifies an exception for D here (if 10% of Debian's code was written in D and the Debian D team was really large we could maybe get one, but not the way it is now).
And tbh, I think finding a good solution here is entirely possible.

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
> So, in summary:
>   1) Is there some perspective on D getting a defined ABI that works with all major D compilers?
>   2) What would the D community recommend on how to deal with the ABI issues currently? A Linux distribution is a bunch of tightly integrated software, and changing one piece in an incompatible way (e.g. by building it with LDC instead of DMD) will have consequences.
>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?
>
> I am interested in some feedback here, since I currently can't see a good way to address these issues.

How do Debian and C++ go along? There is no ABI compatibility between GCC and Clang afaik.

I general, I would favor to build stuff with LDC instead of DMD, because of performance.

On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
>> Who came up with those policies and decided that they apply to D? Because I really don't think they should.
>
> They are the result of years of experience in building complex systems and keeping them secure.
> If you have a dependency chain "X -> Y -> Z" (-> meaning "depends on"), and you find a security bug in Z, you the security team will just need to fix the bug in Z to resolve it in the whole distribution.
> But if the code which has this issue is compiled into all of the packages that depend on them, you will need to rebuild the full dependency chain to actually fix the security issue, which is not only time intensive but also a huge maintenance effort.
> In this simple example it doesn't look like much, but those dependency chains can grow massively large and complicated, and the only way to keep the large software stack maintainable and secure is by splitting pieces cleanly.
>
> Embedded code copies are allowed in rare events, but in these cases the security team needs to be aware of them.
> Sometimes, the licenses also explicitly prevent embedded code copies.
>
> Aside from these issues, splitting things cleanly also makes general package maintenance much easier, and adds flexibility for our users who can mix and match parts of the distribution as they like and combine them with their own code.

No, I understand all of this. What I'm saying that in the case of D, these rules, though making sense, will just not work. You can't replace a piece of code in a template instantiation in a compiled program, shared libraries and stable ABI or not.

> You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system.

The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.

On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev wrote:
> [...]
> Can we treat it more like an interpreted language instead?

An interpreted language would interpret the code on the target system at runtime. This is not what D does, so we can't really treat it like we treat Python (where it is possible no neatly separate Python modules into separate packages - transitions triggering rebuild cascades only exist when we jump to the next major CPython version, which is something distributions are well prepared for. Transitions are only an issue if they happen constantly).
At time, D is treated like C++, since it has much of the same challenges and we know how to deal with C++ - additionally to C++, D unfortunately though also has the unique issues outlined above, which complicate things.

I also want to stress that having a single C++ library like Boost compiled into stuff and rolling dependency transitions when its API/ABI changes with a major release is less of a problem than having the entire language give zero stability and interoperability guarantees on anything that is compiled with it.

On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev wrote:
> On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
>>> Who came up with those policies and decided that they apply to D? Because I really don't think they should.
>> [...]
>> You need to see here that D is not the center of the world and we will need to make it work nicely with the rest of the system.
>
> The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.

There is a really easy way to fix this: SONAMEs. Whenever you change something in the library breaking ABI or API, you bump it's SOVERSION, which will force the distribution to perform a transition and rebuild the dependency chain. If you give absolutely zero stability guarantees, you just set the SOVERSION equal to the project's version and trigger a transition every time (incredibly annoying, but, well, okay).

This has worked nicely for every language. If you don't have templates in your API or don't change the templates between releases, you can survive with one library for a long time.

This is working really great on the level of individual libraries, but if the whole language is ABI-unstable, the issues are much bigger and harder to track.

Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
> Hi there!
> [...]
> If we do that, we will run into the D ABI trap: Libraries compiled with compiler X can not be used from software compiled with D compiler Y. There is actually no ABI stability guarantee even between DMD releases.
> This will make integrating D a huge pain. Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy. Having static libraries in the dependencies doesn't solve the issue. Compiling each library with all D compilers is highly impractical and not really feasible.
> So, how should we proceed here? We could make it "DMD is the only thing on the highway" compiling everything with DMD with zero exceptions, which would leave us with only DMD-internal ABI breakage and bad D code performance for some libraries.

I am working on a project that will require to be used almost exclusively with LDC (targeting the GPU backends of LLVM), and while technically only the mangling needs to match using any other complier will likely be a major workflow impediment. Perhaps it is reasonable to make them compile the stdlibs for LDC, but if it becomes as popular as I hope (dominating it's niché) it may be worthwhile considering shipping prebuilts.

On Monday, 10 April 2017 at 12:59:58 UTC, qznc wrote:
> [...]
> How do Debian and C++ go along? There is no ABI compatibility between GCC and Clang afaik.

Clang offers compatibility for most basic features. There are some ABI compatibility issues though and you find them reported in the Clang/libc++ bugtrackers, and it's a pain (but the Clang/LLVM guys think they can do some ABI better/faster than what GCC offers, so some breakage is deliberate).
In Debian, GCC compiles everything as the system's default compiler, so at least inside the distribution we don't have to worry about potential incompatibilities. Since GCC also supports an enormous amount of architectures and has strong optimization, the case is different there.

In terms of "what happens when users use the OSes C++ libraries and compile with Clang instead of GCC" the situation is similar though: They might run into ABI issues (rarer though than with D). For the distro itself the problem doesn't exist though.

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
> There are a two issues though that we will be facing in Debian soon, and I would like to get some opinion and maybe perspective on from the D community on them.

First I would like to say thank you for all the work you did in getting Tilix packaged for Debian, it is very much appreciated. While I'm not an expert in this topic I'll throw in my two cents anyway :)

So to the topic at end, I tend to agree with Vladimir in that I just don't see it as feasible to make every dependency a shared library. Personally I would suggest that only libphobos and libdruntime be managed as shared libraries by a distro. Otherwise, any other dependencies used in DUB or elsewhere would be statically compiled.

If in time a new D library ends up becoming a keystone foundation for many packages it could be considered for inclusion as a shared library. Otherwise, trying to manage every DUB dependency as a potential shared library is a huge amount of work and I don't feel most of them are mature or well maintained enough to support this approach.

If you just concentrate on compiler, libphobos and libdruntime, you can have separate packages for each compiler toolchain which was Arch does. It has a libphobos package for DMD and a liblphobos for LDC. This then enables developers to specify the tool chain they prefer without interference.

In addition to the question about D and C++. what do distros typically do for Rust and Cargo dependencies, or Java and Maven? Wouldn't that be a similar paradigm to D and DUB dependencies?