I apologize in advance for the large amount of mail that will likely follow, but I want to address all comments.

On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
> [...]
> One issue, though: if we standardize on compiling Debian packages with ldc, then what do we do with libraries that are not ABI-compatible with dmd?  Since users would expect that if they need libfoo, they'd just `apt-get install libfoo-dev` and then they should be able to just run` dmd -L-lfoo` and it should all magically "just work".

That's the problem I would like to see addressed (but given Walter's comment, it won't be feasible to resolve it in the near future).
We could simply do away with "don't use distro packages for your D programming", at least that's what Go recommends.

On Monday, 10 April 2017 at 20:11:28 UTC, Wyatt wrote:
> On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
>>
>> Hmm.  I guess there's no easy way to make dmd/ldc emit dependencies with modified SONAMEs?  So yeah, you're right, every software that depends on said libraries would have to explicitly depend on a different SONAME depending on what they were built with.  OK, crazy idea, nevermind. :-(
>
> Doesn't sounds that crazy; you already do it with GCC versions, right?  (Debian _does_ have something like that, right?  Where you can pick your C compiler.)

Yes, that's why all packages need to honor the CFLAGS/CC env var somehow or get the default C compiler from dpkg at build time, so we can easily apply new C flags globally, and projects to build the distro with Clang work.
We do never change the SONAME of anything, however. We do track symbols and might change the SOVERSION occasionally if breakage is found (making 3 -> 3a etc.), but the soname isn't changed.
The good thing is that Clang and GCC are (with few exceptions) very compatible.

On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger wrote:
> On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:
>> I am reading release notes, so we rebuilt dependencies of LDC -
>
> (I assume you mean reverse dependencies.)
>
>> […] But since no bugs were reported, I assume no issues are present :-)
>
> So do we need to put a reminder about the ABI being unstable into set of every release notes to make sure we won't get angry bug reports once users actually build their own D code against your packages? ;)

Nah, there are several options here, one would simply be to tell people not to use the distro packages with anything but the default D compiler used in the respective Debian release.
Go apparently tells people not to use Debian-shipped go code in their own projects at all.

On Monday, 10 April 2017 at 22:26:46 UTC, Joseph Rushton Wakeling wrote:
> On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
>> This has worked nicely for every language. If you don't have templates in your API or don't change the templates between releases, you can survive with one library for a long time.
>
> But the vast majority of D libraries _do_ have templates (starting with Phobos).  How should this situation be dealt with?
>
> How does Debian deal with, e.g., fixes to the templated code in Boost, which impact on other packages built using those header-only libraries?

Boost's soversion is changed on every release, and the version is included in it's -dev package as well. That's why we have libboost1.62-dev: https://packages.debian.org/de/sid/libboost1.62-dev (and possibly more).
There is also a boost-defaults package setting the current default Boos version for packages to depend on.
If a new Boost comes out, it's soversion and -dev package name changes, triggering a package transition and subsequently a full rebuild of all stuff depending on Boost.

Doing something like this with D libraries would obviously be possible as well.

On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
> On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
>>>
>>> Master sports Phobos 2.071.  Someone will have to see whether latter versions can be built using it.
>>
>>
>> … and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++?
>>
>>  — David
>
> All the regression fixes and none of the bugs!
>
> The current situation is that it should be link-compatible with current upstream/stable.  Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.

First of all, thank you for your tremendous work on GDC! Fellow developers and me were also pretty stunned by you maintaining a quite large amount of different GDC versions in parallel without a huge team - that's some impressive work!

What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also, you were talking about "bugs" on several occasions, what's the thing with that? Is it GCC or general Phobos bugs? It would probably be awesome to have a summary blogpost or similar on the state of GDC, that could potentially also attract volunteers.
Anyway, all a bit off-topic :-)

On Tuesday, 11 April 2017 at 12:11:10 UTC, Matthias Klumpp wrote:
> On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:
>>
>> All the regression fixes and none of the bugs!
>>
>> The current situation is that it should be link-compatible with current upstream/stable.  Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.
>
> First of all, thank you for your tremendous work on GDC! Fellow developers and me were also pretty stunned by you maintaining a quite large amount of different GDC versions in parallel without a huge team - that's some impressive work!
>
> What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also, you were talking about "bugs" on several occasions, what's the thing with that? Is it GCC or general Phobos bugs? It would probably be awesome to have a summary blogpost or similar on the state of GDC, that could potentially also attract volunteers.
> Anyway, all a bit off-topic :-)

I assume the bugs would be DMDFE bugs.

Iain is giving  talk at DConf on GDC (http://dconf.org/2017/talks/buclaw.html) which should help.

On Monday, 10 April 2017 at 23:08:17 UTC, David Nadlinger wrote:
> On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:
>> That's why I have been writing a lot of Makefiles and Meson build definitions lately.
>
> It seems like doing so without having a closer look at the realities of D software (no stable ABI, etc.) might not have been the best use of your time. How would Dub be such fundamentally unfixable that making the official language tools play nice with distro packaging (and the other way round) wouldn't be preferable over manually re-writing build systems all over the place?

Dub is not unfixable, but there are many more issues with it which make it very very hard to use in a distribution context - I tried that and it didn't work well.
I am treating D in Debian like I would treat C++, which is quite fair given the similar featuresets and challenges.

>> But with an unstable ABI the standard library is also affected, which would trigger us to do Haskell-style versioning (which mangles dependencies to depend on a virtual package containing a hash of the GHC version), and that not only sucks but also requires quite a lot of manpower.
>
> So it sounds like there is a solution already for other languages. Could you elaborate some more on the problems with it? I suppose there is some wiki page documenting the process somewhere?

Haskell and OCaml permanently rebuild the whole stack on every new compiler release, which is why they have permanent transition trackers[1], so they basically continuously rebuild. I want to avoid this at all cost for D, as this is very very maintenance intensive and painful, and will require much more people to work on it than D has available in Debian.

[1]: https://release.debian.org/transitions/

>> Rust only has one compiler which strongly optimizes, so we don't have the problem of choosing the right one. Cargo is/was an issue but it's being worked on and seems to work well now: https://wiki.debian.org/Teams/RustPackaging/Cargo
>
> Rust doesn't have a stable ABI either, and it doesn't look like there is any movement in that direction (not that I think that there should be). That the people driving the effort might not be aware of it yet doesn't mean it isn't an issue for them.

It's being worked on[2], but it's not a super-high priority.
There doesn't seem to be a definitive answer on how Rust is handled in Debian yet (but to know for certain, I would need to ask the Rust team).

[2]: https://github.com/rust-lang/rfcs/issues/600

> IIRC OCaml is also very much a statically linked affair. And how does Debian distribute Go binaries? Is there any issue with those being linked statically? If not, let's just distribute D libraries as source and compile/link them statically when building binaries, and problem solved.

Surprisingly it looks like many Go packages are indeed provided as source-installations.
Doing something like this with D would require Makefiles and Dub to pick up sources from system locations properly which isn't really done yet...

> Some of the compiler developers, myself included, understand the issues involving ABI stability and distro packaging quite well (although the latter admittedly only on a general level). In fact, one of my earliest open source memories is of some work in the trenches ensuring ABI stability of some bits of KDE across releases. Yet we are still going to tell you that the D ABI is going to remain unstable for the foreseeable future. This is not something that just requires a man-week or month to "fix" in the compiler, but would impact many other areas as well, for example language evolution.
>
> If you somehow got the impression that this is just due to D developers "not getting it", just have a look at the other recent compiled languages. Go and Rust don't fare any differently, and even Swift, with all its development manpower, doesn't have a stable ABI yet [1]. And I believe header-only C++ template libraries have been mentioned already as well.
>
> I can only speak for myself, of course, but I certainly see the strategic importance of integration into the Linux distribution ecosystem for D, and I'm very happy to work with packagers wherever possible. However, you also need to acknowledge the properties of the ecosystem *you* are working with. If you see a big stretch of difficult terrain in front of you, closing your eyes won't make it go away; you'll only lose time you could spend working around it. ABI instability is something you'll have to work around one way or the other.

That's the whole point of this thread, I want to find the best solution to deal with this issue and also one the D community can live with.
I am not set on any particular solution yet, at the moment I see the problem and I am thinking about how to deal with it in the best possible way (ideally not doing what OCaml/Haskell are doing, although pulling a "soft Haskell" where we rebuild everything when the D ABI changes would be way easier).

On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:
> On 4/10/2017 5:59 AM, Matthias Klumpp wrote:
>> You need to see here that D is not the center of the world and we will need to
>> make it work nicely with the rest of the system. The technical policies work for
>> everything else, so there is nothing that really justifies an exception for D
>> here (if 10% of Debian's code was written in D and the Debian D team was really
>> large we could maybe get one, but not the way it is now).
>> And tbh, I think finding a good solution here is entirely possible.
>
> I think it is possible, too, and thank you for your efforts helping us do this.
>
> The ABI differences between the compilers is unfortunate, and is largely the result of historical accident.
>
> The first problem is that my idea originally was for D to have its own function calling convention, which would free us to innovate to have a more efficient calling convention than the C one. This hasn't panned out in practice, and ldc/gdc decided to sensibly stick with the C ABI. At some point, we should just crowbar dmd to generate the C ABI, but this has its own problems - it'll break code that uses the inline assembler. No obvious solution there.
>
> The next problem is that dmd occasionally changes the interface to the D runtime. Or more accurately, with about every release. This has not been an issue historically for us, as the two have always been a matched set. I'm a lot less sure how to deal with this. I also do not know how the gdc/lds druntime interfaces differ.

Thank you very much for this context! It's really good to know why things are the way they are to properly understand the problem (I am no compiler developer and ABIs are not my expertise - as outsider to this I see pages like https://dlang.org/spec/abi.html though and wonder whether the incompatibilities are seen as an issue in the D community and whether there is a chance to address them before putting work into setting up infrastructure to rebuild the world on compiler updates).

On Monday, 10 April 2017 at 23:33:17 UTC, Walter Bright wrote:
> On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
>> I also want to stress that having a single C++ library like Boost compiled into
>> stuff and rolling dependency transitions when its API/ABI changes with a major
>> release is less of a problem than having the entire language give zero stability
>> and interoperability guarantees on anything that is compiled with it.
>
> How is the g++/clang++ issue handled?

The C ABI is 100% compatible, the C++ ABI is "mostly" compatible, there is some deliberate breakage from the Clang guys though.
The issue isn't actually handled in Debian as all our code is always compiled with GCC, I am not aware of anything defaulting to Clang (although it might exist, but definitely not for library packages).

With only one dominant compiler, things are way easier ^^

On Tue, 2017-04-11 at 11:55 +0000, Matthias Klumpp via Digitalmars-d wrote:
> 
[…]
> That's the problem I would like to see addressed (but given
> Walter's comment, it won't be feasible to resolve it in the near
> future).
> We could simply do away with "don't use distro packages for your
> D programming", at least that's what Go recommends.

The Go community advice against platform packages is principally because they want up-to-date packages on old distributions.

Most programming languages are going this route, used programming language tools and libraries are decreasingly packaged by the OS. Given things like Anaconda and PyPI, all the Fedora and Debian Python packaging is increasingly wasted – except for those packages used by the distribution itself. So I can see Fedora and Debian packaging Python and some bits and pieces, but otherwise it's not needed. Same is true for Ruby, Go, Groovy, Java, etc.

Debian and Fedora really do need to reassess their packaging strategy with respect to software development. Far too much effort is going into packaging when rolling release is far better handled by language specific systems. Go is going this route, Python likewise.

I suggest getting the compilers onto the distributions as a marketing thing not as a real use thing. For me the way Rust works by having it's own shell-based installer is the right way of installing languages. SDKMAN! is an even more extensive system for all JVM-related systems.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder