On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:
> That's why I have been writing a lot of Makefiles and Meson build definitions lately.

It seems like doing so without having a closer look at the realities of D software (no stable ABI, etc.) might not have been the best use of your time. How would Dub be such fundamentally unfixable that making the official language tools play nice with distro packaging (and the other way round) wouldn't be preferable over manually re-writing build systems all over the place?

> But with an unstable ABI the standard library is also affected, which would trigger us to do Haskell-style versioning (which mangles dependencies to depend on a virtual package containing a hash of the GHC version), and that not only sucks but also requires quite a lot of manpower.

So it sounds like there is a solution already for other languages. Could you elaborate some more on the problems with it? I suppose there is some wiki page documenting the process somewhere?

> Rust only has one compiler which strongly optimizes, so we don't have the problem of choosing the right one. Cargo is/was an issue but it's being worked on and seems to work well now: https://wiki.debian.org/Teams/RustPackaging/Cargo

Rust doesn't have a stable ABI either, and it doesn't look like there is any movement in that direction (not that I think that there should be). That the people driving the effort might not be aware of it yet doesn't mean it isn't an issue for them.

IIRC OCaml is also very much a statically linked affair. And how does Debian distribute Go binaries? Is there any issue with those being linked statically? If not, let's just distribute D libraries as source and compile/link them statically when building binaries, and problem solved.

Some of the compiler developers, myself included, understand the issues involving ABI stability and distro packaging quite well (although the latter admittedly only on a general level). In fact, one of my earliest open source memories is of some work in the trenches ensuring ABI stability of some bits of KDE across releases. Yet we are still going to tell you that the D ABI is going to remain unstable for the foreseeable future. This is not something that just requires a man-week or month to "fix" in the compiler, but would impact many other areas as well, for example language evolution.

If you somehow got the impression that this is just due to D developers "not getting it", just have a look at the other recent compiled languages. Go and Rust don't fare any differently, and even Swift, with all its development manpower, doesn't have a stable ABI yet [1]. And I believe header-only C++ template libraries have been mentioned already as well.

I can only speak for myself, of course, but I certainly see the strategic importance of integration into the Linux distribution ecosystem for D, and I'm very happy to work with packagers wherever possible. However, you also need to acknowledge the properties of the ecosystem *you* are working with. If you see a big stretch of difficult terrain in front of you, closing your eyes won't make it go away; you'll only lose time you could spend working around it. ABI instability is something you'll have to work around one way or the other.

 — David


[1] Actually, the Swift guys tried to pin it down leading up to version 4, but in the end decided it was too much work to get done in time and possibly not a wise idea at this point altogether.

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
> Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy.

Yet, from what I could find after a brief search, the Rust packaging guys are planning to do exactly that: https://wiki.debian.org/Teams/RustPackaging/Policy#Library_package_structure

You might want to get in touch with them, particularly regarding

> […] require policy exceptions, which is something we won't do […]

before either side wastes more time on this than necessary.

 — David

On 4/10/2017 5:59 AM, Matthias Klumpp wrote:
> You need to see here that D is not the center of the world and we will need to
> make it work nicely with the rest of the system. The technical policies work for
> everything else, so there is nothing that really justifies an exception for D
> here (if 10% of Debian's code was written in D and the Debian D team was really
> large we could maybe get one, but not the way it is now).
> And tbh, I think finding a good solution here is entirely possible.

I think it is possible, too, and thank you for your efforts helping us do this.

The ABI differences between the compilers is unfortunate, and is largely the result of historical accident.

The first problem is that my idea originally was for D to have its own function calling convention, which would free us to innovate to have a more efficient calling convention than the C one. This hasn't panned out in practice, and ldc/gdc decided to sensibly stick with the C ABI. At some point, we should just crowbar dmd to generate the C ABI, but this has its own problems - it'll break code that uses the inline assembler. No obvious solution there.

The next problem is that dmd occasionally changes the interface to the D runtime. Or more accurately, with about every release. This has not been an issue historically for us, as the two have always been a matched set. I'm a lot less sure how to deal with this. I also do not know how the gdc/lds druntime interfaces differ.

On 4/10/2017 1:43 PM, Iain Buclaw via Digitalmars-d wrote:
> This is an unfortunate distribution problem, things would be different
> if GCC were more like a library.  It's not like anyone is helping me
> with the push.  I am ultimately the one who is doing the tens of
> thousands of lines of code changes, and writing up the thousands of
> lines of documentation just to make it lean and presentable.  ;-)

Iain, thank you for doing this herculean task all by yourself. I know that you need help - anyone want to step up?

I hope that making D fully open source will have a side effect of motivating people to help you get gdc into the main gcc distribution.

On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
> I also want to stress that having a single C++ library like Boost compiled into
> stuff and rolling dependency transitions when its API/ABI changes with a major
> release is less of a problem than having the entire language give zero stability
> and interoperability guarantees on anything that is compiled with it.

How is the g++/clang++ issue handled?

On 11 April 2017 at 01:27, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> The next problem is that dmd occasionally changes the interface to the D runtime. Or more accurately, with about every release. This has not been an issue historically for us, as the two have always been a matched set. I'm a lot less sure how to deal with this. I also do not know how the gdc/lds druntime interfaces differ.


If you are referring to things such as _d_throw, _d_throwc, _d_throwdwarf, others... In most cases, GDC has always done things in the way that works for it, and as a result has almost never changed ABI of library functions that it is tied to.  So we just have the one _d_throw function for throwing dwarf EH, while DMD has gone through several iterations over the years before finding themselves doing things the same way as GDC - by which point you've given the function a different name.  ;-)

On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:
> The next problem is that dmd occasionally changes the interface to the D runtime. […] I also do not know how the gdc/lds druntime
> interfaces differ.

Just to make this very clear to everybody reading this thread: It's not even just that, but also the fact that we guarantee API-, but not ABI-stability for Phobos. Every time we continue to improve the pure/nothrow/@nogc situation by marking up some more code, we are breaking the ABI, because the mangled names of the involved symbols change. The ongoing work on `scope` also breaks the ABI when enabled.

Yes, there are also some differences between the DMD/GDC/LDC runtime interfaces, but this is a comparatively very minor issue that doesn't make sense worrying too much about as long as ABI compatibility between releases isn't going to happen for other reasons such as the above. [1]

 — David


[1] In fact, it looks like – for example with DMD moving to libunwind-based EH as well – the issue is slowly resolving itself anyway and at some point we'll merely have to sit down for a week and iron out the last few kinks.

On 11 April 2017 at 01:33, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 4/10/2017 6:08 AM, Matthias Klumpp wrote:
>>
>> I also want to stress that having a single C++ library like Boost compiled
>> into
>> stuff and rolling dependency transitions when its API/ABI changes with a
>> major
>> release is less of a problem than having the entire language give zero
>> stability
>> and interoperability guarantees on anything that is compiled with it.
>
>
> How is the g++/clang++ issue handled?

I suspect that clang++ keeps ABI compatibility with g++ for both mangling and calling convention, except when functions are not callable from outside the compilation unit (static functions, templates).

Not sure about libstdc++, I suspect they are either compatible for certain things.

On 4/10/2017 4:43 PM, David Nadlinger wrote:
> [1] In fact, it looks like – for example with DMD moving to libunwind-based EH
> as well – the issue is slowly resolving itself anyway and at some point we'll
> merely have to sit down for a week and iron out the last few kinks.

dmd is not moving to a libunwind-based EH, it already has moved to it!

But as an example of ABI change, the latest EH @nogc proposal changes the layout of the Throwable object, and adds some code to the _d_throwdwarf() function.

https://github.com/dlang/druntime/pull/1804

On Monday, April 10, 2017 23:08:17 David Nadlinger via Digitalmars-d wrote:
> IIRC OCaml is also very much a statically linked affair. And how does Debian distribute Go binaries? Is there any issue with those being linked statically? If not, let's just distribute D libraries as source and compile/link them statically when building binaries, and problem solved.

Honestly, I don't see how it really makes much sense to use shared libraries with D except in cases where you have no choice. The lack of ABI compatibility makes them almost useless.

Also, what are we even looking to distribute in debian? I would have thought that the normal thing to do would be to build with dub, in which case, having the compiler and dub be debian packages makes sense but not really anything else. If you're looking to package an application that was written in D, then that becomes another question, but then if you just statically link it, the ABI compatibility problem goes away as does any need to package any D library dependencies.

- Jonathan M Davis