On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>   1) Is there some perspective on D getting a defined ABI that works with all major D compilers?
>   2) What would the D community recommend on how to deal with the ABI issues currently? A Linux distribution is a bunch of tightly integrated software, and changing one piece in an incompatible way (e.g. by building it with LDC instead of DMD) will have consequences.
>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?

My prediction for Walters reply:

1) No. Not worth it, because templates, ctfe, etc.
3) No. Not worth it, because D frontend stuff is more important.

> Also: If you want to help out Debian's D team, feel free to ping me or any other D team member (we are very short handed and are only two active people right now). See https://wiki.debian.org/D (caution, wiki page is very outdated, last touched in 2012)

Tentative ping, but that Wiki page is not helpful. The linked svn repo is empty. Where and how do you work?

I guess the issue are still the same as you wrote here (except 1. is solved):
https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a

So, mostly dub needs work, I guess.

On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?

This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108

On 10 April 2017 at 15:20, Matthias Klumpp via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev wrote:
>>
>> On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:
>>>>
>>>> Who came up with those policies and decided that they apply to D? Because I really don't think they should.
>>>
>>> [...]
>>> You need to see here that D is not the center of the world and we will
>>> need to make it work nicely with the rest of the system.
>>
>>
>> The opposite is also true: requiring a stable shared library API of every packaged D library is just as unreasonable. In fact, to make these rules useful and applicable to all D programs, you'd have to completely forbid templates in the library's public interface, which would immediately exclude Phobos for one.
>
>
> There is a really easy way to fix this: SONAMEs. Whenever you change something in the library breaking ABI or API, you bump it's SOVERSION, which will force the distribution to perform a transition and rebuild the dependency chain. If you give absolutely zero stability guarantees, you just set the SOVERSION equal to the project's version and trigger a transition every time (incredibly annoying, but, well, okay).
>

I believe GDC in Debian is done correctly on this part.  Last time someone else looked, it seemed like LDC and DMD make use of SOVERSION, but do so in an incorrect manner.

> This has worked nicely for every language. If you don't have templates in your API or don't change the templates between releases, you can survive with one library for a long time.
>
> This is working really great on the level of individual libraries, but if the whole language is ABI-unstable, the issues are much bigger and harder to track.
>
> Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.

Everyone should follow GDC's ABI, rather than trying to mimic DMD calling convention. ;-)

On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
> 
> Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.

One big ABI change was in 2.071: https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces. ;-)
Nothing broke because of that?

cheers,
  Johan

On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?
>
> This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108

When templates are done I might give it a shot :)

On Monday, 10 April 2017 at 14:21:43 UTC, Gerald wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>> There are a two issues though that we will be facing in Debian soon, and I would like to get some opinion and maybe perspective on from the D community on them.
>
> First I would like to say thank you for all the work you did in getting Tilix packaged for Debian, it is very much appreciated. While I'm not an expert in this topic I'll throw in my two cents anyway :)

Thanks!

> So to the topic at end, I tend to agree with Vladimir in that I just don't see it as feasible to make every dependency a shared library. Personally I would suggest that only libphobos and libdruntime be managed as shared libraries by a distro. Otherwise, any other dependencies used in DUB or elsewhere would be statically compiled.

This would raise the maintenance effort for D software, and require policy exceptions, which is something we won't do. It's more likely that we'll see less D in the distro than going down this route.
DUB is unusable for Debian packaging anyway, and alternatives like Automake or Meson handle shared libraries really well, so in itself there is no reason not to do it.

> If in time a new D library ends up becoming a keystone foundation for many packages it could be considered for inclusion as a shared library. Otherwise, trying to manage every DUB dependency as a potential shared library is a huge amount of work and I don't feel most of them are mature or well maintained enough to support this approach.

We'll take care of that, it's what we do as a distro and we have a fair amount of experience in handling these kinds of upstream projects.
The goal is to reduce the maintenance cost for things once they are *in* the distro, any prior work to get it into the distribution is very well invested.
That's why I have been writing a lot of Makefiles and Meson build definitions lately.

> If you just concentrate on compiler, libphobos and libdruntime, you can have separate packages for each compiler toolchain which was Arch does.

That's a necessity already and we do the same. But with an unstable ABI the standard library is also affected, which would trigger us to do Haskell-style versioning (which mangles dependencies to depend on a virtual package containing a hash of the GHC version), and that not only sucks but also requires quite a lot of manpower.

> It has a libphobos package for DMD and a liblphobos for LDC. This then enables developers to specify the tool chain they prefer without interference.

Yes, but that's not feasible for use *in the distribution itself* as we can't just pick and choose the right compiler per package if something up in the dependency chain was compiled with the "wrong" compiler.

> In addition to the question about D and C++. what do distros typically do for Rust and Cargo dependencies, or Java and Maven? Wouldn't that be a similar paradigm to D and DUB dependencies?

Maven dependencies are also separated out into smaller packages and maven (unlike dub) plays relatively well with Debian. It also has no ABI issues and interface stability can be tracked. Rust only has one compiler which strongly optimizes, so we don't have the problem of choosing the right one. Cargo is/was an issue but it's being worked on and seems to work well now: https://wiki.debian.org/Teams/RustPackaging/Cargo
(not that I am not involved in Haskell, Rust or Java packaging).

Just because it makes sense: We do have an upstream guide showing some best practices for upstream projects which help us to maintain software for long periods. It also contains language-specific advice and some general info which is valid for all languages:
https://wiki.debian.org/UpstreamGuide

On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d wrote: [...]
> Naturally, when the reference compiler is available in Debian, we would compile everything with that, as it is the development focus and the thing many people test with.
>
> We do, however, have quite a bit of bioinformatics and other D software in the archive where performance matters - so our users and the developers of that software (like BioD, potentially Mir, maybe even Vibe.d) will want the fastest performance and will ask us to compile the libraries with LDC or GDC.
> 
> If we do that, we will run into the D ABI trap: Libraries compiled with compiler X can not be used from software compiled with D compiler Y. There is actually no ABI stability guarantee even between DMD releases.  This will make integrating D a huge pain. Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy.  Having static libraries in the dependencies doesn't solve the issue.  Compiling each library with all D compilers is highly impractical and not really feasible.

This is not a hard problem to solve, IMO.  Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with.  The resulting two .so's can either be distributed as separate packages (for minimum bloat, if that's a concern), or as a single package that contains both binaries (since they have different SONAMEs this should not be a problem).  Then if you compile some software X that depends on this library, it will pick up the correct version of the library depending on which compiler you compiled with.


> The other issue is architecture support. In Debian we are strongly encouraged to support as many architectures as possible, to the point of having to justify why arch X is not supported.

I'd say this makes a good case for my proposed solution above.

DMD is unlikely to support other archs than amd64/ia32 in the foreseeable future, so the justification for dmd being unavailable for arch X would be that upstream DMD simply doesn't support it.  This, however, should not prevent us from using gdc/ldc on those other archs, so that we can still ship packages for those archs.  They will merely require ldc rather than dmd. And obviously, libraries built for that arch will only support the ldc SONAME, not the dmd one. (This may be an argument for bundling both SONAMEs in a single package -- it gets messy if we start shipping multiple libraries, some of which may not be available on all archs. By shipping a single package that includes both versions for ia32/amd64, we can simply omit the DMD-compiled version from other archs.)

Unfortunately, I realize that this means some packages that require the latest DMD would not be available on all archs, if they require features unavailable in gdc/ldc.  But this problem may not be a huge one, now that ldc is mostly up-to-date with dmd (at most 1 release behind, IIRC). GDC may lag behind a bit more because it is unfortunately tied to GCC releases, so we may have to forego using gdc for building newer D packages. But we should be able to ship most D packages compiled with both.

Furthermore, I wonder if we should standardize on ldc for most D software instead of dmd, unless that software absolutely depends on features only available on dmd.  My reasons are:

- While dmd compiles very fast, it consistently (IME) produces code that
  runs about 20-30% slower than code produced by gdc (and I presume ldc
  as well). Since we're talking about building Debian packages on
  Debian's buildd's, which are background batch processes, compilation
  speed is no big deal, but the performance of the executable *is* a big
  deal. The last thing we want is to give a poor impression of D by
  shipping official Debian packages that have subpar performance.

- DMD is unlikely to target other archs than ia32/amd64 in the
  foreseeable future, AFAIK, unless the recent relicensing triumph of
  dmd's backend makes this future more likely.  There will definitely be
  resistance among DDs because of lack of support for other archs. LDC,
  in contrast, supports all of Debian's supported archs.

- LDC is already available in Debian, meaning that we can already start
  adding D packages built with ldc without needing to work through the
  red tape involved in adding a new compiler to the archive.

- The only case where I can see we'd want to compile with dmd is if the
  target software uses features that are only available on dmd.  Since
  LDC's release cycle is now very close to DMD's, I don't foresee there
  will be too many D projects out there that can't be compiled with LDC.
  So there should really only be a small number of packages that we'd
  have to fallback to dmd, which will also be limited in supported archs
  -- meaning that it will probably be challenging to get it into the
  Debian archive in the first place.

- If we standardize on using ldc to compile Debian packages, we can
  sidestep the ABI compatibility issue (though the SONAME hack will
  probably still be necessary in order to provide users with libraries
  that are usable with dmd).

Because of this, my first inclination is to standardize on ldc rather than dmd, where building Debian packages are concerned.

Of course, this in no way diminishes the priority in *distributing* dmd in the Debian archive as -dev packages. Just don't build other Debian packages with it. :-D


[...]
> Also: If you want to help out Debian's D team, feel free to ping me or any other D team member (we are very short handed and are only two active people right now). See https://wiki.debian.org/D (caution, wiki page is very outdated, last touched in 2012)

I'd like to be involved, since I *am* technically a Debian developer (though haven't been very active for the past long while -- I do try to at least keep the meager few packages I maintain up-to-date, though).


T

-- 
"Holy war is an oxymoron." -- Lazarus Long

On Monday, 10 April 2017 at 14:33:34 UTC, qznc wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>>   1) Is there some perspective on D getting a defined ABI that works with all major D compilers?
>>   2) What would the D community recommend on how to deal with the ABI issues currently? A Linux distribution is a bunch of tightly integrated software, and changing one piece in an incompatible way (e.g. by building it with LDC instead of DMD) will have consequences.
>>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?
>
> My prediction for Walters reply:
>
> 1) No. Not worth it, because templates, ctfe, etc.

That's short-sighted IMHO, because if the template doesn't change, ABI/API doesn't change. Also, some projects use D as better C and don't expose this functionality. It should be up to the project to set the level of API/ABI stability, and not to the compiler to make everything unstable by default.

> [...]
> Tentative ping, but that Wiki page is not helpful. The linked svn repo is empty. Where and how do you work?

Yeah, the page is really poor, it was last touched in 2012. I made a few updates to at least link to the current Git repo.
We generally work on various Git repositories, but not all of them are run by the D team (e.g. libundead and libbiod as well as several games are things I am aware of that aren't D-team maintained but are part of other team's work).

One can find all stuff using D by testing the reverse-build-depends on the LDC and GDC compilers.

> I guess the issue are still the same as you wrote here (except 1. is solved):
> https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a
>
> So, mostly dub needs work, I guess.

Yes, but since Meson is working well and Meson scripts are easy to write, it's not a super high priority item anymore.
As I said earlier, work as a distribution developer is pretty much always about reducing long-term maintenance cost, and not about less work short-term, which means we will gladly write Meson or Automake scripts to integrate software into Debian if there is a demand for it.

On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>>   3) Will DMD support more architectures in the near future? How should the architecture issue be handled?
>
> This can be definitively answered as "no", https://issues.dlang.org/show_bug.cgi?id=15108

> *Walter Bright*:
> Doing an ARM back end wouldn't be that hard. It's much less complex than x86. Most of the work would be deleting about half of the x86 code generator :-)

:D - doesn't sound like a flat-out no, much more like there just wasn't someone doing the work yet.

On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
> [...]
> Everyone should follow GDC's ABI, rather than trying to mimic DMD calling convention. ;-)

GDC is working very well, and using it would actually be the natural choice for us as GCC is the default compiler.
However, there are a few problems with that:
 1) GDC doesn't compile a large amount of D code currently out there due to having an outdated standard library and runtime. While we would like to use it, it doesn't help if it just can't compile the things.

 2) GDC isn't part of the official GCC. This has quite wide reaching implications, most importantly other distros not offering it - in Fedora / Red Hat, this is the reason why GDC isn't available. This means fewer people test with GDC and we're basically compiling upstream projects with a compiler they never ever tested, potentially creating new issues that we can't easily forward upstream. This is less of an issue with LDC as LDC is more widely available in other distributions.

On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:
> On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
>> 
>> Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.
>
> One big ABI change was in 2.071: https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces. ;-)
> Nothing broke because of that?

I am reading release notes, so we rebuilt dependencies of LDC - I have no idea about GDC-depending D code though. But since no bugs were reported, I assume no issues are present :-)