On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:
> On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d wrote: [...]
>> [...]
>> If we do that, we will run into the D ABI trap: Libraries compiled with compiler X can not be used from software compiled with D compiler Y. There is actually no ABI stability guarantee even between DMD releases.  This will make integrating D a huge pain. Recompiling the dependency-chain of a software from source when compiling a package using the "right" compiler and statically adding the code is forbidden by distro policy.  Having static libraries in the dependencies doesn't solve the issue.  Compiling each library with all D compilers is highly impractical and not really feasible.
>
> This is not a hard problem to solve, IMO.  Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with.

This would require to hack each and every build system to support this *and* if there is a pkgconfig file for the shared library, to change all depending software to check for multiple library names which is a bit crazy...

> The resulting two .so's can either be distributed as separate packages (for minimum bloat, if that's a concern), or as a single package that contains both binaries (since they have different SONAMEs this should not be a problem).

Since one library == one package in Debian, it would have to be multiple packages, otherwise we would need to override Lintian errors/warnings, which is always a bad idea.

> Then if you compile some software X that depends on this library, it will pick up the correct version of the library depending on which compiler you compiled with.

Unfortunately not without special support in the software's build system :-/

> [...]
> DMD is unlikely to support other archs than amd64/ia32 in the foreseeable future, so the justification for dmd being unavailable for arch X would be that upstream DMD simply doesn't support it.  This, however, should not prevent us from using gdc/ldc on those other archs, so that we can still ship packages for those archs.  They will merely require ldc rather than dmd. And obviously, libraries built for that arch will only support the ldc SONAME, not the dmd one. (This may be an argument for bundling both SONAMEs in a single package -- it gets messy if we start shipping multiple libraries, some of which may not be available on all archs. By shipping a single package that includes both versions for ia32/amd64, we can simply omit the DMD-compiled version from other archs.)

Conditional build-dependencies are a bit annoying, but with a metapackage "d-compiler" or similar, using different D compilers on different architectures would definitely be possible.

> Unfortunately, I realize that this means some packages that require the latest DMD would not be available on all archs, if they require features unavailable in gdc/ldc.  But this problem may not be a huge one, now that ldc is mostly up-to-date with dmd (at most 1 release behind, IIRC). GDC may lag behind a bit more because it is unfortunately tied to GCC releases, so we may have to forego using gdc for building newer D packages. But we should be able to ship most D packages compiled with both.

Compiling with multiple compilers is a really big effort with rather questionable gain, IMO.
But as far as LDCs compatibility with other D projects goes: That is really good, the only reason you sometimes can't compile some random D code with LDC might be bugs, but not old standard libraries.

> Furthermore, I wonder if we should standardize on ldc for most D software instead of dmd, unless that software absolutely depends on features only available on dmd.  My reasons are:
>
> - While dmd compiles very fast, it consistently (IME) produces code that
>   runs about 20-30% slower than code produced by gdc (and I presume ldc
>   as well). Since we're talking about building Debian packages on
>   Debian's buildd's, which are background batch processes, compilation
>   speed is no big deal, but the performance of the executable *is* a big
>   deal. The last thing we want is to give a poor impression of D by
>   shipping official Debian packages that have subpar performance.
>
> - DMD is unlikely to target other archs than ia32/amd64 in the
>   foreseeable future, AFAIK, unless the recent relicensing triumph of
>   dmd's backend makes this future more likely.  There will definitely be
>   resistance among DDs because of lack of support for other archs. LDC,
>   in contrast, supports all of Debian's supported archs.
>
> - LDC is already available in Debian, meaning that we can already start
>   adding D packages built with ldc without needing to work through the
>   red tape involved in adding a new compiler to the archive.

I agree with all of that, I think sticking with LDC might indeed be the least painful thing to do...

> - The only case where I can see we'd want to compile with dmd is if the
>   target software uses features that are only available on dmd.

Oddly, it usually is the other way round :P - some projects explicitly want LDC or GDC.

> [...]
> Of course, this in no way diminishes the priority in *distributing* dmd in the Debian archive as -dev packages. Just don't build other Debian packages with it. :-D

Indeed. It needs license review (urgh... ^^), but when it's through that there is no reason not to have it in Debian main and Ubuntu.

> [...]
>> Also: If you want to help out Debian's D team, feel free to ping me or any other D team member (we are very short handed and are only two active people right now). See https://wiki.debian.org/D (caution, wiki page is very outdated, last touched in 2012)
>
> I'd like to be involved, since I *am* technically a Debian developer (though haven't been very active for the past long while -- I do try to at least keep the meager few packages I maintain up-to-date, though).

It's unlikely that I will maintain the dmd package (I might co-maintain it or sponsor uploads though - reason is simply lack of time and -ETOOMANYPROJECTS), so any help there is highly appreciated and I'd help with it whenever I can (I just can't make long-term commitments).
If DMD is uploaded as part of the D team, spreading the maintenance cost will be rather simple too :-)

On Mon, Apr 10, 2017 at 06:12:26PM +0000, Matthias Klumpp via Digitalmars-d wrote:
> On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:
> > On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d wrote: [...]
> > > [...]
> > > If we do that, we will run into the D ABI trap: Libraries compiled
> > > with compiler X can not be used from software compiled with D
> > > compiler Y. There is actually no ABI stability guarantee even
> > > between DMD releases.  This will make integrating D a huge pain.
> > > Recompiling the dependency-chain of a software from source when
> > > compiling a package using the "right" compiler and statically
> > > adding the code is forbidden by distro policy.  Having static
> > > libraries in the dependencies doesn't solve the issue.  Compiling
> > > each library with all D compilers is highly impractical and not
> > > really feasible.
> > 
> > This is not a hard problem to solve, IMO.  Just build the library into two separate binaries, each with a SONAME that encodes the ABI it is compatible with.
> 
> This would require to hack each and every build system to support this *and* if there is a pkgconfig file for the shared library, to change all depending software to check for multiple library names which is a bit crazy...

Hmm.  I guess there's no easy way to make dmd/ldc emit dependencies with modified SONAMEs?  So yeah, you're right, every software that depends on said libraries would have to explicitly depend on a different SONAME depending on what they were built with.  OK, crazy idea, nevermind. :-(


[...]
> > - The only case where I can see we'd want to compile with dmd is if
> >   the target software uses features that are only available on dmd.
> 
> Oddly, it usually is the other way round :P - some projects explicitly want LDC or GDC.

Well, if they explicitly want ldc/gdc, then we'll just have to do with that.  But I think ldc should be able to compile anything that compiles with gdc, so we might still be able to standardize on ldc by default.


[...]
> > Of course, this in no way diminishes the priority in *distributing* dmd in the Debian archive as -dev packages. Just don't build other Debian packages with it. :-D
> 
> Indeed. It needs license review (urgh... ^^), but when it's through that there is no reason not to have it in Debian main and Ubuntu.

One issue, though: if we standardize on compiling Debian packages with ldc, then what do we do with libraries that are not ABI-compatible with dmd?  Since users would expect that if they need libfoo, they'd just `apt-get install libfoo-dev` and then they should be able to just run` dmd -L-lfoo` and it should all magically "just work".


[...]
> > I'd like to be involved, since I *am* technically a Debian developer (though haven't been very active for the past long while -- I do try to at least keep the meager few packages I maintain up-to-date, though).
> 
> It's unlikely that I will maintain the dmd package (I might co-maintain it or sponsor uploads though - reason is simply lack of time and -ETOOMANYPROJECTS), so any help there is highly appreciated and I'd help with it whenever I can (I just can't make long-term commitments).  If DMD is uploaded as part of the D team, spreading the maintenance cost will be rather simple too :-)

Yeah, the reason I haven't been very active in Debian is also due to ENOTIME, so I doubt I'd be able to keep up with maintaining dmd.  But if it's spread among more D team members, it could be more workable.


T

-- 
Heuristics are bug-ridden by definition. If they didn't have bugs, they'd be algorithms.

On 4/10/2017 8:11 AM, Jack Stouffer wrote:
> On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:
>>   3) Will DMD support more architectures in the near future? How should the
>> architecture issue be handled?
>
> This can be definitively answered as "no",
> https://issues.dlang.org/show_bug.cgi?id=15108

With it being open source now, this could change if someone wants to pick up the flag.

On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:
>
> Hmm.  I guess there's no easy way to make dmd/ldc emit dependencies with modified SONAMEs?  So yeah, you're right, every software that depends on said libraries would have to explicitly depend on a different SONAME depending on what they were built with.  OK, crazy idea, nevermind. :-(

Doesn't sounds that crazy; you already do it with GCC versions, right?  (Debian _does_ have something like that, right?  Where you can pick your C compiler.)

-Wyatt

On 10 April 2017 at 19:48, Matthias Klumpp via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
>>
>> [...]
>> Everyone should follow GDC's ABI, rather than trying to mimic DMD calling
>> convention. ;-)
>
>
> GDC is working very well, and using it would actually be the natural choice
> for us as GCC is the default compiler.
> However, there are a few problems with that:
>  1) GDC doesn't compile a large amount of D code currently out there due to
> having an outdated standard library and runtime. While we would like to use
> it, it doesn't help if it just can't compile the things.
>

Master sports Phobos 2.071.  Someone will have to see whether latter versions can be built using it.


>  2) GDC isn't part of the official GCC. This has quite wide reaching
> implications, most importantly other distros not offering it - in Fedora /
> Red Hat, this is the reason why GDC isn't available. This means fewer people
> test with GDC and we're basically compiling upstream projects with a
> compiler they never ever tested, potentially creating new issues that we
> can't easily forward upstream. This is less of an issue with LDC as LDC is
> more widely available in other distributions.
>

This is an unfortunate distribution problem, things would be different if GCC were more like a library.  It's not like anyone is helping me with the push.  I am ultimately the one who is doing the tens of thousands of lines of code changes, and writing up the thousands of lines of documentation just to make it lean and presentable.  ;-)

On 10 April 2017 at 19:50, Matthias Klumpp via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:
>>
>> On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
>>>
>>>
>>> Btw, at time we are just ignore the ABI issues, and surprisingly nothing broke yet, indicating that ABI breakage isn't very common or not affecting commonly used interfaces much.
>>
>>
>> One big ABI change was in 2.071:
>> https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces.
>> ;-)
>> Nothing broke because of that?
>
>
> I am reading release notes, so we rebuilt dependencies of LDC - I have no idea about GDC-depending D code though. But since no bugs were reported, I assume no issues are present :-)

That change is in gdc-7.x - so still in Debian unstable.

On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
> Master sports Phobos 2.071.  Someone will have to see whether latter versions can be built using it.

… and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++?

 — David

On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:
> I am reading release notes, so we rebuilt dependencies of LDC -

(I assume you mean reverse dependencies.)

> […] But since no bugs were reported, I assume no issues are present :-)

So do we need to put a reminder about the ABI being unstable into set of every release notes to make sure we won't get angry bug reports once users actually build their own D code against your packages? ;)

 — David

On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:
> This has worked nicely for every language. If you don't have templates in your API or don't change the templates between releases, you can survive with one library for a long time.

But the vast majority of D libraries _do_ have templates (starting with Phobos).  How should this situation be dealt with?

How does Debian deal with, e.g., fixes to the templated code in Boost, which impact on other packages built using those header-only libraries?

On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:
>>
>> Master sports Phobos 2.071.  Someone will have to see whether latter versions can be built using it.
>
>
> … and some weird Frankensteinian mix of several frontend versions, I take it, maybe enough to build Phobos, but not necessarily compatible for user code? Or did you port all the changes since 2.068.2 back to C++?
>
>  — David

All the regression fixes and none of the bugs!

The current situation is that it should be link-compatible with current upstream/stable.  Enough so that when someone has the time to test, it should just be a case switching the sources and building the D version.