Am Thu, 21 Apr 2016 01:01:01 +0000
schrieb Matthias Klumpp <matthias@tenstral.net>:

> Hello!
> Me bringing dub to Debian (and subsequently Ubuntu) has sparked
> quite some interest in getting more D applications shipped in
> Linux distributions.

Having been in a similar situation years ago, I can share my experience with packaging Dlang libraries and programs.

> ## Why is every D compiler shipping an own version of Phobos?
> If one assumes that the D2 language specification is stable, and
> D compilers implement it completely, the question is why every
> compiler is shipping an own copy of Phobos. This is a major pain,
> since e.g. GDCs Phobos is behind what is documented on dlang.org,
> but also because the compilers sometimes accidentally use the
> "wrong" Phobos version (e.g. GDC trying to use the one of LDC),
> leading to breakage.
> Also, distributors hate code duplication, so deduplicating this
> would be awesome (druntime being compiler-specific makes sense to
> me, Phobos not so much...).
> It's not an essential thing, but would be quite nice...

The compiler devs would need to explain the coupling of druntime and Phobos. What I know is that Dlang's ABI and Phobos' API are still changing. There are at this very moment talks about changing the function name mangling, a deprecation of broken visibility rules and the shared keyword will need a rewrite at some point in the future.

In practice you have a different Phobos per compiler and language release. I currently have 9 versions of Phobos installed, most of them in 4 flavors: 32-bit/64-bit, shared and static.

> ## Where should D source-code / D interfaces be put?
> If I install a D library or source-only module as a distribution
> package, where should the sources be put? So far, I have seen:
>   * /usr/include/d
>   * /usr/include/dlang/
>   * /usr/include/d/(dmd|gdc|ldc)
>   * /usr/share/dlang
> Having one canonical path would be awesome ;-)

As explained Phobos is installed in multiple versions, so I
mostly used the compilers' defaults.
For third party libraries I had the same question and figured
the best would be a poll. Since November 2013 over 100 votes
were cast with ~2/3 preferring /usr/include/dlang/
over /usr/include/d/:
http://www.easypolls.net/poll.html?p=52828149e4b06cfb69b97527
This is today the de-facto path on Arch and Gentoo at least.

> ## dub: Where should dub modules be installed?
> ## dub: How to install modules?

For packaging purposes I avoided dub in the past as too much of a black box, for the flexibility usually expected on Gentoo. Many packages ship with make files that integrate better with the system's package manager.

> ## Shared library support in all compilers

+1. For dmd, I changed the default in dmd.conf to link with the shared Phobos. It's been working pretty well since around 2.065 or so. I also install packages like gtkd with .so

> ## Stable ABI among compilers

I agree with you, but let me add that you need multiple Phobos versions installed in parallel in some cases. You may have an app that depends on a library that doesn't compile with the latest dmd/Phobos. IIRC gtkd for Gtk2 doesn't work on dmd-2.071 any more.

-- 
Marco

Am Thu, 21 Apr 2016 15:34:35 +0000
schrieb Matthias Klumpp <matthias@tenstral.net>:

> That doesn't seem to be the case for LDC on Debian... It installs Phobos into /usr/include/d/std, which makes GDC go crazy as soon as LDC is installed too.

See also: http://forum.dlang.org/post/mailman.1433.1460903305.26339.digitalmars-d-ldc@puremagic.com

-- 
Marco

> > You can only install headers for one library version with this approach! A versioned approach is nicer /usr/include/d/libfoo/1.0.0 but requires explicit compiler support and it's unlikely this will happen (or explicit dub support and you compile everything through dub).
> 
> Would be nice, but since we - most of the time - only allow one version of a specific software package to be present in distributions, it isn't a huge issue.

For gtkd 2.4.2 and 3.2.3 I have these directories:
/usr/include/dlang/gtkd-2
/usr/include/dlang/gtkd-3
It makes sense, since Gtk2 can be installed in parallel with
Gtk3 and the major version suffix ensures both packages wont
overwrite each other's imports.

-- 
Marco

Am Thu, 21 Apr 2016 15:34:35 +0000
schrieb Matthias Klumpp <matthias@tenstral.net>:

> Hi, and thanks for your detailed explanations!

You're welcome :-)

> On Thursday, 21 April 2016 at 11:49:13 UTC, Johannes Pfau wrote:
> > On Thursday, 21 April 2016 at 01:01:01 UTC, Matthias Klumpp wrote:
> >> [...]
> >
> > You currently can't install druntime or phobos headers in this directory, as each compiler will have slightly modified versions and then you end up with /usr/include/d/(dmd|gdc|ldc).
> 
> That doesn't seem to be the case for LDC on Debian... It installs
> Phobos into /usr/include/d/std, which makes GDC go crazy as soon
> as LDC is installed too.
> I suspect this is a packaging bug, if so, I'll report a bug
> against LDC.
> 

IIRC GDC makes sure to install in some lib/gcc directory to avoid such
problems. But it's possible ldc defaults to the include directory. As
an example, Arch Linux configures all compilers to
use /usr/include/dlang&(ldc|gdc|dmd):
https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/ldc#n37
https://projects.archlinux.org/svntogit/community.git/tree/trunk/folders.diff?h=packages/gdc
https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/dmd#n54

> > But all other library headers should be shareable between compilers and can be placed in /usr/include/d. (This might even work for phobos, I don't think we have many compiler specific changes there. But then you need to use the same frontend version for all compilers.)
> 
> This is probably a very naive question, but: Isn't the D
> specification finished? If the language is done, why would there
> be a dependence of Phobos on a specific compiler?
> Or is this because the new code in Phobos might expose bugs in
> the compiler itself, which causes these incompatibilities?

We only introduce backwards compatible changes, but it's still possible that a new feature will be used in a new phobos version. Most of the time it's some obscure template bug fix which will require a new frontend version. And phobos is often one of the first libraries to use bug fixes or new compiler features.

> [...]
> > See above. The main question is: Do you want to have a C style install of one version of a library and have gdc find the includes and library when running gdc main.a -libfoo
> 
> For plain gdc/ldc, I'd say: yes
> 
> > or do you want dub-style support for multiple library versions which means you need to compile everything through dub.
> 
> For stuff using dub, I'd say yes to that too ;-)

Where do ruby / python package managers install their packages? I guess they're special as they use arch-independent sources?

Anyway, for a one-version system wide install standard /usr/include/d
and /usr/lib is probably best.
I don't know a reasonable location for a dub 'repository' though.

> >> ## Shared library support in all compilers
> >
> > This is mainly a GDC issue. DMD and LDC support shared libs, although I don't think these libraries are ABI compatible.
> 
> Sounds more and more like LDC is - at time - the better choice over GDC...

GDC is indeed lagging behind a little bit. We're working on it ;-)

On Thursday, 21 April 2016 at 15:41:17 UTC, Matthias Klumpp wrote:
> Asgen is super-boring code ;-) Mainly because the task it performs can be represented without using much black magic.

One interesting possibility with D is that you can test what performance impact boundschecking has on your code with just a compiler switch. Is asgen performance sensitive?

> LDC also fails to compile asgen:
> ```
> /usr/include/d/std/parallelism.d-mixin-3811(3837): Error: template core.atomic.atomicOp cannot deduce function from argument types !("+=")(shared(ulong), int), candidates are:
> /usr/include/d/core/atomic.d(178):        core.atomic.atomicOp(string op, T, V1)(ref shared T val, V1 mod) if (__traits(compiles, mixin("val" ~ op ~ "mod")))

RMW operations on shared data are deprecated. The template filter on atomicOp is a little different: https://dlang.org/phobos/core_atomic.html#.atomicOp
Looks like an LDC bug.

FYI: I don't want to comment here for now because Debian packaging is
very different from Arch packaging but I am keeping my eyes on the thread :)

On Friday, 22 April 2016 at 09:18:48 UTC, Kagamin wrote:
> Looks like an LDC bug.

Or not. If template constraint doesn't match, the error should be "template instance does not match template declaration".

On Friday, 22 April 2016 at 09:18:48 UTC, Kagamin wrote:
> RMW operations on shared data are deprecated. The template filter on atomicOp is a little different: https://dlang.org/phobos/core_atomic.html#.atomicOp
> Looks like an LDC bug.

Yes, this was indeed an LDC issue. We have our own implementation, and apparently missed that the template constraint on the DMD version was updated: https://github.com/ldc-developers/ldc/pull/1456

 — David

On 2016-04-21 03:01, Matthias Klumpp wrote:

> ## How complete are the free compilers?
> This is an important question, because we would need to know whether we
> can expect D code to be compiled by any compiler, or whether there are
> tradeoffs that must be made.
> This question is asking manly how complete LDC and GDC are compared to
> each other, but also whether both are implementing the D2 specification
> completely.

GDC does not completely implement the D2 specification. Not sure about LDC. GDC is several versions behind DMD. LDC are much closer to DMD these days. Perhaps only one version behind.

> The question here is also, which compiler should be the default (which
> IMHO would be the most complete, most bug-free actively maintained one ^^).

I would suggest LDC because it more up to date with DMD.

> ## Why is every D compiler shipping an own version of Phobos?
> If one assumes that the D2 language specification is stable, and D
> compilers implement it completely,

No. It tries to avoid breaking changes on purpose (there will always be bug fixes that breaks code and introduction of new bugs). But new additions are added almost for each new release.

> the question is why every compiler is shipping an own copy of Phobos.

I'm guessing because Phobos might depend on bugs that was fixed in the latest release and new features are used early. A language feature might also have been added because it was needed for a larger Phobos addition.

> ## Where should D source-code / D interfaces be put?
> If I install a D library or source-only module as a distribution
> package, where should the sources be put? So far, I have seen:
>   * /usr/include/d
>   * /usr/include/dlang/
>   * /usr/include/d/(dmd|gdc|ldc)
>   * /usr/share/dlang
> Having one canonical path would be awesome ;-)

Personally I would say that all libraries should be used through Dub, making this a non issue.

> ## dub: Where should dub modules be installed?
> DUB currently only searches online for it's packages, but if we install
> them as distributor, we want dub to also look locally in some path where
> the packages can be found, and - if possible - use a prebuilt
> shared/static library.
> (reported as https://github.com/dlang/dub/issues/811 )
>
> ## dub: How to install modules?
> Ideally, dub would also have a "dub install" command, to install the
> binary, "headers (.di)"/sources and data into standard directories on
> Linux.
> (reported as https://github.com/dlang/dub/issues/811 )

For libraries, as I above, use Dub. For tools/applications they can hopefully be packaged completely separately.

> ++++++
> Aside from these things, there are also some other things which would be
> very useful:
>
> ## Shared library support in all compilers
> In distributions, we hate duplicating binary code copies. At time, D
> links everything statically, which means that when a bug is discovered
> in a library used by many other tools (worst case: the standard
> library), we will need to recompile all depending software.
> This is really annoying for the security team, while not being as bad as
> having actual duplicate source-code copies which need to be patched
> individually.

Most D applications are linked statically because dynamic libraries have not been supported at all until recently. In addition to that most computers will not have the D runtime and standard library installed.

> ## Stable ABI among compilers
> Simple: Compile your library with GDC, have it used by code compiled
> with LDC. Otherwise, shared libraries would be of very limited use.

Yeah, that would be nice :). Currently there's no guarantee about ABI compatibility at all.

-- 
/Jacob Carlborg

On 23 April 2016 at 21:35, Jacob Carlborg via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 2016-04-21 03:01, Matthias Klumpp wrote:
>
>> ## How complete are the free compilers?
>> This is an important question, because we would need to know whether we
>> can expect D code to be compiled by any compiler, or whether there are
>> tradeoffs that must be made.
>> This question is asking manly how complete LDC and GDC are compared to
>> each other, but also whether both are implementing the D2 specification
>> completely.
>
>
> GDC does not completely implement the D2 specification. Not sure about LDC. GDC is several versions behind DMD. LDC are much closer to DMD these days. Perhaps only one version behind.
>

To be fair, DMD doesn't implement the D2 specification either.