Am Tue, 11 Apr 2017 07:44:45 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d@puremagic.com>:

> On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d wrote:
> > On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:
> > > [...]
> > > The problem with /usr/include/d is that is where .di files
> > > would be located not .d. This would also match up with the
> > > c/c++ usage of it.
> >
> > When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^
> 
> Putting .di files in the include directory makes sense when compared
> to C++, but it's definitely true that they're used quite rarely. They
> gain you very little and cost you a lot (e.g. no CTFE). But unless
> someone were looking to put both .di and .d files for the same module
> on the same system, it wouldn't really be an issue to put them both
> in the include directory. I would expect open source librares to
> use .di files very rarely though.
> 
> - Jonathan M Davis
> 

I'd think of .d files as a superset of .di files. There's no reason to install both (assuming same library version) and having the .d files will produce better cross-module inlining so these are preferred.

Of course for proprietary libraries .di files are required. But nevertheless .di and .d can be mixed in /usr/include/d.

The only downside is that inlining from /usr/include could cause licensing problems (e.g. LGPL allows only linking, not sure how inlining would be affected. But it seems they have an Exception for inlining)


-- Johannes

On Tuesday, 11 April 2017 at 21:08:05 UTC, Johannes Pfau wrote:
> I'd think of .d files as a superset of .di files.

Note that there is zero difference between them. The compiler treats both *exactly* the same way (it just considers the extensions to be aliases of each other).

It is just a naming convention.

On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
> Last time someone else looked, it seemed like LDC and DMD make use of SOVERSION, but do so in an incorrect manner.

You know what exactly is the problem? Any suggestion what to use instead?

It's currently using libphobos2.so.0.74, where using major version 0 to mean unstable might be a misapplication of semver (http://semver.org/#spec-item-4) for SONAME.
Our point releases might also contain small ABI incompatibilities, so they aren't really eligible as patch version.

https://github.com/dlang/phobos/blob/c648097621db225aa59e07d775f54dbaa847c68a/posix.mak#L151

Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code@dawg.eu>:

> On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:
> > Last time someone else looked, it seemed like LDC and DMD make use of SOVERSION, but do so in an incorrect manner.
> 
> You know what exactly is the problem? Any suggestion what to use instead?
> 
> It's currently using libphobos2.so.0.74, where using major version 0 to mean unstable might be a misapplication of semver (http://semver.org/#spec-item-4) for SONAME.

I still haven't found some definitive documentation about this, but it seems linux shared library working essentially works like this:

There's a major and a minor number. There's sometimes a patch version,
but there's no conclusive documentation. Reading the ldconfig source
code you can have as many version levels as you want [3] (though
essentially this is treated like one minor version, it only affects
comparing minor versions). But there are different ldconfig
implementations (linux, bsd) so I don't know if multi-level
minor versions are portable.

* Major version reflects ABI level. A new major version can break ABI
  or add new stuff to ABI.
* Minor versions can only extend the ABI of the major version but
  should not break any ABI.
* Micro / Patch version is mostly unused in ldconfig. It only affects
  comparing minor versions, i.e. when you have libphobos.so.74.0.1 and
  libphobos.so.74.0.2 ldconfig will link
  libphobos.so.74=>libphobos.so.74.0.2

Filename format: lib[name].so.[major].[minor][.patch] Soname format: always lib[name].so.[major]

It is possible to install and use multiple major versions. Every major version will always use the last installed minor version. (The distribution will manage symlinks for libfoo.[major] to libfoo.[major].[minor] for the largest minor version). Additionally a libfoo.so is installed (for linking / development only, might even be in -dev packages) to point to the latest libfoo.[major] symlink.

The libfoo.[major] symlink is used when linking with -lfoo. The dependency encoded in the executable will use the soname though, so it will encode libfoo.[major]. If you install a new major library version, all existing executables will continue to use the major version hardcoded in the executable. If you update a major version to a new minor version, all executables using the major version soname will use the new minor version.

This means:
* Increase major every time you break ABI
* Increase some minor level every time you only extend ABI

So DMD should not keep the major version fixed as 0 (every time you update libphobos you break all existing binaries, as you break the ABI of libphobos.so.0).

In GDC we use libtool which encodes like this [1][2]: libgphobos.so.[major].0.[release] e.g. libgphobos.74.0.2 This is not 100% safe if a minor release breaks ABI though.

BTW: Interestingly even with these complicated rules you can end up in situation where versioning does not work: If you link application APP against libfoo.so on a system with libfoo.so.1.2 the soname will only encode libfoo.so.1. Now ship the library to a system with libfoo.so.1.1 and you may have missing symbols due to extended ABI in libfoo.so.1.2. TLDR: Downgrading libraries is not safe with this versioning approach.



[1] https://autotools.io/libtool/version.html
[2]
https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html

ldconfig is responsible for maintaining the symlinks, here's how it determines which version of a library is newer:

[3] https://github.com/lattera/glibc/blob/master/elf/ldconfig.c#L939 https://github.com/lattera/glibc/blob/master/elf/dl-cache.c#L138

On 11 April 2017 at 23:02, Johannes Pfau via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Am Tue, 11 Apr 2017 14:21:57 +0000
> schrieb Matthias Klumpp <mak@debian.org>:
>
>> can be used by Automake
>> (native),
>
> Do you maintain D support for automake? I wrote some basic D support
> for autoconf and libtool
> (https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/m4) but
> no automake support except for some hacks in
> https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/d_rules.am
>
> I guess I should upstream this some time.
>
> -- Johannes
>

Yeah, you should.  I upstreamed D support in DejaGNU back when I added D tests to GDB.

https://www.mail-archive.com/dejagnu@gnu.org/msg00723.html

Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code@dawg.eu>:

> Our point releases might also contain small ABI incompatibilities, so they aren't really eligible as patch version.

I've actually been hit by this in one point release on Gentoo, where I used dynamic linking for Dlang as soon as it was possible. Programs would suddenly complain about a mismatch between the size of some compiler generated item after a point release update of dmd. Just posting this to give the discussion a bit more substance. :)

-- 
Marco

Am Tue, 11 Apr 2017 07:40:12 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d@puremagic.com>:

> It could always just be distributed as a static library. There arguably isn't much point in distributing it as a shared library anyway - particularly when it's not ABI compatible across versions. I actively avoid having phobos as a shared library on my systems, because it just causes versioning problems when programs are built against it. All of those problems go away with a static library. And so much of Phobos is templated anyway that there isn't even much to share.
> 
> - Jonathan M Davis

I see what you're doing there, but your last point is wishful thinking. Dynamically linked binaries can share megabytes of code. Even Phobos - although heavily templated - has proven to be very amenable to sharing. For example, a "Hello world!" program using `writeln()` has these sizes when compiled with `dmd -O -release -inline`:

     static Phobos2 : 806968 bytes
    dynamic Phobos2 :  18552 bytes

That's about 770 KiB to share or 97.7% of its total size! Awesome!

Another package we have on Gentoo now is the tiling terminal emulator Tilix, compiled with its default `dmd -O`:

     static Phobos2+GtkD : 14828272 bytes
    dynamic Phobos2+GtkD :  6126464 bytes

So here we get ~8.3 megabytes in binary size reduction due to sharing. (Though 6 MiB for a terminal is still a lot - are we slim yet? :P )

Some forms of proprietary code and GPL code need clean shared library interfaces, free from templates and attribute inference. I mentioned this before: As far as feasible have full attribute inference for non-exported functions and reliance on only explicit attributes for exported functions, so that code changes can't break the public API (due to mangling changes).

When Dlang's ABI changes, release a major version and make a public announcement so that everyone can schedule the system update to a convenient time as is done with GCC C++ ABI changes.

-- 
Marco

Am Tue, 11 Apr 2017 15:03:36 +0000
schrieb qznc <qznc@web.de>:

> On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:
> > But if we just use dub - which _is_ the official packaging and build tool - then we avoid these issues. Ideally, the compiler and dub would be part of the distro, but libraries don't need to be. And it sounds like that's basically how the Go and Rust folks want to function as well. So, it would make sense for these languages to simply not have their libraries be included in distros. The build tools are plenty.
> 
> This is not compatible with Debian. Debian requires to include *everything*. You must be able to build every package without network access only from source packages.
> 
> Essentially, somebody must fix this dub issue: https://github.com/dlang/dub/issues/838

I created a similar issue 3 years ago: https://github.com/dlang/dub/issues/342

A key point is how environment variables that everyone agreed upon made "make" a flexible build tool, being able to use a user specified compiler, installation directory, compiler flags and more. This flexibility is relied on by Linux packaging tools to produce binaries for a specific target.

Then there is the problem that at some point you rely on non-Dlang libraries that only the system package manager can provide and update. In the past this always ended in the system package manager winning over the language tool as the common program to update and clean the system.

Dicebot and bioinfornatics (Fedora) felt that dub was for
developers and wasn't tailored to system administration at
that time. p0nce, s-ludwig and markos (Debian) also gave input
and the net result was:

A Makefile generator for dub shall be written that exposes the desired environment variables and can be used by existing Linux package managers.

-- 
Marco

On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
>      static Phobos2 : 806968 bytes
>     dynamic Phobos2 :  18552 bytes
>
> That's about 770 KiB to share or 97.7% of its total size! Awesome!

By the way, using LDC: 402736 bytes for the static build (Linux x86_64). ;)

 — David

On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:
>
> I see what you're doing there, but your last point is wishful thinking. Dynamically linked binaries can share megabytes of code. Even Phobos - although heavily templated - has proven to be very amenable to sharing. For example, a "Hello world!" program using `writeln()` has these sizes when compiled with `dmd -O -release -inline`:
>
>      static Phobos2 : 806968 bytes
>     dynamic Phobos2 :  18552 bytes
>
> That's about 770 KiB to share or 97.7% of its total size! Awesome!

Is all of that active code, or is some of that (statically knowable) never getting executed (as in could've been removed at compile/link time)?