On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via Digitalmars-d wrote:
> 
[…]
> At time I am playing around with the idea of using pkg-config[1]
> files to enlist the sources a D library consists of.
> By doing that, we would have a very build-system agnostic way of
> doing storing that information that can be used by Automake
> (native), Dub, Meson and even plain Makefiles. It's also very
> widely used (although it is commonly not used to list sources).

And SCons.

What about CMake? (Which is the CLion build system using Make underneath, but they are rapidly moving to supporting Ninja.)

BTW, I'd like to reiterate that the D support in Meson is great, thanks for getting that  in place. Meson and CMake are the only systems that can install stuff sensibly.

> […]
-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d wrote:
> On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:
> > [...]
> > The problem with /usr/include/d is that is where .di files
> > would be located not .d. This would also match up with the
> > c/c++ usage of it.
>
> When I asked about this a while back, I was told to just install the sources into the include D as "almost nobody uses .di files except for proprietary libraries" (and do those even exist?). But in any case, any path would be fine with me as long as people can settle on using it - `/usr/share/d` would be available ^^

Putting .di files in the include directory makes sense when compared to C++, but it's definitely true that they're used quite rarely. They gain you very little and cost you a lot (e.g. no CTFE). But unless someone were looking to put both .di and .d files for the same module on the same system, it wouldn't really be an issue to put them both in the include directory. I would expect open source librares to use .di files very rarely though.

- Jonathan M Davis

On Tue, 2017-04-11 at 07:40 -0700, Jonathan M Davis via Digitalmars-d wrote:
> On Tuesday, April 11, 2017 14:21:57 Matthias Klumpp via Digitalmars-d wrote:
> > This will *not* solve the issues with Phobos breakage though, as Phobos is a shared library.
> 
> It could always just be distributed as a static library. There
> arguably
> isn't much point in distributing it as a shared library anyway -
> particularly when it's not ABI compatible across versions. I actively
> avoid
> having phobos as a shared library on my systems, because it just
> causes
> versioning problems when programs are built against it. All of those
> problems go away with a static library. And so much of Phobos is
> templated
> anyway that there isn't even much to share.
> 
> - Jonathan M Davis

Having played a bit with GtkD, you always want this as a shared library for development.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:
> But if we just use dub - which _is_ the official packaging and build tool - then we avoid these issues. Ideally, the compiler and dub would be part of the distro, but libraries don't need to be. And it sounds like that's basically how the Go and Rust folks want to function as well. So, it would make sense for these languages to simply not have their libraries be included in distros. The build tools are plenty.

This is not compatible with Debian. Debian requires to include *everything*. You must be able to build every package without network access only from source packages.

Essentially, somebody must fix this dub issue: https://github.com/dlang/dub/issues/838

On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:
> [...]
> Having played a bit with GtkD, you always want this as a shared library for development.

Yeah, GtkD is pretty massive and takes quite a large amount of time to compile... Redoing that for each software depending on it is pretty wasteful.

On Tuesday, 11 April 2017 at 14:43:15 UTC, Russel Winder wrote:
> On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via Digitalmars-d wrote:
>> 
> […]
>> At time I am playing around with the idea of using pkg-config[1]
>> files to enlist the sources a D library consists of.
>> By doing that, we would have a very build-system agnostic way of
>> doing storing that information that can be used by Automake
>> (native), Dub, Meson and even plain Makefiles. It's also very
>> widely used (although it is commonly not used to list sources).
>
> And SCons.
>
> What about CMake? (Which is the CLion build system using Make underneath, but they are rapidly moving to supporting Ninja.)

CMake supports this as well: https://cmake.org/cmake/help/v3.0/module/FindPkgConfig.html

The current idea is in case a library "foobar" would be packaged, to have a "foobar-src.pc" pkgconfig file (additionally to a potentially also existing "foobar.pc" file), containing something like this:

```
prefix=/usr/local
exec_prefix=${prefix}
includedir=${prefix}/include/d/foo

Name: foobar
Description: The foo library (sources)
Version: 1.0.0
Cflags: -I${includedir} -d-version=blahblub
Sources: ${includedir}/alpha.d ${includedir}/beta.d ${includedir}/gamma.d
```

Build systems would then need to explicitly fetch the Sources field and add its expanded values to the project's sources.
(Using Cflags for this would be messy, as the build system might want to deal with flags and sources separately).

Alternatively dub could also define a layout and we write plugins for each build-system to make it work.

This will be really annoying with large libraries like GtkD though, which will require substantially longer to build. Maybe it's worth keeping some libraries precompiled (thereby locking all their reverse-dependencies to whatever D compiler was used to compile the library).

One problem with the pkg-config approach is that to support the precompiled case too, build systems would need to search for both "foobar" and "foobar-src" and only fail dependency detection if both of them are missing.

At least with CMake/Meson that's easy to do though (although it's a bit messy).

On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:
> If you could change the SOVERSION with every one of these changes, or simply just tie it to the respective Phobos release, distributions would automatically do the right thing and compile all D code using Phobos against the new version.

As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you.

(Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.)

 — David

On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:
> On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:
>> If you could change the SOVERSION with every one of these changes, or simply just tie it to the respective Phobos release, distributions would automatically do the right thing and compile all D code using Phobos against the new version.
>
> As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you.
>
> (Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.)

Phobos is versioned properly in both GDC and LDC, and as long as that continues, no problems exist at all :-)
https://packages.debian.org/search?suite=stretch&keywords=phobos

This would - as said - also work for other D shared libraries, unless the D ABI in general breaks or someone tries to build a program with GDC and uses a library that was built with LDC or DMD (or any other possible compiler combination).

On 11 April 2017 at 17:48, Matthias Klumpp via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:
>>
>> On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:
>>>
>>> If you could change the SOVERSION with every one of these changes, or simply just tie it to the respective Phobos release, distributions would automatically do the right thing and compile all D code using Phobos against the new version.
>>
>>
>> As you mention, this is already done in LDC; not just the Debian packages, but also upstream. The soname will always be `libphobos2-ldc.so.74` or what have you.
>>
>> (Thinking about it, it should probably include the patch version as well, although we haven't had a situation so far where we would have wanted to release multiple LDC versions for ABI-incompatible patch releases.)
>
>
> Phobos is versioned properly in both GDC and LDC, and as long as that continues, no problems exist at all :-) https://packages.debian.org/search?suite=stretch&keywords=phobos
>
> This would - as said - also work for other D shared libraries, unless the D ABI in general breaks or someone tries to build a program with GDC and uses a library that was built with LDC or DMD (or any other possible compiler combination).
>

I thought I might try a "will it blend" experiment:

$ cat foo.d
module foo;

int times2(int a)
{
   return a * 2;
}

void throwIt(Throwable t)
{
    throw t;
}

$ cat test.d
import foo;

void main()
{
    import std.stdio;
    writeln("2 x 2 = ", times2(2));

    Throwable t = new Exception("Die!");
    try
    {
        throwIt(t);
    }
    catch (Exception e)
    {
        writeln("Caught Exception for: ", e.msg);
    }
}


1. Naive:

$ dmd -g -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffc1a293000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f0f78bed000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f789e5000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f0f787ce000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f78405000)
    /lib64/ld-linux-x86-64.so.2 (0x0000557aa8c5f000)
$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fff020b9000)
    foo.so => ./foo.so (0x00007f262d66d000)
    libgphobos.so.71 => //usr/lib64/libgphobos.so.71 (0x00007f262cfd7000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007f262ccde000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f262cac8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f262c6d6000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f262c3cc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f262c1af000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f262bfab000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f262bda2000)
    /lib64/ld-linux-x86-64.so.2 (0x0000564d0bc19000)
$ ./a.out
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bc01ae in gc_disable () from ./foo.so

Ouch, DMD's statically linked libphobos has overridden GDC's gc_disable().


2. Mixing shared libs.

$ dmd -g -defaultlib=phobos2 -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffc84dfb000)
--> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74
(0x00007f7574f53000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7574b8a000)
--> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7574985000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f7574768000)
--> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f757445f000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7574256000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f7574040000)
    /lib64/ld-linux-x86-64.so.2 (0x000055c715c3e000)
$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007ffd9e3eb000)
    foo.so => ./foo.so (0x00007fd09d9cc000)
    libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fd09d336000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fd09d03d000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fd09ce27000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd09ca35000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd09c72b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fd09c50e000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd09c30a000)
--> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74
(0x00007fd09bae2000)
    /lib64/ld-linux-x86-64.so.2 (0x000056135abd3000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd09b8da000)
$ ./a.out
Fatal Error while loading '/usr/lib/x86_64-linux-gnu/libphobos2.so.0.74':
    The module 'std.array' is already defined in '/usr/lib64/libgphobos.so.71'.

This seems bad, but it's actually a good sign.  GDC's copy of rt.sections.checkModuleCollisions() has loaded DMD's shared libphobos and has correctly determined that in there exists a module that shares the same name.


3. Telling DMD not to link libphobos

$ dmd -g -defaultlib= -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffdc2155000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44ede1d000)
    /lib64/ld-linux-x86-64.so.2 (0x000055e14d836000)
$ gdc -g test.d foo.so -shared-libphobos
foo.so: undefined reference to `_d_throwdwarf'
collect2: error: ld returned 1 exit status

This is expected, GDC's libgdruntime doesn't have this.  So I added it as a stub:

extern(C) void _d_throwdwarf(Throwable o)
{
   throw o;
}


$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fffb05ee000)
    foo.so => ./foo.so (0x00007fe818644000)
    libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fe817fae000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fe817cb5000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fe817a9f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe8176ad000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe8173a3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fe817186000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe816f82000)
    /lib64/ld-linux-x86-64.so.2 (0x00005644b677f000)
$ ./a.out
2 x 2 = 4
Caught Exception for: Die!


Success!  There may be some hope, but you currently need a compatibility library for linking DMD compiled objects with GDC programs.  And you must cut out Phobos out of the third party D library so that the library for the compiler building the application holds the one and only reference to symbols.

On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:
> Having played a bit with GtkD, you always want this as a shared library for development.

Why would a shared library be preferable to a static library for that (which might still be re-used between different projects, etc.)?

 — David