Thread overview
Meson support for Mir and Lubeck
Aug 04, 2018
9il
Aug 05, 2018
Matthias Klumpp
Sep 09, 2018
9il
Sep 10, 2018
Russel Winder
Sep 10, 2018
9il
Sep 10, 2018
Russel Winder
Sep 10, 2018
Matthias Klumpp
Aug 05, 2018
Matthias Klumpp
Sep 09, 2018
9il
August 04, 2018
The Meson Build system [13] support was added to the following Dub packages:

asdf [1] - JSON serialisation library
cblas [2] - CBLAS header for Dlang
lapack [3] - LAPACK header for Dlang
lubeck [4] - High level linear algebra library
mir-algorithm [5] - Core algorithm library and a home for Dlang multidimensional array package - ndslice
mir-blas [6] - ndslice wrapper for CBLAS
mir-lapack [7] - ndslice wrapper for LAPACK
mir-linux-kernel [8] - Linux headers for Dlang
mir-optim [9] - Optimisation framework
mir-random [10] - Advanced random number engines and random distributions

The packages have been tested in a real world project on OSX with `$ meson configure -Ddefault_library=static/`.

They can be included into a meson project using Meson Wrap files in the `subprojects` directory.
See mir-optim project for an example.

This work has been sponsored by Symmetry Investments [11] and Kaleidic Associates [12].

[1]  https://github.com/libmir/asdf
[2]  https://github.com/DlangScience/cblas
[3]  https://github.com/libmir/lapack
[4]  https://github.com/kaleidicassociates/lubeck
[5]  https://github.com/libmir/mir-algorithm
[6]  https://github.com/libmir/mir-blas
[7]  https://github.com/libmir/mir-lapack
[8]  https://github.com/libmir/mir-linux-kernel
[9]  https://github.com/libmir/mir-optim
[10] https://github.com/libmir/mir-random
[11] https://github.com/kaleidicassociates
[12] http://symmetryinvestments.com
[13] http://mesonbuild.com/

Best regards,
Ilya Yaroshenko
August 05, 2018
On Saturday, 4 August 2018 at 18:23:15 UTC, 9il wrote:
> The Meson Build system [13] support was added to the following Dub packages:
>
> asdf [1] - JSON serialisation library
> cblas [2] - CBLAS header for Dlang
> lapack [3] - LAPACK header for Dlang
> lubeck [4] - High level linear algebra library
> mir-algorithm [5] - Core algorithm library and a home for Dlang multidimensional array package - ndslice
> mir-blas [6] - ndslice wrapper for CBLAS
> mir-lapack [7] - ndslice wrapper for LAPACK
> mir-linux-kernel [8] - Linux headers for Dlang
> mir-optim [9] - Optimisation framework
> mir-random [10] - Advanced random number engines and random distributions
> [...]

Neat! This will allow us to package Mir for Debian. At the moment, there is nothing using Mir in Debian (obviously) and nothing using it is planned for packaging, but adding it might be useful still, since it can also - with some limits - be used from non-D code.

I only worry about potential name clashes with Mir (the display server) in Ubuntu ^^
August 05, 2018
On Saturday, 4 August 2018 at 18:23:15 UTC, 9il wrote:
> [...]
> The packages have been tested in a real world project on OSX with `$ meson configure -Ddefault_library=static/`.
> [...]

Hmm, shouldn't this be `meson --default-library=static ..`?
September 09, 2018
On Sunday, 5 August 2018 at 02:42:43 UTC, Matthias Klumpp wrote:
> On Saturday, 4 August 2018 at 18:23:15 UTC, 9il wrote:
>> [...]
>> The packages have been tested in a real world project on OSX with `$ meson configure -Ddefault_library=static/`.
>> [...]
>
> Hmm, shouldn't this be `meson --default-library=static ..`?

Both options are valid, but it seems that your variants works better for projects with third dependencies. Thanks)
September 09, 2018
On Sunday, 5 August 2018 at 02:36:29 UTC, Matthias Klumpp wrote:
> On Saturday, 4 August 2018 at 18:23:15 UTC, 9il wrote:
>> The Meson Build system [13] support was added to the following Dub packages:
>>
>> asdf [1] - JSON serialisation library
>> cblas [2] - CBLAS header for Dlang
>> lapack [3] - LAPACK header for Dlang
>> lubeck [4] - High level linear algebra library
>> mir-algorithm [5] - Core algorithm library and a home for Dlang multidimensional array package - ndslice
>> mir-blas [6] - ndslice wrapper for CBLAS
>> mir-lapack [7] - ndslice wrapper for LAPACK
>> mir-linux-kernel [8] - Linux headers for Dlang
>> mir-optim [9] - Optimisation framework
>> mir-random [10] - Advanced random number engines and random distributions
>> [...]
>
> Neat! This will allow us to package Mir for Debian. At the moment, there is nothing using Mir in Debian (obviously) and nothing using it is planned for packaging, but adding it might be useful still, since it can also - with some limits - be used from non-D code.
>

Looks like that only betterC projects are good enough to become Debian packages. Generally because of the have stable C ABI that does not depend on D compiler version at all.

Maybe we can pack Mir Optim, what is the proper way to do it?
Its features:
 * stable betterC API/ABI, C/C++ headers
 * two times faster then Alglib for multiple (heavy) curve calibration (recent release)
 * allows to use user specific threads through C API, while free version of Alglib is strictly single thread.

> I only worry about potential name clashes with Mir (the display server) in Ubuntu ^^

Interesting, say libmir-optim(-dev) (I don't think we will have a lot of betterC packages)
September 10, 2018
On Sun, 2018-09-09 at 21:22 +0000, 9il via Digitalmars-d-announce wrote:
> 
[…]
> Looks like that only betterC projects are good enough to become Debian packages. Generally because of the have stable C ABI that does not depend on D compiler version at all.

I do not follow the logic here at all. Any and all projects with a CMake,
Meson, SCons, even Make build be Debian packages. Many Debian packages depend
on specific versions of things like GCC runtime or LLVM. The Debian packaging
system allows for many versions of libraries to co-exists. Thus supporting
multiple versions of Druntime and Phobos is possible.

> 
[…]
> > I only worry about potential name clashes with Mir (the display server) in Ubuntu ^^
> 

This is going to be a naming problem. Debian has many of these sort of naming clash and usually it is best for the smaller, newer project to accept that they need to choose a non-clashing name. Recent example the Mu editor, but there are many instances of this. D's Mir needs to choose a name that doesn't clash with Canonical's Mir.


-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk



September 10, 2018
On Monday, 10 September 2018 at 06:31:44 UTC, Russel Winder wrote:
> On Sun, 2018-09-09 at 21:22 +0000, 9il via Digitalmars-d-announce wrote:
>> 
> […]
>> Looks like that only betterC projects are good enough to become Debian packages. Generally because of the have stable C ABI that does not depend on D compiler version at all.
>
> I do not follow the logic here at all. Any and all projects with a CMake,
> Meson, SCons, even Make build be Debian packages. Many Debian packages depend
> on specific versions of things like GCC runtime or LLVM. The Debian packaging
> system allows for many versions of libraries to co-exists. Thus supporting
> multiple versions of Druntime and Phobos is possible.

Interesting, maybe we can go forward with D specific libraries in the future. Is there any D library that is used by application packages?

Mir Optim can be easily used by other libraries and languages, developers don't need to know D at all as well as depend on DRuntime and D compiler.

The problem with Compiler/DRuntime version that it seems like that if, for example one man released library A that is depend on DRuntime v1, and other man released library B that depends on DRuntime v2, how can I use them in my project together if this DRuntimes are not compatible at ABI level? Maybe we can link dynamically them together, but how GC will work then (in case of non BetterC library)?

If a solution of this issue exist, I would be very surprised if it is easy to go solution. betterC libraries with fixed ABI, and C/C++ API looks to me like a right way to develop D packages for Debian.

>> 
> […]
>> > I only worry about potential name clashes with Mir (the display server) in Ubuntu ^^
>> 
>
> This is going to be a naming problem. Debian has many of these sort of naming clash and usually it is best for the smaller, newer project to accept that they need to choose a non-clashing name. Recent example the Mu editor, but there are many instances of this. D's Mir needs to choose a name that doesn't clash with Canonical's Mir.

We can choose other library prefix for packages instead of "mir", say "mirmodule", so it would be libmirmodule-optim. Would this work? I don't have resource to rename all mir and dependencies.
September 10, 2018
On Mon, 2018-09-10 at 07:25 +0000, 9il via Digitalmars-d-announce wrote: […]

> Interesting, maybe we can go forward with D specific libraries in the future. Is there any D library that is used by application packages?

Obvious example is Tilix and GtkD. They are compiled with a given version of LDC (and hence Phobos and Druntime). Even if LDC is upgraded, as it just has been, the Druntime and Phobos required by Tilix and GtkD are still there so nothing breaks.

This doesn't mean it is all plain sailing: trying to use the updated LDC with the not yet updated GtkD leads to two versions of Phobos seeming to be required. Code still works though.

> Mir Optim can be easily used by other libraries and languages, developers don't need to know D at all as well as depend on DRuntime and D compiler.
> 
> The problem with Compiler/DRuntime version that it seems like that if, for example one man released library A that is depend on DRuntime v1, and other man released library B that depends on DRuntime v2, how can I use them in my project together if this DRuntimes are not compatible at ABI level? Maybe we can link dynamically them together, but how GC will work then (in case of non BetterC library)?

Pass. All I have evidence for is that I am compiling D code with LDC 1.11.0 which requires Phobos and Druntime 2.082 linking with GtkD which required Phobos 2.078 and it all just works.

> If a solution of this issue exist, I would be very surprised if it is easy to go solution. betterC libraries with fixed ABI, and C/C++ API looks to me like a right way to develop D packages for Debian.

Possibly, but much better to go with what is simplest and most efficacious for the purpose in the context.  Libraries destined for use by many different languages clearly need to use the C linkage of the moment.

It is mooted that some currently C implemented libraries may be reimplemented in Rust, but will offer a C linkage as nothing else makes sense.

[…]
> 
> We can choose other library prefix for packages instead of "mir", say "mirmodule", so it would be libmirmodule-optim. Would this work? I don't have resource to rename all mir and dependencies.

I wonder if this is all a storm in a teacup. As far as I can tell there is no libmir, libmir-optim, libmir-dev, or libmir-optim-dev on Debian, so get in there quick before something changes.

On Ubuntu the libraries for Mir the Canonical display server are libmiral… or libmirclient… so there is no actual clash of package name as yet.


-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk



September 10, 2018
[ Replying to lots of stuff in one message - evil, I know... ;-) ]

On Sunday, 9 September 2018 at 21:22:42 UTC, 9il wrote:
> On Sunday, 5 August 2018 at 02:36:29 UTC, Matthias Klumpp wrote:
>> [...]
>
> Looks like that only betterC projects are good enough to become Debian packages. Generally because of the have stable C ABI that does not depend on D compiler version at all.

Having a stable API/ABI is a *huge* plus, but we can also deal with pretty much any D library and D project in Debian, as long as it's build process is able to discover local dependencies and doesn't do evil stuff (like connecting to the network or writing to HOME).
It also has to build with LDC or have no D library dependencies and build with GDC (that's a bit oversimplified though).

In practice this means we can have dub projects with zero dependencies (except for the standard library) that build only executables in Debian packages with a bunch of hacks, or have any D project that uses Makefiles/Meson/CMake/Automake.
Mir has the latter now, so that's fine.


On Monday, 10 September 2018 at 06:31:44 UTC, Russel Winder wrote:
> On Sun, 2018-09-09 at 21:22 +0000, 9il via Digitalmars-d-announce wrote:
>> 
> [...] Many Debian packages depend
> on specific versions of things like GCC runtime or LLVM. The Debian packaging
> system allows for many versions of libraries to co-exists. Thus supporting
> multiple versions of Druntime and Phobos is possible.

It is possible, but heavily discouraged, and also not something I would ever even try to do. Debian carries *one* version of Phobos/DRuntime per compiler that we support and that all D code in the archive has to build with.
There may be occasions where multiple versions are in the archive, but that happens only briefly during transitions.
For security reasons (only one thing to patch in stable) and maintainability reasons (the less to maintain the better) anything that doesn't build with the current version of D/Phobos will either be patched upstream or in Debian or dropped from Debian entirely.
So far this case has never happened though, all breakage of D code in Debian was usually addressed pretty quickly - except for strange compiler bugs on less common architectures.


>> 
> […]
>> > I only worry about potential name clashes with Mir (the display server) in Ubuntu ^^
>> 
> This is going to be a naming problem. Debian has many of these sort of naming clash and usually it is best for the smaller, newer project to accept that they need to choose a non-clashing name. Recent example the Mu editor, but there are many instances of this. D's Mir needs to choose a name that doesn't clash with Canonical's Mir.

This was more meant as a joke from my side ;-) Technically, nobody from Canonical bothered to upload a "mir" package to Debian yet, so technically the name is free to grab for us. However, introducing a src:mir package into Debian would either force Ubuntu to adapt and rename their display server source package (to mir-display-server) or not have D's mir for quite a while.
Socially I think claiming the src:mir name isn't the best idea, therefore - it's unfriendly to a derivative.
However, src:libmir is free and that's what we could go with as a pretty good compromise.
As for the binary package names, none of the D-mir names clash with Ubuntu's mir, so that's a non-issue.


On Monday, 10 September 2018 at 07:25:07 UTC, 9il wrote:
> On Monday, 10 September 2018 at 06:31:44 UTC, Russel Winder wrote:
>> On Sun, 2018-09-09 at 21:22 +0000, 9il via [...]
>
> Interesting, maybe we can go forward with D specific libraries in the future. Is there any D library that is used by application packages?

Quite a few. Debian uses GDC and LDC for compiling its D packages. LDC on the amd64, arm64, armhf and i386 architectures and GDC on all other archs.
The respective default D compiler is set via the default-d-compiler metapackage[1].

This means that whenever there's a new compiler release, ABI gets broken "thanks" to D's unstable ABI. Fortunately all new compilers also come with a new Phobos shared library that has had its SONAME bumped. Which means we can track which D compiler a package was built with via the phobos/druntime libraries they link to.
This enables us to rebuild dependencies in order via a regular library transition, coincidentally one is going on right now. See[2] and tick "Good" to see all D packages that are part of this transition.

This results in the fun conundrum though that as soon as we have D code that *doesn't* link against druntime or phobos, we can't track whether it was rebuilt with the latest compiler and it might fall through the cracks.
I wonder how much of an issue this is - other languages implemented hacks to deal with this issue by depending on artificial virtual ABI packages denoting the current compiler version.

> Mir Optim can be easily used by other libraries and languages, developers don't need to know D at all as well as depend on DRuntime and D compiler.
>
> The problem with Compiler/DRuntime version that it seems like that if, for example one man released library A that is depend on DRuntime v1, and other man released library B that depends on DRuntime v2, how can I use them in my project together if this DRuntimes are not compatible at ABI level? Maybe we can link dynamically them together, but how GC will work then (in case of non BetterC library)?

In Debian, both libraries and all projects would either build with the latest DRuntime, be adjusted to build with it or be dropped from the distribution if they can't keep up.
So, this issue shouldn't exist.

> If a solution of this issue exist, I would be very surprised if it is easy to go solution. betterC libraries with fixed ABI, and C/C++ API looks to me like a right way to develop D packages for Debian.

It certainly is nicer - but we deal with even worse languages than D when it comes to ABI instability. For example, Haskell and OCaml require a full rebuild of all packages even on minor compiler version changes. Therefore, virtual packages containing checksums of the compiler exist, it's quite insane. These languages are also in an eternal transition[3].
This works, because there are actually bigger teams behind those languages which can handle the work.

Which brings me to a new point:
Adding libraries to Debian for their own sage is sometimes done if someone is using them / is interested in them in itself, but often they get pulled in because another tool that is in Debian needs them.
Mir would be the former case, adding a library without something using it that's also in Debian.
I like the Mir project, but since I am not using it at the moment (as scientist I actually might be tempted to do so in future) and since I already have way too many packages to care for, someone else would need to step up and do the packaging work and - more importantly - the maintenance work.
While the Debian D team[4] officially lists 4 members, it's basically me doing all the work at the moment with Markos helping with LDC maintenance as its original maintainer.
So, tl;dr: I will help with reviewing packages and sponsoring packages into Debian, but in order to get the package in, someone would need to step up and do the work.

FWIW, the Debian D Team is not the only entity withing Debian maintaining D code, the GNOME Team has gtk-d, glib-d and tilix and the Debian Games team also has a bunch of D stuff and a bit is also in the Debian Med/Science team, but the majority of D code is concentrated in the D team.

Cheers,
    Matthias

[1]: https://packages.debian.org/sid/default-d-compiler
[2]: https://release.debian.org/transitions/html/auto-ldc.html
[3]: https://release.debian.org/transitions/html/haskell.html
[4]: https://salsa.debian.org/groups/d-team/

P.S: As for "D libraries in Debian", we have vibe.d, diet-ng, libundead, stdx-allocator, mustache-d, containers, glib-d, gtk-d, libbiod. Quite a bunch, actually!