https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #1 from hsteoh@quickfur.ath.cx ---
Alternatively, DMD needs to invoke gcc with -no-pie when linking D executables.

However, I rather we don't go this route, because this would preclude ASLR (Address Space Layout Randomization) on all generated D executables. In the long run I think this is a bad strategic move.

--

https://issues.dlang.org/show_bug.cgi?id=16794

Jonathan M Davis <issues.dlang@jmdavisProg.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |issues.dlang@jmdavisProg.co
                   |                            |m
         Depends on|                            |15935

--- Comment #2 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
I've tried to bootstrap dmd on (k)ubuntu 16.10 in two different ways:

1. Compiling and installing dmd 2.067.1 and then trying to build 2.071.0 with
it.
2. Compiling 2.071.0 with the ldc currently in the ubunto repos which shows

LDC - the LLVM D compiler (0.17.1):
  based on DMD v2.068.2 and LLVM 3.8.0

as its version.

In both cases, to get dmd building, I've had to add -Wno-narrowing to dmd's src/posix.mak like so

BACK_FLAGS := -I$(ROOT) -I$(TK) -I$(C) -I. -DDMDV2=1 -Wno-narrowing

or dmd failed to compile due to errors related to narrowing conversions. When I built druntime and Phobos, I compiled them with PIC=1. I was able to build dmd 2.067.1 and its corresponding druntime and phobos in this manner (after which I added -fPIC to its dmd.conf). However, whether I built dmd 2.070.0 with dmd 2.067.1 or with ldmd2 -fPIC, I ran into the same build error. dmd 2.070.0 built fine, but its druntime failed with

Internal error: backend/elfobj.c 1027
posix.mak:184: recipe for target 'generated/linux/release/64/libdruntime.a'
failed

which looks like is what is being hit in bug #15935, though there, it's reporting it for dub rather than druntime.

--

https://issues.dlang.org/show_bug.cgi?id=16794

Martin Nowak <code@dawg.eu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |code@dawg.eu

--- Comment #3 from Martin Nowak <code@dawg.eu> ---
See https://issues.dlang.org/show_bug.cgi?id=5278#c32 for a solution, compiling
with -fPIC -defaultlib=libphobos2.so.
I'm not certain whether we should build a static library (intended for static
linking) with position independent code?
Does Ubuntu have static libraries, and if so what's their guideline?

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #4 from hsteoh@quickfur.ath.cx ---
@Martin: that forces you to link with Phobos dynamically.  It doesn't work if you want to statically-link Phobos.

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #5 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
Personally, I actively avoid using the shared version of Phobos, because it makes it very annoying to deal with multiple versions of it, which comes up often when you're actually working on Phobos rather than just installing dmd from the .zip or .deb or whatever. It's also just way nicer to be able to build a binary and not have to worry about rebuilding it because of changes to other stuff on your system. So, regardless of what happens to the actual .deb file, I definitely think that it should be _possible_ to compile phobos as a static library and use it with -fPIC, and right now, as far as I can tell, it isn't - or at least, what you get doesn't work correctly, since the resulting dmd can't build libdruntime.a.

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #6 from hsteoh@quickfur.ath.cx ---
To be frank, versioning problems with shared libraries are the fault of the distributors (i.e., us).  The current way shared libraries work, the only sane setup is to make the shared library name (or more specifically, the version part of it) *unique* across all potentially-incompatible builds, even if they are ultimately built from the same version of the compiler toolchain. And when linking, executables should link to the specific version of the library that they are compiled with, not with a generic name like libphobos.so that will pick up whatever version you currently have installed, because that WILL break as soon as you upgrade the library.  (This is especially bad with our current state of affairs where basically every compiler release is ABI-incompatible with the previous release.)

Currently it seems libphobos.so is versioned by DMD release, which apparently makes sense but actually is not good enough. The version must be bumped for every ABI-incompatible change, not just when the API changes. This includes compiler flags used when building the shared library that may introduce ABI-breakages.

But this approach is also troublesome because it means the .deb's must be built in a way that multiple versions of libphobos.so must be simultaneously installable, otherwise after upgrading to the new release the older libphobos.so is gone and all your old executables won't run anymore. Debian does provide the mechanisms for doing this, but it's pretty involved, and I'm not sure if we have the manpower to babysit a potentially large set of libphobos packages that have to be simultaneously maintained for a good amount of time in order not to break old executables.

In comparisons, static linking looks pretty attractive to me. Especially at this stage in D's development.

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #7 from hsteoh@quickfur.ath.cx ---
As for building libphobos.a with -fPIC, it's definitely possible, as I described. At least, I managed to get it to work on my system.  I'm not sure what is causing problems on your system that makes this impossible.

To recap, what I did to make it work is:

(1) Bootstrap dmd from an older non-PIE dmd installation:
    a) create a wrapper script for gcc/g++ (say, g++wrapper) that invokes it
with -no-pie.
    b) run make in the dmd subdir with HOST_CXX set to this wrapper, and
HOST_DMD set to your bootstrapping compiler. Make sure the bootstrapping
compiler's dmd.conf (preferably dedicated for bootstrapping) that has the right
paths to its corresponding version of druntime/phobos (which, presumably, are
from a non-PIE release).
    c) if you're having trouble bootstrapping dmd in spite of this, you may
want to check if gcc-specific flags like -Wnarrowing are being passed to gcc
when compiling dmd modules. If not, you may need to add a hack to g++wrapper
that echoes "g++" when the command-line contains "-version". This is a hack to
workaround posix.mak bogonity that tries to parse the output of `cc -version`
for identifying signs of gcc, but fails horribly with certain versions of
Debian's gcc distribution.

(2) (Re)build druntime/phobos with the new compiler, with PIC=1 passed to the
makefile to force libphobos.a to be built with -fPIC.

(3) Edit dmd.conf (of the bootstrapped compiler) and add -fPIC to the default
flags.

(4) (optionally) install dmd/libphobos in your system directories.

The new compiler should now be able to successfully compile and link PIE executables that are statically linked to libphobos.a. Note that libphobos.a built this way will be PIC, and may not be linkable to non-PIE executables anymore.

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #8 from hsteoh@quickfur.ath.cx ---
P.S. Note that you have to rebuild *both* druntime and phobos (in that order) with PIC=1, otherwise it may not link successfully. And also, you might want to run `make clean` in both druntime and phobos to make sure stale libraries aren't still lying around, before building with PIC=1. Sometimes the makefile just picks up an older non-PIE object file instead and the link will fail.

--

https://issues.dlang.org/show_bug.cgi?id=16794

--- Comment #9 from Jonathan M Davis <issues.dlang@jmdavisProg.com> ---
@hsteoh Well, as I explained before, when I used either my bootstrapped 2.067.1 or ldc to compile the D toolchain, building druntime failed, with the compiler hitting the backend error in bug# 15935. Now, I bootstrapped it differently than you did, and maybe that's the difference. I don't know. But when I bootstrapped it, I used PIC=1 for druntime and Phobos and dmd.conf had -fPIC in it (though the failure was then with the newly built dmd that had been built with either dmd 2.067.1 or ldc, so dmd.conf wouldn't have mattered for the druntime build).

--

https://issues.dlang.org/show_bug.cgi?id=16794

Martin Nowak <code@dawg.eu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Official .deb packages must |.deb not working on Ubuntu
                   |compile libphobos2.a with   |16.10 because of default
                   |-fPIC                       |PIE linking

--- Comment #10 from Martin Nowak <code@dawg.eu> ---
Let me clarify a few points.

- We're not going to ship PIC for static archives on all platforms. The
performance penalty is too big for that.
  At the moment we only have a single linux build
(https://github.com/dlang/installer/blob/8b3c7efe1e049f73ec5fc526a849904bf80b434d/create_dmd_release/build_all.d#L33),
so custom static PIC libraries for Ubuntu require quite some changes to the
build scripts.

- Static linking partly defeats the purpose of PIE and ASLR. I'm still missing some reference on how the Ubuntu team deals with static libraries.

Using -fno-pie might be the better choice for now.

--