Hi all,

I am excited to share news about two changes that recently made their way into the development version of LDC, changes that might be interesting for many of you Linux users out there.

The first is that LDC now supports linker-level dead code elimination on Linux. If you happen to be familiar with the -f{function,data}-sections/--gc-sections options of the GNU toolchain, their equivalent is now enabled by default. For a set of small-ish programs that make use of different parts of Phobos, I've seen executable size improvements of close to 4x (!) in my tests. However, as --gc-sections is known to occasionally cause trouble with third-party code that relies on specific linker behavior, this optimization can be disabled with a new LDC switch, -disable-linker-strip-dead.

And secondly, proper support for building druntime/Phobos as shared libraries and loading D shared objects dynamically has now arrived in LDC! As you might be aware, Martin Nowak has spent a considerable amount of effort on adding runtime loading capabilities to DMD and druntime during the last year or so. LDC now offers the same level of functionality, to a good part based on Martin's solid work. To build a copy of LDC with druntime/Phobos as a shared library, which is also required for proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to CMake. Oh, and for now this is Linux-only too, sorry.

Even though I am currently not aware of any remaining issues, there are likely a few rough edges still. To be able to confidently ship these features as part of the next release, we'd very much appreciate early feedback. Just grab LDC from Git master and let us know how things went over on digitalmars.D.ldc. If you run into a specific bug, you can also directly open a ticket at https://github.com/ldc-developers/ldc/issues.

Cheers,
David

On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:
> I am excited to share news about two changes that recently made their way into the development version of LDC, changes that might be interesting for many of you Linux users out there.

Sounds great -- congratulations to all the LDC team!  I'll try and get this installed and see how it goes.

On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:
> The first is that LDC now supports linker-level dead code elimination on Linux. If you happen to be familiar with the -f{function,data}-sections/--gc-sections options of the GNU toolchain, their equivalent is now enabled by default. For a set of small-ish programs that make use of different parts of Phobos, I've seen executable size improvements of close to 4x (!) in my tests.

Matches my experience trying things just now -- 612 KB vs 3.2 MB for one example on my system (the hap.random benchmark executable), 453 KB vs 2.3 MB for another.

No linking problems so far, but then, I haven't exactly tried to stretch things ;-)

Now to try out the shared library support ...

This is all great news, congratulations!

On the gc-sections front, Martin had gotten it to work for DMD on Linux but then had to back it out because it failed with the ld.gold linker.

If it works for ldc with ld.gold, can you please help explain what went wrong for dmd?

On 7/8/2014 1:39 PM, Walter Bright wrote:
> This is all great news, congratulations!
>
> On the gc-sections front, Martin had gotten it to work for DMD on Linux but then
> had to back it out because it failed with the ld.gold linker.
>
> If it works for ldc with ld.gold, can you please help explain what went wrong
> for dmd?

https://github.com/D-Programming-Language/dmd/pull/3715

On Tuesday, 8 July 2014 at 17:54:48 UTC, David Nadlinger wrote:
> And secondly, proper support for building druntime/Phobos as shared libraries and loading D shared objects dynamically has now arrived in LDC! As you might be aware, Martin Nowak has spent a considerable amount of effort on adding runtime loading capabilities to DMD and druntime during the last year or so. LDC now offers the same level of functionality, to a good part based on Martin's solid work. To build a copy of LDC with druntime/Phobos as a shared library, which is also required for proper runtime loading, simply pass -DBUILD_SHARED_LIBS=ON to CMake. Oh, and for now this is Linux-only too, sorry.

Tried building hap.random's benchmarknew with the shared-library build of ldc, and got this error when I tried to run the resulting executable:

Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
	The module 'std.range' is already defined in './benchmarknew'.

On Tuesday, 8 July 2014 at 20:39:34 UTC, Walter Bright wrote:
> This is all great news, congratulations!

Thanks. There was quite a bit of staring at obscure backtraces and object file dumps involved. The most persistent issue was what revealed itself to be a nasty race condition in core.thread.Fiber due to the way GCC and LLVM implement the ELF general-dynamic TLS model only after I had spent almost two days looking for it in the wrong place (quite fittingly, this now is issue #666 on our GitHub tracker). Strangely enough, I really enjoy these kinds of challenges from time to time, though.

> On the gc-sections front, Martin had gotten it to work for DMD on Linux but then had to back it out because it failed with the ld.gold linker.
>
> If it works for ldc with ld.gold, can you please help explain what went wrong for dmd?

As far as I can tell at this point, LDC works equally well with both ld.bfd and ld.gold. Martin and I were discussing some of the specifics at the main --gc-sections ticket already [1], but it boils down to the fact that I chose to implement the ModuleInfo reference pinning in a much less beautiful, but more commonplace way.

Specifically, I didn't even try to utilize R_arch_NONE relocations to fake the section dependencies, as this would have required dropping down to the machine code emission layer of LLVM. Sticking with a solution that is representable in LLVM IR meant that we can continue to use the LLVM-provided command line tools when investigating issues in the optimization/machine code generation pipeline. As a side-effect, there isn't anything special about the emitted code, making it less much probable to hit a weird edge case in one of the linkers. The current solution is ugly, though, and leads to some tens of bytes of extra junk per module linked, so we might eventually replace it with something more sophisticated.

David


[1] https://issues.dlang.org/show_bug.cgi?id=879

On Tuesday, 8 July 2014 at 20:43:19 UTC, Joseph Rushton Wakeling wrote:
> Tried building hap.random's benchmarknew with the shared-library build of ldc, and got this error when I tried to run the resulting executable:
>
> Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
> 	The module 'std.range' is already defined in './benchmarknew'.

As strange as it might sound, I'm actually rather relieved that you ran into this issue. I previously thought Martin's collision detection code was at fault (see https://github.com/D-Programming-Language/druntime/pull/791#issuecomment-48090942), but suddenly couldn't reproduce the issue any longer. I'll look into it.

Oh, and it seems like you might be able to work around it for now by (ironically, given Martin's experiences) using ld.gold instead of ld.bfd.

David

On Tuesday, 8 July 2014 at 22:44:30 UTC, David Nadlinger wrote:
> As strange as it might sound, I'm actually rather relieved that you ran into this issue. I previously thought Martin's collision detection code was at fault (see https://github.com/D-Programming-Language/druntime/pull/791#issuecomment-48090942), but suddenly couldn't reproduce the issue any longer. I'll look into it.

Thanks -- let me know if there's anything you'd like me to test.

> Oh, and it seems like you might be able to work around it for now by (ironically, given Martin's experiences) using ld.gold instead of ld.bfd.

Well, personally I don't have any need for a workaround (it's no problem to keep using static libraries for the moment).  But I'm very happy to keep trying stuff out in the process of resolving this issue.

On Tuesday, 8 July 2014 at 22:44:30 UTC, David Nadlinger wrote:
> On Tuesday, 8 July 2014 at 20:43:19 UTC, Joseph Rushton Wakeling wrote:
>> Tried building hap.random's benchmarknew with the shared-library build of ldc, and got this error when I tried to run the resulting executable:
>>
>> Fatal Error while loading '/opt/ldc/lib/libphobos2-ldc.so.65':
>> 	The module 'std.range' is already defined in './benchmarknew'.
> […]

In an unexpected turn of events, it seems like you can also avoid this problem with ld.bfd by simply building with "-disable-linker-strip-dead".

This isn't supposed to happen. Under any circumstances. At all.

Running the program in a debugger, it seems like the __bss_start/_end symbols are off, just as I encountered previously. In fact, they apparently point to an address range that isn't mapped into the process at all!

I couldn't track down why this is happening yet. Here is something odd I noticed about the bfd-linked binary, though:
---
$ readelf --dyn-syms benchmarknew.bfd

Symbol table '.dynsym' contains 407 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 000000000065c518     0 NOTYPE  LOCAL  DEFAULT   29 _edata
     2: 000000000065d228     0 NOTYPE  LOCAL  DEFAULT   30 _end
     3: 000000000065c518     0 NOTYPE  LOCAL  DEFAULT   30 __bss_start
[…]
---
The __bss_start/_end symbols have somehow turned LOCAL, whereas they should be globally visible to override the ones defined in libdruntime.so. I suspect that this is involved in the nonsensical values of 0x50b4d8/0x50b728 I get inside druntime. Note also that the offset between these two addresses is also not remotely correct.

David