September 27, 2021
On Monday, 27 September 2021 at 06:59:32 UTC, Vladimir Panteleev wrote:
> On Sunday, 26 September 2021 at 16:30:04 UTC, Jacob Carlborg wrote:
>> I'm using an approach were I'm cross-compiling on the host and then link and run the tests in a Docker container with the help of QEMU:
>
> Very interesting, thanks! Though I guess linking elsewhere would preclude any kind of LTO :)

I was cross compiling and cross linking on my iMac for ARM with ld.lld, after having scraped only the necessary ubuntu libs and the ARM distribution of ldc.
For linking, it was something like that (copy/pasted from the D build script):


    "/usr/local/opt/llvm/bin/ld.lld ",
    "-o %o ",

    "--sysroot=/ ",
    "--build-id ",
    "--eh-frame-hdr ",
    "-m aarch64linux ",
    "--hash-style=gnu ",
    "-z now ",
    "-z relro ",

    "--lto-O2 ",
    "--gc-sections ",

    "-dynamic-linker /lib/ld-linux-aarch64.so.1 ",

    "-L/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/gcc/aarch64-linux-gnu/7 ",
    "-L/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu ",
    "-L/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib ",
    "-L/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu ",
    "-L/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib ",

    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu/Scrt1.o ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu/crti.o ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/gcc/aarch64-linux-gnu/7/crtbeginS.o ",

    "%f ",

    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/ldc-1.27.1/lib/libphobos2-ldc.a  ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/ldc-1.27.1/lib/libdruntime-ldc.a ",

    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/librt-2.27.so ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/libdl-2.27.so ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/libpthread.so.0 ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu/libpthread_nonshared.a ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/libm.so.6 ",

    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/libc.so.6 ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu/libc_nonshared.a ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1 ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/gcc/aarch64-linux-gnu/7/libgcc.a ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/gcc/aarch64-linux-gnu/7/libgcc_s.so ",

    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/gcc/aarch64-linux-gnu/7/crtendS.o ",
    "/Users/pinver/Projects/Workspace/Prefix/ubuntu_a64/18.10/usr/lib/aarch64-linux-gnu/crtn.o"
```
September 28, 2021

On Monday, 27 September 2021 at 06:59:32 UTC, Vladimir Panteleev wrote:

>

Very interesting, thanks! Though I guess linking elsewhere would preclude any kind of LTO :)

The llvm-lto2 tool provides LTO as an isolated step without linking. If you have it as a cross-compiler.

October 01, 2021

On Monday, 27 September 2021 at 02:27:32 UTC, kinke wrote:

>

Thx, but it doesn't catch my interest - no docker on my machine, plus the qemu AArch64 VM is on a different box. I haven't heard of any other reports of this kind, and as there are AArch64 distro packages of LDC (incl. Android packages which I know are used by some people), I guess I would have heard if such basic failures were wide-spread.

Rented an AWS EC2 ARM box today and tried it out there. The std.process segfaults indeed appear to be specific to the QEMU/binfmt interaction, however, exceptions still did not work. I narrowed it down to -static, confirming your guess. Seems to be an issue with unwinding?

  * frame #0: 0x0000000000518180 btdu-static-aarch64`gsignal + 224
    frame #1: 0x0000000000400434 btdu-static-aarch64`abort + 244
    frame #2: 0x000000000050e1f4 btdu-static-aarch64`uw_init_context_1 + 580
    frame #3: 0x000000000050eb44 btdu-static-aarch64`_Unwind_Backtrace + 84
    frame #4: 0x000000000054d214 btdu-static-aarch64`backtrace + 68
    frame #5: 0x00000000004fb524 btdu-static-aarch64`_D4core7runtime19defaultTraceHandlerFPvZC6object9Throwable9TraceInfo + 116
    frame #6: 0x00000000004fcf08 btdu-static-aarch64`_d_createTrace + 76
    frame #7: 0x00000000004e1a6c btdu-static-aarch64`_d_throw_exception + 176
    frame #8: 0x00000000004399cc btdu-static-aarch64`_D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZNn(file=(length = 39, ptr = u8"/home/ubuntu/btdu/source/btdu/subproc.d"), line=109, msg=(length = 33, ptr = u8"Unexpected subprocess termination")) at exception.d:517:9

To reproduce:

$ echo 'import std.stdio, std.process; void main() { writeln(spawnProcess(["/bin/true"]).wait()); }' > test.d
$ ldc2-1.27.1-linux-aarch64/bin/ldc2 -static test.d
$ ./test
Aborted (core dumped)

Worth upgrading this to an issue?

Also, any ideas for workarounds? (Non-static builds lose most benefits of distributing pre-built binaries in the first place.)

Thanks!

October 01, 2021

On Friday, 1 October 2021 at 11:18:52 UTC, Vladimir Panteleev wrote:

>

To reproduce:

Sorry, wrong code, should be:

>
$ echo 'import std.stdio; void main() { try throw new Exception("Caught"); catch (Exception e) writeln(e); }' > test.d
October 01, 2021

On Friday, 1 October 2021 at 11:18:52 UTC, Vladimir Panteleev wrote:

>

Rented an AWS EC2 ARM box today and tried it out there. The std.process segfaults indeed appear to be specific to the QEMU/binfmt interaction, however, exceptions still did not work. I narrowed it down to -static, confirming your guess. Seems to be an issue with unwinding?

  * frame #0: 0x0000000000518180 btdu-static-aarch64`gsignal + 224
    frame #1: 0x0000000000400434 btdu-static-aarch64`abort + 244
    frame #2: 0x000000000050e1f4 btdu-static-aarch64`uw_init_context_1 + 580
    frame #3: 0x000000000050eb44 btdu-static-aarch64`_Unwind_Backtrace + 84
    frame #4: 0x000000000054d214 btdu-static-aarch64`backtrace + 68
    frame #5: 0x00000000004fb524 btdu-static-aarch64`_D4core7runtime19defaultTraceHandlerFPvZC6object9Throwable9TraceInfo + 116
    frame #6: 0x00000000004fcf08 btdu-static-aarch64`_d_createTrace + 76
    frame #7: 0x00000000004e1a6c btdu-static-aarch64`_d_throw_exception + 176
    frame #8: 0x00000000004399cc btdu-static-aarch64`_D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZNn(file=(length = 39, ptr = u8"/home/ubuntu/btdu/source/btdu/subproc.d"), line=109, msg=(length = 33, ptr = u8"Unexpected subprocess termination")) at exception.d:517:9

Hmm, apparently coming from libbacktrace. IIRC, EH and backtrace code in druntime for Linux AArch64 doesn't really diverge from x64.

I guess you could try compiling druntime manually, defining version (DRuntime_Use_Libunwind) to use libunwind instead of libbacktrace. See https://wiki.dlang.org/Building_LDC_runtime_libraries for how to do that quite painlessly.

October 01, 2021

On Friday, 1 October 2021 at 11:41:06 UTC, kinke wrote:

>

I guess you could try compiling druntime manually, defining version (DRuntime_Use_Libunwind) to use libunwind instead of libbacktrace. See https://wiki.dlang.org/Building_LDC_runtime_libraries for how to do that quite painlessly.

Wow, that looks cool.

version=DRuntime_Use_Libunwind seems to have bitrotten a bit though I think:

core/internal/backtrace/handler.d(89): Error: undefined identifier `Dl_info`
core/internal/backtrace/handler.d(90): Error: undefined identifier `dladdr`
October 01, 2021

On Friday, 1 October 2021 at 11:46:37 UTC, Vladimir Panteleev wrote:

>

version=DRuntime_Use_Libunwind seems to have bitrotten a bit though I think:

Argh, yeah, it's currently only enabled by default for musl, and that isn't CI-tested, going the usual way... ;) - Probably needs to be fixed in upstream druntime.

October 01, 2021

On Friday, 1 October 2021 at 11:49:20 UTC, kinke wrote:

>

On Friday, 1 October 2021 at 11:46:37 UTC, Vladimir Panteleev wrote:

>

version=DRuntime_Use_Libunwind seems to have bitrotten a bit though I think:

Argh, yeah, it's currently only enabled by default for musl, and that isn't CI-tested, going the usual way... ;) - Probably needs to be fixed in upstream druntime.

I think there's more. I added the import and the runtime built, however, the application now fails to link because it can't find unw_getcontext etc. However, looking in /usr/include, I see:

#define unw_getcontext(uc)              unw_tdep_getcontext(uc)

This include file is in /usr/include/aarch64-linux-gnu, which I guess might mean that the libunwind ABI interface differs across architectures :(

October 01, 2021

On Friday, 1 October 2021 at 11:57:53 UTC, Vladimir Panteleev wrote:

>

I think there's more. I added the import and the runtime built, however, the application now fails to link because it can't find unw_getcontext etc. However, looking in /usr/include, I see:

#define unw_getcontext(uc)              unw_tdep_getcontext(uc)

This include file is in /usr/include/aarch64-linux-gnu, which I guess might mean that the libunwind ABI interface differs across architectures :(

Argh... and that might be yet another macro, and possibly even buggy if not from a pretty recent libunwind version: https://github.com/libunwind/libunwind/pull/221/files

[For 32-bit ARM, we have this to account for libunwind macro trouble: https://github.com/ldc-developers/druntime/blob/ldc/src/ldc/arm_unwind.c]

October 01, 2021

On Friday, 1 October 2021 at 11:18:52 UTC, Vladimir Panteleev wrote:

>

Also, any ideas for workarounds? (Non-static builds lose most benefits of distributing pre-built binaries in the first place.)

I guess going down the rabbit hole all the way isn't really what you're looking for. ;) - I'd probably settle for dynamically linking against an old glibc, and only linking the other libs statically. Won't work on your phone, but I guess btrfs isn't really a thing on phones. :]