Hi,

Why shared library size is so big ?

I tried to build https://github.com/etcimon/libasync as a shared library:


build/libasync.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4df6b6d12da6d6cf2108178a14b61fc9453a8e9e, stripped

2,6M	build/libasync.so

ldd build/libasync.so
	linux-vdso.so.1 (0x00007ffd65d54000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f5181e94000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f5181c7d000)
	libphobos2-ldc.so.74 => /lib64/libphobos2-ldc.so.74 (0x00007f51815b3000)
	libdruntime-ldc.so.74 => /lib64/libdruntime-ldc.so.74 (0x00007f51812a3000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f518109f000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5180e81000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f5180b36000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5180780000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5182527000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5180569000)

Its just 16K lines of codes, why 2.6M ? Do you have tips for reduce this size ?

Thanks

On Monday, 2 July 2018 at 09:35:53 UTC, boolangery wrote:
> Hi,
>
> Why shared library size is so big ?
>
> I tried to build https://github.com/etcimon/libasync as a shared library:
>
>
> build/libasync.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4df6b6d12da6d6cf2108178a14b61fc9453a8e9e, stripped
>
> 2,6M	build/libasync.so
>
> ldd build/libasync.so
> 	linux-vdso.so.1 (0x00007ffd65d54000)
> 	librt.so.1 => /lib64/librt.so.1 (0x00007f5181e94000)
> 	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f5181c7d000)
> 	libphobos2-ldc.so.74 => /lib64/libphobos2-ldc.so.74 (0x00007f51815b3000)
> 	libdruntime-ldc.so.74 => /lib64/libdruntime-ldc.so.74 (0x00007f51812a3000)
> 	libdl.so.2 => /lib64/libdl.so.2 (0x00007f518109f000)
> 	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5180e81000)
> 	libm.so.6 => /lib64/libm.so.6 (0x00007f5180b36000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007f5180780000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007f5182527000)
> 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5180569000)
>
> Its just 16K lines of codes, why 2.6M ? Do you have tips for reduce this size ?
>
> Thanks

Hello,

On POSIX by default, all the symbols in a shared library are exported. That makes a lot of symbols that can't be stripped automatically by the linker!

And it won't change anytime soon (https://github.com/ldc-developers/ldc/issues/2431).

The one option you have with LDC to have smaller shared libraries is to use:

$ ldc2 -exported_symbols_list list-of-symbols.lst -dead_strip <rest-of-cmdline>

Good luck!

On Monday, 2 July 2018 at 12:21:04 UTC, Guillaume Piolat wrote:
>
> On POSIX by default, all the symbols in a shared library are exported. That makes a lot of symbols that can't be stripped automatically by the linker!
>
> And it won't change anytime soon (https://github.com/ldc-developers/ldc/issues/2431).
>
> The one option you have with LDC to have smaller shared libraries is to use:
>
> $ ldc2 -exported_symbols_list list-of-symbols.lst -dead_strip <rest-of-cmdline>
>
> Good luck!

hmmm doesn't seem to work for me

ldc2: Unknown command line argument '-exported_symbols_list'.  Try: 'ldc2 -help'
ldc2: Did you mean '-import-instr-limit'?
ldc2: Unknown command line argument '-dead_strip'.  Try: 'ldc2 -help'
ldc2: Did you mean '-defaultlib'?

On Monday, 2 July 2018 at 13:06:12 UTC, boolangery wrote:
> On Monday, 2 July 2018 at 12:21:04 UTC, Guillaume Piolat wrote:
>>
>> On POSIX by default, all the symbols in a shared library are exported. That makes a lot of symbols that can't be stripped automatically by the linker!
>>
>> And it won't change anytime soon (https://github.com/ldc-developers/ldc/issues/2431).
>>
>> The one option you have with LDC to have smaller shared libraries is to use:
>>
>> $ ldc2 -exported_symbols_list list-of-symbols.lst -dead_strip <rest-of-cmdline>
>>
>> Good luck!
>
> hmmm doesn't seem to work for me
>
> ldc2: Unknown command line argument '-exported_symbols_list'.  Try: 'ldc2 -help'
> ldc2: Did you mean '-import-instr-limit'?
> ldc2: Unknown command line argument '-dead_strip'.  Try: 'ldc2 -help'
> ldc2: Did you mean '-defaultlib'?

Those are linker (ld) flags, so use the `-L` prefix with LDC: `-L-exported_symbols_list -L<file.lst> -L-dead_strip`.

On Monday, 2 July 2018 at 15:56:56 UTC, kinke wrote:
> Those are linker (ld) flags, so use the `-L` prefix with LDC: `-L-exported_symbols_list -L<file.lst> -L-dead_strip`.

Thanks you both, is there a way to automate symbol list creation with D public symbols ?

On Tuesday, 3 July 2018 at 06:39:58 UTC, boolangery wrote:
> On Monday, 2 July 2018 at 15:56:56 UTC, kinke wrote:
>> Those are linker (ld) flags, so use the `-L` prefix with LDC: `-L-exported_symbols_list -L<file.lst> -L-dead_strip`.
>
> Thanks you both, is there a way to automate symbol list creation with D public symbols ?

No.
Each of our products has 3 of these symbol lists for POSIX, whereas for Windows you don't need anything.
That's precisely why https://github.com/ldc-developers/ldc/issues/2431 would be helpful.

On Tuesday, 3 July 2018 at 10:38:01 UTC, Guillaume Piolat wrote:
> No.
> Each of our products has 3 of these symbol lists for POSIX, whereas for Windows you don't need anything.
> That's precisely why https://github.com/ldc-developers/ldc/issues/2431 would be helpful.

So I achieved to reduce library size to around 1M, it still too big for my target.

An equivalent library in C++ is about 100K.

I noticed that the .text EFL section of the D one is so huge compared to the C++ one.

On Tuesday, 3 July 2018 at 16:45:19 UTC, boolangery wrote:
> So I achieved to reduce library size to around 1M, it still too big for my target.
>
> An equivalent library in C++ is about 100K.
>
> I noticed that the .text EFL section of the D one is so huge compared to the C++ one.

Unless DMD generates a smaller lib (unlikely IMO), I suggest moving this discussion to somewhere appropriate (general D or libasync if it contains template bloat).

On Monday, 2 July 2018 at 12:21:04 UTC, Guillaume Piolat wrote:
> And it won't change anytime soon (https://github.com/ldc-developers/ldc/issues/2431).

As always, all it takes is someone to step up and either contribute directly by PR or be willing to pay someone else to do it.