I started poking around and examining the details of how the GNU linker works, to solve some annoying issues with LDC.  In the process I the following things that may be useful low-hanging fruit for reducing binary size:

1.  If you have an ar library of object files, by default no dead code elimination is apparently done within an object file, or at least not nearly as much as one would expect.  Each object file in the ar library either gets pulled in or doesn't.

2.  When something is compiled with -lib, DMD writes libraries with one object file **per function**, to get around this.  GDC and LDC don't.  However, if you compile the object files and then manually make an archive with the ar command (which is common in a lot of build processes, such as gtkD's), this doesn't apply.

3.  The defaults can be overridden if you compile your code with -ffunction-sections and -fdata-sections (DMD doesn't support this, GDC and LDC do) and link with --gc-sections.  -ffunction-sections and -fdata-sections cause each function or piece of static data to be written as its own section in the object file, instead of having one giant section that's either pulled in or not.  --gc-sections garbage collects unused sections, resulting in much smaller binaries especially when the sections are fine-grained.

On one project I'm working on, I compiled all the libs I use with GDC using -ffunction-sections -fdata-sections.  The stripped binary is 5.6 MB when I link the app without --gc-sections, or 3.5 MB with --gc-sections.  Quite a difference.  The difference would be even larger if Phobos were compiled w/ -ffunction-sections and -fdata-sections.  (See https://bitbucket.org/goshawk/gdc/issue/293/ffunction-sections-fdata-sections-for ).

DMD can't compile libraries with -ffunction-sections or -fdata-sections and due to other details of my build process that are too complicated to explain here, the results from DMD aren't directly comparable to those from GDC.  However, --gc-sections reduces the DMD binaries from 11 MB to 9 MB.

Bottom line:  If we want to reduce D's binary size there are two pieces of low-hanging fruit:

1.  Make -L--gc-sections the default in dmd.conf on Linux and probably other Posix OS's.

2.  Add -ffunction-sections and -fdata-sections or equivalents to DMD and compile Phobos with these enabled.  I have no idea how hard this would be, but I imagine it would be easy for someone who's already familiar with object file formats.

> Bottom line:  If we want to reduce D's binary size there are two pieces of low-hanging fruit:
>
> 1.  Make -L--gc-sections the default in dmd.conf on Linux and probably other Posix OS's.
>
> 2.  Add -ffunction-sections and -fdata-sections or equivalents to DMD and compile Phobos with these enabled.  I have no idea how hard this would be, but I imagine it would be easy for someone who's already familiar with object file formats.

Seems like --gc-sections _can_ have its pitfalls:
http://blog.flameeyes.eu/2009/11/21/garbage-collecting-sections-is-not-for-production

Also I read somewhere that --gc-sections isn't always supported (no standard switch or something like that).

I personally see no reason not to use -ffunction-sections and -fdata-sections for compiling phobos though, cause a test with gdc didn't even result in a much bigger lib file, nor did it take significantly longer to compile/link.
That site I linked claims though, that it does mean serious overhead even if --gc-sections is omitted then.
So we have to do tests with huge codebases first.

On Tue, 20 Dec 2011 19:14:03 +0100, dsimcha <dsimcha@yahoo.com> wrote:

> I started poking around and examining the details of how the GNU linker works, to solve some annoying issues with LDC.  In the process I the following things that may be useful low-hanging fruit for reducing binary size:
>
> 1.  If you have an ar library of object files, by default no dead code elimination is apparently done within an object file, or at least not nearly as much as one would expect.  Each object file in the ar library either gets pulled in or doesn't.
>
> 2.  When something is compiled with -lib, DMD writes libraries with one object file **per function**, to get around this.  GDC and LDC don't.  However, if you compile the object files and then manually make an archive with the ar command (which is common in a lot of build processes, such as gtkD's), this doesn't apply.
>
> 3.  The defaults can be overridden if you compile your code with -ffunction-sections and -fdata-sections (DMD doesn't support this, GDC and LDC do) and link with --gc-sections.  -ffunction-sections and -fdata-sections cause each function or piece of static data to be written as its own section in the object file, instead of having one giant section that's either pulled in or not.  --gc-sections garbage collects unused sections, resulting in much smaller binaries especially when the sections are fine-grained.
>
Only newer versions of binutils actually support --gc-sections.
There also was a bug that it clears the EH sections.

> On one project I'm working on, I compiled all the libs I use with GDC using -ffunction-sections -fdata-sections.  The stripped binary is 5.6 MB when I link the app without --gc-sections, or 3.5 MB with --gc-sections.  Quite a difference.  The difference would be even larger if Phobos were compiled w/ -ffunction-sections and -fdata-sections.  (See https://bitbucket.org/goshawk/gdc/issue/293/ffunction-sections-fdata-sections-for ).
>
> DMD can't compile libraries with -ffunction-sections or -fdata-sections and due to other details of my build process that are too complicated to explain here, the results from DMD aren't directly comparable to those from GDC.  However, --gc-sections reduces the DMD binaries from 11 MB to 9 MB.
>
> Bottom line:  If we want to reduce D's binary size there are two pieces of low-hanging fruit:
>
> 1.  Make -L--gc-sections the default in dmd.conf on Linux and probably other Posix OS's.
>
> 2.  Add -ffunction-sections and -fdata-sections or equivalents to DMD and compile Phobos with these enabled.  I have no idea how hard this would be, but I imagine it would be easy for someone who's already familiar with object file formats.

On 12/20/2011 10:14 AM, dsimcha wrote:
> 1. Make -L--gc-sections the default in dmd.conf on Linux and probably other
> Posix OS's.

I tried that years ago, and it created executables that always crashed. I seem to recall that it removed some crucial sections :-)

Maybe things are better now and it will work.

> 2. Add -ffunction-sections and -fdata-sections or equivalents to DMD and compile
> Phobos with these enabled. I have no idea how hard this would be, but I imagine
> it would be easy for someone who's already familiar with object file formats.

I didn't know about those flags.

Am 20.12.2011, 19:14 Uhr, schrieb dsimcha <dsimcha@yahoo.com>:

> I started poking around and examining the details of how the GNU linker works, to solve some annoying issues with LDC.  In the process I the following things that may be useful low-hanging fruit for reducing binary size:
>
> 1.  If you have an ar library of object files, by default no dead code elimination is apparently done within an object file, or at least not nearly as much as one would expect.  Each object file in the ar library either gets pulled in or doesn't.
>
> 2.  When something is compiled with -lib, DMD writes libraries with one object file **per function**, to get around this.  GDC and LDC don't.  However, if you compile the object files and then manually make an archive with the ar command (which is common in a lot of build processes, such as gtkD's), this doesn't apply.
>
> 3.  The defaults can be overridden if you compile your code with -ffunction-sections and -fdata-sections (DMD doesn't support this, GDC and LDC do) and link with --gc-sections.  -ffunction-sections and -fdata-sections cause each function or piece of static data to be written as its own section in the object file, instead of having one giant section that's either pulled in or not.  --gc-sections garbage collects unused sections, resulting in much smaller binaries especially when the sections are fine-grained.
>
> On one project I'm working on, I compiled all the libs I use with GDC using -ffunction-sections -fdata-sections.  The stripped binary is 5.6 MB when I link the app without --gc-sections, or 3.5 MB with --gc-sections.  Quite a difference.  The difference would be even larger if Phobos were compiled w/ -ffunction-sections and -fdata-sections.  (See https://bitbucket.org/goshawk/gdc/issue/293/ffunction-sections-fdata-sections-for ).
>
> DMD can't compile libraries with -ffunction-sections or -fdata-sections and due to other details of my build process that are too complicated to explain here, the results from DMD aren't directly comparable to those from GDC.  However, --gc-sections reduces the DMD binaries from 11 MB to 9 MB.
>
> Bottom line:  If we want to reduce D's binary size there are two pieces of low-hanging fruit:
>
> 1.  Make -L--gc-sections the default in dmd.conf on Linux and probably other Posix OS's.
>
> 2.  Add -ffunction-sections and -fdata-sections or equivalents to DMD and compile Phobos with these enabled.  I have no idea how hard this would be, but I imagine it would be easy for someone who's already familiar with object file formats.

Nice of you to start some discussion on these flags. I use them myself (and a few others that seem to affect code size) in a 'tiny' target inside the D Makefile I use.
Currently it looks like this:

	dmd <sources,directory,bin> -m32 -O -release -noboundscheck -L--strip-all -L-O1 -L-znodlopen -L-znorelro -L--no-copy-dt-needed-entries -L--relax -L--sort-common -L--gc-sections -L-lrt -L--as-needed
	strip <bin> -R .comment -R .note.ABI-tag -R .gnu.hash -R .gnu.version -R .jcr -R .got

That's not even funny, I know :D

On Tuesday, 20 December 2011 at 19:36:19 UTC, Marco Leise wrote:
> Nice of you to start some discussion on these flags. I use them myself (and a few others that seem to affect code size) in a 'tiny' target inside the D Makefile I use.
> Currently it looks like this:
>
> 	dmd <sources,directory,bin> -m32 -O -release -noboundscheck -L--strip-all -L-O1 -L-znodlopen -L-znorelro -L--no-copy-dt-needed-entries -L--relax -L--sort-common -L--gc-sections -L-lrt -L--as-needed
> 	strip <bin> -R .comment -R .note.ABI-tag -R .gnu.hash -R .gnu.version -R .jcr -R .got
>
> That's not even funny, I know :D

How far down do you get in terms of size with this?

Am 20.12.2011, 20:45 Uhr, schrieb Trass3r <un@known.com>:

> On Tuesday, 20 December 2011 at 19:36:19 UTC, Marco Leise wrote:
>> Nice of you to start some discussion on these flags. I use them myself (and a few others that seem to affect code size) in a 'tiny' target inside the D Makefile I use.
>> Currently it looks like this:
>>
>> 	dmd <sources,directory,bin> -m32 -O -release -noboundscheck -L--strip-all -L-O1 -L-znodlopen -L-znorelro -L--no-copy-dt-needed-entries -L--relax -L--sort-common -L--gc-sections -L-lrt -L--as-needed
>> 	strip <bin> -R .comment -R .note.ABI-tag -R .gnu.hash -R .gnu.version -R .jcr -R .got
>>
>> That's not even funny, I know :D
>
> How far down do you get in terms of size with this?


I've just installed 2.057 and there were changes that affect code size, so I'll test it once more:

Simple "Hello, world!"
  dmd-2.057
    normal release build (64-bit):   604_156 bytes
    + in 32-bit                  :   490_033 bytes
    + strip-all                  :   250_008 bytes
    + all the other flags        :   147_256 bytes (-102_752 bytes, ~41%)
  gcc-Version 4.6.1 20110627 (gdc 0.30, using dmd 2.055)
    32-bit release, smallest     :   577_856 bytes

Small GUI application using gtk and cairo
  dmd-2.057
    normal release build (32-bit): 5_832_735 bytes
    + strip-all                  : 2_519_124 bytes
    + all the other flags        : 2_243_240 bytes (-275_884 bytes, ~11%)
  gcc-Version 4.6.1 20110627 (gdc 0.30, using dmd 2.055)
    32-bit release, smallest     : 3_204_832 bytes

On 12/20/11 19:59, Trass3r wrote:
> Seems like --gc-sections _can_ have its pitfalls: http://blog.flameeyes.eu/2009/11/21/garbage-collecting-sections-is-not-for-production
> 
> Also I read somewhere that --gc-sections isn't always supported (no standard switch or something like that).

The scenario in that link apparently involves a hack, where a completely unused symbol
is used to communicate with another program/library (which checks for its presence with
dlsym(3)).
The linker will omit that symbol, as nothing else references it - the solution is to
simply reference it from somewhere. Or explicitly place it in a used section. Or
incrementally link in the unused symbols _after_ the gc pass. Or...

If you use such hacks you have to handle them specially; there's no way for the compiler to magically know which unreferenced symbols are not really unused. (which is also why this optimization isn't very useful for shared libs - every visible symbol has to be assumed used, for obvious reasons)

The one potential problematic case i mentioned in that gdc bug mentioned above is this: If the D runtime (most likely GC) needs to know the start/end of the data and bss sections _and_ does it in a way that can confuse it if some unreferenced parts of these sections disappear and/or are reordered, then turning on the section GC could uncover this bug. From the few simple tests i ran here everything seems to work fine, but I did not check the code to confirm there are no incorrect assumptions present.

> I personally see no reason not to use -ffunction-sections and -fdata-sections for compiling phobos though, cause a test with gdc didn't even result in a much bigger lib file, nor did it take significantly longer to compile/link.

737k -> 320k executable size reduction is a compelling argument.

> That site I linked claims though, that it does mean serious overhead even if --gc-sections is omitted then.

?

> So we have to do tests with huge codebases first.

yes.

artur

> The one potential problematic case i mentioned in that gdc bug mentioned above is this:
> If the D runtime (most likely GC) needs to know the start/end of the data and bss
> sections _and_ does it in a way that can confuse it if some unreferenced parts of these
> sections disappear and/or are reordered, then turning on the section GC could uncover
> this bug. From the few simple tests i ran here everything seems to work fine, but I did
> not check the code to confirm there are no incorrect assumptions present.

The first things to break are module ctors and unittests.

On 12/21/11 10:01, Kagamin wrote:
>> The one potential problematic case i mentioned in that gdc bug mentioned above is this: If the D runtime (most likely GC) needs to know the start/end of the data and bss sections _and_ does it in a way that can confuse it if some unreferenced parts of these sections disappear and/or are reordered, then turning on the section GC could uncover this bug. From the few simple tests i ran here everything seems to work fine, but I did not check the code to confirm there are no incorrect assumptions present.
> 
> The first things to break are module ctors and unittests.

Could you elaborate?

Real code, that i was testing with previously, contained static module ctors which worked, and i just tried a simple two module test w/ multiple shared_[cd]tors + static_[cd]tors + unittests -- all of them are called as expected.

FWIW for this trivial test module which imports only stdio, --gc-sections alone reduces the stripped executable size from 724416 to 249120.

artur