Hi,

I'm trying to build and package my D application using the Nix "purely functional" package manager (https://nixos.org/nix/).

A LITTLE INTRODUCTION

This package manager determines a unique hash from everything that goes into a build cycle (build scripts, sources, versions of dependencies, etc) and builds every derivation into a directory identified by that hash. Conversely, if a directory with that hash already exists, the package doesn't need to be built because it's up-to-date, by definition.

Because those hashes are very unlikely to occur by chance, the package manager crawls all files in the newly built directory for references to hashes of the input dependencies, and records those as run-time dependencies.

It's then possible to zip up a single package and all of its transitive run time dependencies, using the command nix-copy-closure (which for example includes the exact glibc the application was built against), and deploy it in a minimal busybox environment, for example.

MY ISSUE

This is all just a big preamble for my question: apparently, DMD leaves references to the compiler directory in a compiled binary. This causes the "dependency crawler" process to register DMD, and in turn all of _its_ dependencies, as run-time dependencies of my application, which is adding ~100MB to the size of my package bundle. Which, obviously, should not be necessary.

I can tell using 'strings' that they're all references to include files:

    vagrant@vagrant-ubuntu-utopic-64:~$ strings webapp | grep dmd

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d-mixin-736

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/uni.d

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/core/demangle.d

/nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d

But other than that, I'm at a loss as to what these strings are for, and more importantly, how I can get rid of them to cut my run-time dependency set.

Any ideas?

Thanks.

I bet it's debug info.

How do you build your application (command line parameters)? Did you try stripping your binary?

-Steve

On Jan 5, 2015, at 2:16 PM, Rico Huijbers via dmd-internals <dmd-internals@puremagic.com> wrote:

> Hi,
> 
> I'm trying to build and package my D application using the Nix "purely functional" package manager (https://nixos.org/nix/).
> 
> A LITTLE INTRODUCTION
> 
> This package manager determines a unique hash from everything that goes into a build cycle (build scripts, sources, versions of dependencies, etc) and builds every derivation into a directory identified by that hash. Conversely, if a directory with that hash already exists, the package doesn't need to be built because it's up-to-date, by definition.
> 
> Because those hashes are very unlikely to occur by chance, the package manager crawls all files in the newly built directory for references to hashes of the input dependencies, and records those as run-time dependencies.
> 
> It's then possible to zip up a single package and all of its transitive run time dependencies, using the command nix-copy-closure (which for example includes the exact glibc the application was built against), and deploy it in a minimal busybox environment, for example.
> 
> MY ISSUE
> 
> This is all just a big preamble for my question: apparently, DMD leaves references to the compiler directory in a compiled binary. This causes the "dependency crawler" process to register DMD, and in turn all of _its_ dependencies, as run-time dependencies of my application, which is adding ~100MB to the size of my package bundle. Which, obviously, should not be necessary.
> 
> I can tell using 'strings' that they're all references to include files:
> 
>     vagrant@vagrant-ubuntu-utopic-64:~$ strings webapp | grep dmd
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d-mixin-736
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/uni.d
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/core/demangle.d
>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
> 
> But other than that, I'm at a loss as to what these strings are for, and more importantly, how I can get rid of them to cut my run-time dependency set.
> 
> Any ideas?
> 
> Thanks.
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

Yes, I’m also thinking of some kind of debug info, but I’m not sure what kind.

I’m using DUB, compiling with —build=release, which leads to the following command lines:

    dmd -lib -of... -release -inline -O -w -version=... -I... ...files...   # Compiling vibe.d
    dmd -c -of... -release -inline -O -w -version=... -I... -Jviews ...files...  #  Compiling my app
    dmd -of... app.o libvibe-d.a -L--no-as-needed -L-levent -L-levent_pthreads -L-lssl -L-lcrypto  # Linking

I’ve already tried ‘strip’ (should have mentioned that), but it makes no difference.

Cheers,
Rico

On 05 Jan 2015, at 21:17, Steven Schveighoffer via dmd-internals <dmd-internals@puremagic.com> wrote:

> I bet it's debug info.
> 
> How do you build your application (command line parameters)? Did you try stripping your binary?
> 
> -Steve
> 
> On Jan 5, 2015, at 2:16 PM, Rico Huijbers via dmd-internals <dmd-internals@puremagic.com> wrote:
> 
>> Hi,
>> 
>> I'm trying to build and package my D application using the Nix "purely functional" package manager (https://nixos.org/nix/).
>> 
>> A LITTLE INTRODUCTION
>> 
>> This package manager determines a unique hash from everything that goes into a build cycle (build scripts, sources, versions of dependencies, etc) and builds every derivation into a directory identified by that hash. Conversely, if a directory with that hash already exists, the package doesn't need to be built because it's up-to-date, by definition.
>> 
>> Because those hashes are very unlikely to occur by chance, the package manager crawls all files in the newly built directory for references to hashes of the input dependencies, and records those as run-time dependencies.
>> 
>> It's then possible to zip up a single package and all of its transitive run time dependencies, using the command nix-copy-closure (which for example includes the exact glibc the application was built against), and deploy it in a minimal busybox environment, for example.
>> 
>> MY ISSUE
>> 
>> This is all just a big preamble for my question: apparently, DMD leaves references to the compiler directory in a compiled binary. This causes the "dependency crawler" process to register DMD, and in turn all of _its_ dependencies, as run-time dependencies of my application, which is adding ~100MB to the size of my package bundle. Which, obviously, should not be necessary.
>> 
>> I can tell using 'strings' that they're all references to include files:
>> 
>>     vagrant@vagrant-ubuntu-utopic-64:~$ strings webapp | grep dmd
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d-mixin-736
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/uni.d
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/core/demangle.d
>>     /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
>> 
>> But other than that, I'm at a loss as to what these strings are for, and more importantly, how I can get rid of them to cut my run-time dependency set.
>> 
>> Any ideas?
>> 
>> Thanks.
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals@puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
> 
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals


_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

Welp, that's beyond me :) Usually, -release and strip should get rid of all debug information.

You may want to post something to the d.learn forum, here: http://forum.dlang.org/group/digitalmars.D.learn. Many very smart and knowledgeable people are there.

-Steve

On Jan 6, 2015, at 3:20 AM, Rico Huijbers via dmd-internals <dmd-internals@puremagic.com> wrote:

> Yes, I’m also thinking of some kind of debug info, but I’m not sure what kind.
> 
> I’m using DUB, compiling with —build=release, which leads to the following command lines:
> 
>    dmd -lib -of... -release -inline -O -w -version=... -I... ...files...   # Compiling vibe.d
>    dmd -c -of... -release -inline -O -w -version=... -I... -Jviews ...files...  #  Compiling my app
>    dmd -of... app.o libvibe-d.a -L--no-as-needed -L-levent -L-levent_pthreads -L-lssl -L-lcrypto  # Linking
> 
> I’ve already tried ‘strip’ (should have mentioned that), but it makes no difference.
> 
> Cheers,
> Rico
> 
> On 05 Jan 2015, at 21:17, Steven Schveighoffer via dmd-internals <dmd-internals@puremagic.com> wrote:
> 
>> I bet it's debug info.
>> 
>> How do you build your application (command line parameters)? Did you try stripping your binary?
>> 
>> -Steve
>> 
>> On Jan 5, 2015, at 2:16 PM, Rico Huijbers via dmd-internals <dmd-internals@puremagic.com> wrote:
>> 
>>> Hi,
>>> 
>>> I'm trying to build and package my D application using the Nix "purely functional" package manager (https://nixos.org/nix/).
>>> 
>>> A LITTLE INTRODUCTION
>>> 
>>> This package manager determines a unique hash from everything that goes into a build cycle (build scripts, sources, versions of dependencies, etc) and builds every derivation into a directory identified by that hash. Conversely, if a directory with that hash already exists, the package doesn't need to be built because it's up-to-date, by definition.
>>> 
>>> Because those hashes are very unlikely to occur by chance, the package manager crawls all files in the newly built directory for references to hashes of the input dependencies, and records those as run-time dependencies.
>>> 
>>> It's then possible to zip up a single package and all of its transitive run time dependencies, using the command nix-copy-closure (which for example includes the exact glibc the application was built against), and deploy it in a minimal busybox environment, for example.
>>> 
>>> MY ISSUE
>>> 
>>> This is all just a big preamble for my question: apparently, DMD leaves references to the compiler directory in a compiled binary. This causes the "dependency crawler" process to register DMD, and in turn all of _its_ dependencies, as run-time dependencies of my application, which is adding ~100MB to the size of my package bundle. Which, obviously, should not be necessary.
>>> 
>>> I can tell using 'strings' that they're all references to include files:
>>> 
>>>    vagrant@vagrant-ubuntu-utopic-64:~$ strings webapp | grep dmd
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d-mixin-736
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/format.d
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/uni.d
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/core/demangle.d
>>>    /nix/store/3rj0sxwbp2lfddwya2smvkj4cl7y3lvs-dmd-2.066.1/include/d2/std/variant.d
>>> 
>>> But other than that, I'm at a loss as to what these strings are for, and more importantly, how I can get rid of them to cut my run-time dependency set.
>>> 
>>> Any ideas?
>>> 
>>> Thanks.
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals@puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>> 
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals@puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
> 
> 
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals


_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 01/05/2015 08:16 PM, Rico Huijbers via dmd-internals wrote:
>
> But other than that, I'm at a loss as to what these strings are for, and
> more importantly, how I can get rid of them to cut my run-time
> dependency set.
>
> Any ideas?

If you already ran strip -d then those result most likely from exceptions and assertions which sometime carry additional location info.
Not sure how can get rid of those.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 1/7/2015 3:55 PM, Martin Nowak via dmd-internals wrote:
> On 01/05/2015 08:16 PM, Rico Huijbers via dmd-internals wrote:
>>
>> But other than that, I'm at a loss as to what these strings are for, and
>> more importantly, how I can get rid of them to cut my run-time
>> dependency set.
>>
>> Any ideas?
>
> If you already ran strip -d then those result most likely from exceptions and assertions which sometime carry additional location info.
> Not sure how can get rid of those.


These come from assert's, which print out the file/line of the failure. Unfortunately, the file name gets repeated in every generated object file. Those filename strings should be put in COMDATs, which will then result in one instance of the file name string per executable.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

>
> If you already ran strip -d then those result most likely from exceptions
>> and assertions which sometime carry additional location info. Not sure how can get rid of those.
>
>
> These come from assert's, which print out the file/line of the failure. Unfortunately, the file name gets repeated in every generated object file. Those filename strings should be put in COMDATs, which will then result in one instance of the file name string per executable.
>
> And maybe relative file names would make sense?