Hey all,

I'm trying to compile a release-level binary but it looks like the resulting executable has metadata in it that I would like to avoid.  I've tried using both LDC and DMD with the below commands (I've tried many variations of them, you can see the switches are in an attempt to remove this problem):

dmd -m64 -O -J. -release -inline -boundscheck=off -L/SUBSYSTEM:WINDOWS -L/ENTRY:mainCRTStartup .\program.d

and

ldc2.exe --link-internally -O --boundscheck=off --o- --release --relocation-model=pic --static .\program.d


However, after doing those, if you look at the strings in the compiled exe, you'll see things like:

$ strings -a program.exe | grep 'dmd2'
C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
C:\D\dmd2\windows\bin\..\..\src\phobos\std\utf.d
C:\D\dmd2\windows\bin\..\..\src\phobos\std\base64.d


This problem is more egregious when I am using a home folder, like "C:\Users\<my name>\" instead of "C:\D\".  Am I missing something?  Is there a way to compile D without leaking metadata like this in a production release binary?

On Tuesday, 29 December 2020 at 16:13:53 UTC, Raikia wrote:
> Hey all,
>
> [...]
> $ strings -a program.exe | grep 'dmd2'
> C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
> C:\D\dmd2\windows\bin\..\..\src\phobos\std\utf.d
> C:\D\dmd2\windows\bin\..\..\src\phobos\std\base64.d
>
>
> This problem is more egregious when I am using a home folder, like "C:\Users\<my name>\" instead of "C:\D\".  Am I missing something?  Is there a way to compile D without leaking metadata like this in a production release binary?

I believe those are because of Exception constructors that use the __FILE__ special keyword. You might patch the final executable and replace the string content with spaces or 'x's.

On Tuesday, 29 December 2020 at 19:30:53 UTC, Basile B. wrote:
> On Tuesday, 29 December 2020 at 16:13:53 UTC, Raikia wrote:
>> Hey all,
>>
>> [...]
>> $ strings -a program.exe | grep 'dmd2'
>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\utf.d
>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\base64.d
>>
>>
>> This problem is more egregious when I am using a home folder, like "C:\Users\<my name>\" instead of "C:\D\".  Am I missing something?  Is there a way to compile D without leaking metadata like this in a production release binary?
>
> I believe those are because of Exception constructors that use the __FILE__ special keyword. You might patch the final executable and replace the string content with spaces or 'x's.

Interesting.  I was able to clobber it with bbe with no issues.  I'm surprised the compiler doesn't strip out this potentially sensitive metadata, but I guess I'll just patch it out as part of my build process.  Thanks!

On 12/29/20 4:27 PM, Raikia wrote:
> On Tuesday, 29 December 2020 at 19:30:53 UTC, Basile B. wrote:
>> On Tuesday, 29 December 2020 at 16:13:53 UTC, Raikia wrote:
>>> Hey all,
>>>
>>> [...]
>>> $ strings -a program.exe | grep 'dmd2'
>>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
>>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\utf.d
>>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\base64.d
>>>
>>>
>>> This problem is more egregious when I am using a home folder, like "C:\Users\<my name>\" instead of "C:\D\".  Am I missing something?  Is there a way to compile D without leaking metadata like this in a production release binary?
>>
>> I believe those are because of Exception constructors that use the __FILE__ special keyword. You might patch the final executable and replace the string content with spaces or 'x's.
> 
> Interesting.  I was able to clobber it with bbe with no issues. I'm surprised the compiler doesn't strip out this potentially sensitive metadata, but I guess I'll just patch it out as part of my build process.  Thanks!

Yeah, that's a bit surprising. I think it's for use in debuggers so they can pull up the exact file where the exception was thrown.

But I would think a feature should exist that masks the base directory of exception file names.

Probably worth an enhancement request.

-Steve

On Tuesday, 29 December 2020 at 21:27:07 UTC, Raikia wrote:
> Interesting.  I was able to clobber it with bbe with no issues.
>  I'm surprised the compiler doesn't strip out this potentially sensitive metadata, but I guess I'll just patch it out as part of my build process.  Thanks!

Other super safe options are to use either a VM or a CI service to build the releases.

On Tuesday, 29 December 2020 at 23:11:25 UTC, Steven Schveighoffer wrote:
> But I would think a feature should exist that masks the base directory of exception file names.
>
> Probably worth an enhancement request.
>
> -Steve

Also aren't dmd output binaries supposed to be "reproducible" ?

On 12/29/20 7:46 PM, Basile B. wrote:
> On Tuesday, 29 December 2020 at 23:11:25 UTC, Steven Schveighoffer wrote:
>> But I would think a feature should exist that masks the base directory of exception file names.
>>
>> Probably worth an enhancement request.
>>
> 
> Also aren't dmd output binaries supposed to be "reproducible" ?

If you had an option to change __FILE__ to be canonical, then it could be.

i.e. instead of:

C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d

you stored:

(imports)\std\file.d

where (imports) (or maybe some other token) was substituted in for the base of every import. Files passed on the command line would just store the __FILE__ as it was passed.

-Steve

On Wednesday, 30 December 2020 at 01:21:37 UTC, Steven Schveighoffer wrote:
> On 12/29/20 7:46 PM, Basile B. wrote:
>> On Tuesday, 29 December 2020 at 23:11:25 UTC, Steven Schveighoffer wrote:
>>> But I would think a feature should exist that masks the base directory of exception file names.
>>>
>>> Probably worth an enhancement request.
>>>
>> 
>> Also aren't dmd output binaries supposed to be "reproducible" ?
>
> If you had an option to change __FILE__ to be canonical, then it could be.
>
> i.e. instead of:
>
> C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
>
> you stored:
>
> (imports)\std\file.d
>
> where (imports) (or maybe some other token) was substituted in for the base of every import. Files passed on the command line would just store the __FILE__ as it was passed.
>
> -Steve

actually the string "C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d"
looks like it comes from how the ini file variables are expanded.

__FILE__ is not supposed to represent an absolute file name unless the compiler get passed absolute file names. That why __FILE_FULL_PATH__ was added at some point.

On 12/29/20 11:43 PM, Basile B. wrote:
> On Wednesday, 30 December 2020 at 01:21:37 UTC, Steven Schveighoffer wrote:
>> On 12/29/20 7:46 PM, Basile B. wrote:
>>> On Tuesday, 29 December 2020 at 23:11:25 UTC, Steven Schveighoffer wrote:
>>>> But I would think a feature should exist that masks the base directory of exception file names.
>>>>
>>>> Probably worth an enhancement request.
>>>>
>>>
>>> Also aren't dmd output binaries supposed to be "reproducible" ?
>>
>> If you had an option to change __FILE__ to be canonical, then it could be.
>>
>> i.e. instead of:
>>
>> C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d
>>
>> you stored:
>>
>> (imports)\std\file.d
>>
>> where (imports) (or maybe some other token) was substituted in for the base of every import. Files passed on the command line would just store the __FILE__ as it was passed.
>>
> 
> actually the string "C:\D\dmd2\windows\bin\..\..\src\phobos\std\file.d"
> looks like it comes from how the ini file variables are expanded.
> 
> __FILE__ is not supposed to represent an absolute file name unless the compiler get passed absolute file names. That why __FILE_FULL_PATH__ was added at some point.

When I tested this code, it spit out a full path:

import std.stdio;
import std.range;

void main()
{
    char[] str = [0xff, 0xff];
    writeln(str.front);
}

------

std.utf.UTFException@/Users/steves/.dvm/compilers/dmd-2.094.2/osx/bin/../../src/phobos/std/utf.d(1508): Invalid UTF-8 sequence (at index 1)

The path comes from the ini file to generate the import directory. I believe the ini configuration gives a full path (the path to the ini file, and then the relative path to the import).

I think if the import was in the project and not the dmd compiler directory, then full paths wouldn't be happening.

But that doesn't mean you can't FORCE the compiler to behave better via an option switch when importing from elsewhere.

-Steve