Is it possible to parse a file at compile time without embedding it into the binary?

I have a sort of "configuration" file that defines how to create some objects. I'd like to be able to read how to create them but not have that config file stick around in the binary.

e.g., (simple contrived example follows)

Config.txt
   x, my1
   y, my1
   z, my2


class my1 { }
class my2 { }

void parseConfig(A)
{
    ....
}

void main()
{
   parseConfig('Config.txt') // Effectively creates a mixin that mixes in auto x = new my1; auto y = new my1; auto z = new my2;
}


If parseConfig uses import('Config.txt') then config.txt will end up in the binary which I do not want. It would be easier to be able to use import and strip it out later if possible. Config.txt may contain secure information, which is why is doesn't belong in the binary.

On Monday, 6 June 2016 at 17:31:52 UTC, Pie? wrote:
> Is it possible to parse a file at compile time without embedding it into the binary?
>
> I have a sort of "configuration" file that defines how to create some objects. I'd like to be able to read how to create them but not have that config file stick around in the binary.
>
> e.g., (simple contrived example follows)
>
> Config.txt
>    x, my1
>    y, my1
>    z, my2
>
>
> class my1 { }
> class my2 { }
>
> void parseConfig(A)
> {
>     ....
> }
>
> void main()
> {
>    parseConfig('Config.txt') // Effectively creates a mixin that mixes in auto x = new my1; auto y = new my1; auto z = new my2;
> }
>
>
> If parseConfig uses import('Config.txt') then config.txt will end up in the binary which I do not want. It would be easier to be able to use import and strip it out later if possible. Config.txt may contain secure information, which is why is doesn't belong in the binary.

Most compilers, I believe, will not embed a string if it is not used anywhere at runtime. DMD might not though, I'm not sure.

But reading sensitive data at compile-time strikes me as dangerous, depending on your use case. If you are reading sensitive information at compile time, you are presumably going to include that information in your binary (otherwise why would you read it?), and your binary is not secure.

On Monday, 6 June 2016 at 21:31:32 UTC, Alex Parrill wrote:
> On Monday, 6 June 2016 at 17:31:52 UTC, Pie? wrote:
>> Is it possible to parse a file at compile time without embedding it into the binary?
>>
>> I have a sort of "configuration" file that defines how to create some objects. I'd like to be able to read how to create them but not have that config file stick around in the binary.
>>
>> e.g., (simple contrived example follows)
>>
>> Config.txt
>>    x, my1
>>    y, my1
>>    z, my2
>>
>>
>> class my1 { }
>> class my2 { }
>>
>> void parseConfig(A)
>> {
>>     ....
>> }
>>
>> void main()
>> {
>>    parseConfig('Config.txt') // Effectively creates a mixin that mixes in auto x = new my1; auto y = new my1; auto z = new my2;
>> }
>>
>>
>> If parseConfig uses import('Config.txt') then config.txt will end up in the binary which I do not want. It would be easier to be able to use import and strip it out later if possible. Config.txt may contain secure information, which is why is doesn't belong in the binary.
>
> Most compilers, I believe, will not embed a string if it is not used anywhere at runtime. DMD might not though, I'm not sure.

This doesn't seem to be the case. In a release build, even though I never "use" the string, it is embedded. I guess this is due to not using enum but enum seems to be much harder to work with if not impossible.


> But reading sensitive data at compile-time strikes me as dangerous, depending on your use case. If you are reading sensitive information at compile time, you are presumably going to include that information in your binary (otherwise why would you read it?), and your binary is not secure.

Not necessarily, You chased that rabbit quite far! The data your reading could contain sensitive information only used at compile time and not meant to embed. For example, the file could contain login and password to an SQL database that  you then connect, at compile time and retrieve that information the disregard the password(it is not needed at run time).

On Monday, 6 June 2016 at 21:57:20 UTC, Pie? wrote:
> On Monday, 6 June 2016 at 21:31:32 UTC, Alex Parrill wrote:
>> On Monday, 6 June 2016 at 17:31:52 UTC, Pie? wrote:
>>> Is it possible to parse a file at compile time without embedding it into the binary?
>>>
>>> I have a sort of "configuration" file that defines how to create some objects. I'd like to be able to read how to create them but not have that config file stick around in the binary.
>>>
>>> e.g., (simple contrived example follows)
>>>
>>> Config.txt
>>>    x, my1
>>>    y, my1
>>>    z, my2
>>>
>>>
>>> class my1 { }
>>> class my2 { }
>>>
>>> void parseConfig(A)
>>> {
>>>     ....
>>> }
>>>
>>> void main()
>>> {
>>>    parseConfig('Config.txt') // Effectively creates a mixin that mixes in auto x = new my1; auto y = new my1; auto z = new my2;
>>> }
>>>
>>>
>>> If parseConfig uses import('Config.txt') then config.txt will end up in the binary which I do not want. It would be easier to be able to use import and strip it out later if possible. Config.txt may contain secure information, which is why is doesn't belong in the binary.
>>
>> Most compilers, I believe, will not embed a string if it is not used anywhere at runtime. DMD might not though, I'm not sure.
>
> This doesn't seem to be the case. In a release build, even though I never "use" the string, it is embedded. I guess this is due to not using enum but enum seems to be much harder to work with if not impossible.
>
>
>> But reading sensitive data at compile-time strikes me as dangerous, depending on your use case. If you are reading sensitive information at compile time, you are presumably going to include that information in your binary (otherwise why would you read it?), and your binary is not secure.
>
> Not necessarily, You chased that rabbit quite far! The data your reading could contain sensitive information only used at compile time and not meant to embed. For example, the file could contain login and password to an SQL database that  you then connect, at compile time and retrieve that information the disregard the password(it is not needed at run time).

This is definitely possible, but it can depend on your compiler. If you use an enum, it'll be treated as a compile-time constant - so if you never store it anywhere (i.e. enum File = import('file.txt'); string file = File; is a no-no at global scope), you should be fine.

If you do find yourself in the precarious situation of storing the data, then it's up to your compiler to detect that there are no runtime references to the data and elide it. LDC and GDC most likely do this, but I doubt DMD would.

For safety, you should try and reformulate your code in terms of enums and local variables; this *should* work with DMD, but it's possible it's not smart enough to catch onto the fact that the function is never used at run-time (and therefore does not need to be included in the executable).

On Tuesday, 7 June 2016 at 02:04:41 UTC, Mithun Hunsur wrote:
> On Monday, 6 June 2016 at 21:57:20 UTC, Pie? wrote:
>> On Monday, 6 June 2016 at 21:31:32 UTC, Alex Parrill wrote:
>>> [...]
>>
>> This doesn't seem to be the case. In a release build, even though I never "use" the string, it is embedded. I guess this is due to not using enum but enum seems to be much harder to work with if not impossible.
>>
>>
>>> [...]
>>
>> Not necessarily, You chased that rabbit quite far! The data your reading could contain sensitive information only used at compile time and not meant to embed. For example, the file could contain login and password to an SQL database that  you then connect, at compile time and retrieve that information the disregard the password(it is not needed at run time).
>
> This is definitely possible, but it can depend on your compiler. If you use an enum, it'll be treated as a compile-time constant - so if you never store it anywhere (i.e. enum File = import('file.txt'); string file = File; is a no-no at global scope), you should be fine.
>
> If you do find yourself in the precarious situation of storing the data, then it's up to your compiler to detect that there are no runtime references to the data and elide it. LDC and GDC most likely do this, but I doubt DMD would.
>
> For safety, you should try and reformulate your code in terms of enums and local variables; this *should* work with DMD, but it's possible it's not smart enough to catch onto the fact that the function is never used at run-time (and therefore does not need to be included in the executable).

Ok, I will assume it will be able to be removed for release. It is an easy check(just search if binary contains file info). I'm sure an easy fix could be to write 0's over the data in the binary if necessary.

If I use an enum dmd does *not* remove it in release build. I will work on parsing the file using CTFE and hopefully dmd will not try to keep it around, or it can be solved using gdc/ldc or some other method.

If I use an enum dmd DOES remove it in release build. But I'm not sure for the general case yet.

On Monday, 6 June 2016 at 21:57:20 UTC, Pie? wrote:
> On Monday, 6 June 2016 at 21:31:32 UTC, Alex Parrill wrote:
>> But reading sensitive data at compile-time strikes me as dangerous, depending on your use case. If you are reading sensitive information at compile time, you are presumably going to include that information in your binary (otherwise why would you read it?), and your binary is not secure.
>
> Not necessarily, You chased that rabbit quite far! The data your reading could contain sensitive information only used at compile time and not meant to embed. For example, the file could contain login and password to an SQL database that  you then connect, at compile time and retrieve that information the disregard the password(it is not needed at run time).

Accessing a SQL server at compile time seems like a huge abuse of CTFE (and I'm pretty sure it's impossible at the moment). Why do I need to install and set up a MySQL database in order to build your software?

On Tuesday, 7 June 2016 at 22:09:58 UTC, Alex Parrill wrote:
>
> Accessing a SQL server at compile time seems like a huge abuse of CTFE (and I'm pretty sure it's impossible at the moment). Why do I need to install and set up a MySQL database in order to build your software?

Presumably you wouldn't be building it at all, since this seems like a technique to provide obfuscated binaries where people aren't privvy to exactly what was used to compile it.

On Tuesday, 7 June 2016 at 22:09:58 UTC, Alex Parrill wrote:
> On Monday, 6 June 2016 at 21:57:20 UTC, Pie? wrote:
>> On Monday, 6 June 2016 at 21:31:32 UTC, Alex Parrill wrote:
>>> [...]
>>
>> Not necessarily, You chased that rabbit quite far! The data your reading could contain sensitive information only used at compile time and not meant to embed. For example, the file could contain login and password to an SQL database that  you then connect, at compile time and retrieve that information the disregard the password(it is not needed at run time).
>
> Accessing a SQL server at compile time seems like a huge abuse of CTFE (and I'm pretty sure it's impossible at the moment). Why do I need to install and set up a MySQL database in order to build your software?

Lol, who says you have access to my software? You know, the problem with assumptions is that they generally make no sense when you actually think about them.

On Thursday, 9 June 2016 at 22:02:44 UTC, Joerg Joergonson wrote:
> Lol, who says you have access to my software? You know, the problem with assumptions is that they generally make no sense when you actually think about them.

oh, yeah. it suddenly reminds me about some obscure thing. other people told me that they were able to solve the same problems with something they called "build system"...