Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications.

http://wiki.dlang.org/DIP81

Please discuss!

On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
> Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications.
>
> http://wiki.dlang.org/DIP81
>
> Please discuss!

1. How do you order imports and exports?

2. How do you support parallel compilation on a cluster?

I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.

I'd prefer it if these were always done as separate steps so it can be more easily integrated into other build scripts. (Build systems and scripts suck btw, but they suck less than the confusion caused when files can be read and written arbitrarily by the compiler, which already has hidden dependencies).

On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:
> I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.

Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.

On Wednesday, 12 August 2015 at 19:12:53 UTC, Ola Fosheim Grøstad wrote:
> On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:
>> I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.
>
> Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.

I see your point. But not using the file system would imply that the debugging and transparency advantages would not work.

Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system.
Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.

Please no.

On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:
> Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system.
> Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.

I think you should forget that there is a filesystem. You may have at least 4 storage areas:

1. a source bundle
2. generated source bundle
3. an output bundle
4. perhaps a temporary storage area

So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1.

When you reference a module you simply look at 1. first, if it does not exist you try 2.

If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.

And for this to work you also need a highly concurrent compiler.

On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:
> I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.
>
> I'd prefer it if these were always done as separate steps so it can be more easily integrated into other build scripts. (Build systems and scripts suck btw, but they suck less than the confusion caused when files can be read and written arbitrarily by the compiler, which already has hidden dependencies).

+100

On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
> Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications.
>
> http://wiki.dlang.org/DIP81
>
> Please discuss!

The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.

A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.