On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
> On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
>> [...]
>
> The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.
>
> A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.

I think this is a much better idea.

Nic

On Wednesday, 12 August 2015 at 20:39:31 UTC, Ola Fosheim Grøstad wrote:
> On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:
>> Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system.
>> Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.
>
> I think you should forget that there is a filesystem. You may have at least 4 storage areas:
>
> 1. a source bundle
> 2. generated source bundle
> 3. an output bundle
> 4. perhaps a temporary storage area
>
> So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1.
>
> When you reference a module you simply look at 1. first, if it does not exist you try 2.
>
> If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.

I am not sure if I understand your idea correctly:
The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one?
Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?

On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
> On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
>> Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications.
>>
>> http://wiki.dlang.org/DIP81
>>
>> Please discuss!
>
> The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.
>
> A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.

While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible.

Think again about the vibe.d example:
There is one mixin handeling all the template-type combinations.
How would such a mixin be expanded? The resulting code differs from template to template and from type to type.

On Thursday, 13 August 2015 at 00:58:14 UTC, JDemler wrote:
> On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
>> On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
>>> Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications.
>>>
>>> http://wiki.dlang.org/DIP81
>>>
>>> Please discuss!
>>
>> The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.
>>
>> A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
>
> While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible.
>
> Think again about the vibe.d example:
> There is one mixin handeling all the template-type combinations.
> How would such a mixin be expanded? The resulting code differs from template to template and from type to type.

I suppose such an expansion flag would need to expand templates as well, which is still not a bad idea. Templates can be hard to follow sometimes and expanding them out in all their forms could be helpful for debugging and for tooling as well. Though there would be a lot of expansions.

On 13/08/2015 6:37 a.m., JDemler wrote:
> Triggered by the original forum thread I wrote a DIP to further explain
> the idea of the writing files at compile time feature and its implications.
>
> http://wiki.dlang.org/DIP81
>
> Please discuss!

Problem:
	debugging
		The debugger cannot attach to mixedin code; one has to print the resulting code with pragma(msg, ) to inspect it
Fix:
	Make D interface file generation include the source code e.g. statements.
	Pros:
		- Already have most of the code to do this
	Cons:
		- None.

Problem:
	compile-speed
		Even if the resulting code does not change, mixedin code has to be compiled anyway
True. Why is this a problem exactly?

Problem:
	scalability
		If the generated code is needed in two seperated places of the code, it has to be mixedin twice or has to be mixedin into a special module which introduces overhead
Not a problem? The result should *theoretically* the return value should be cached. We could even go so far as to have ``mixin(string)`` as the return type. To cache the AST and force compile time only. Perhaps even make it return the assembly code itself for runtime?

Problem:
	transparency
As user of a library which relys on compile time code generation one has often no idea what code is generated
See debugging problem for the solution.

We do not need to add a new language feature based upon these problems and use cases. We can solve it by simply extending one we already have. D interface files. Making them generate the full source code instead of just the interface.

On 13-Aug-2015 02:27, Tofu Ninja wrote:
> On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
>> Triggered by the original forum thread I wrote a DIP to further
>> explain the idea of the writing files at compile time feature and its
>> implications.
>>
>> http://wiki.dlang.org/DIP81
>>
>> Please discuss!
>
> The benefits of this I see are debugging, actually having the generated
> files makes it much simpler to see what is going wrong. Otherwise the
> utility of this can be achieved with string mixins.

pragma(msg, your_mixin);

>
> A simple alternative to this would be a flag to the compiler to expand
> mixins and output the new files. This would also be great for tooling,
> an IDE could use this and allow you to expand a mixin in place to see
> what it looks like. And currently all the auto complete engines I have
> seen for D don't handle mixins very well. Expanding them would make
> autocompletion a simpler job.

I agree with Ola, something more restricted then filesystem is required.

-- 
Dmitry Olshansky

On 2015-08-13 01:27, Tofu Ninja wrote:

> A simple alternative to this would be a flag to the compiler to expand
> mixins and output the new files. This would also be great for tooling,
> an IDE could use this and allow you to expand a mixin in place to see
> what it looks like. And currently all the auto complete engines I have
> seen for D don't handle mixins very well. Expanding them would make
> autocompletion a simpler job.

I agree, this is much better. I remember the old Eclipse plugin, Descent. It had a compile time view which showed how mixins where expanded and how some language features where lowered, i.e. "scope" to try-catch-finally.

-- 
/Jacob Carlborg

On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole wrote:
> [...]

So in the vibe.d example we would generate a file diet.di which would include all the generated diet templates of the whole project? Maybe 100 * 50 extra lines of code for a medium sized project?
This is not debuggable, understandable or transparent.
Especially because the generated code lies inside a file of a library which I should not need to know about.

Also I do not understand how we could expand template instatiations. Maybe it would be possible for basic templates that only take types but imagine a template that takes a parse tree as a value. How would an expanded version of that look like?

On 13/08/2015 7:57 p.m., JDemler wrote:
> On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole wrote:
>> [...]
>
> So in the vibe.d example we would generate a file diet.di which would
> include all the generated diet templates of the whole project? Maybe 100
> * 50 extra lines of code for a medium sized project?
> This is not debuggable, understandable or transparent.
> Especially because the generated code lies inside a file of a library
> which I should not need to know about.

If it is done via D interface files, you would be in control of where it lands. It would not go into a dub package directory that you do not control for your project.

You want it to be debuggable, this is. Is it nice and pretty? No.
If you are interested in only one code path i.e. specific template arguments to outputted and not the others, then perhaps a pragma can be used to limit it.

> Also I do not understand how we could expand template instatiations.
> Maybe it would be possible for basic templates that only take types but
> imagine a template that takes a parse tree as a value. How would an
> expanded version of that look like?

Template if statement. Also argument overloads.

Also remember, the information must exist to expand it. If it didn't, we sure couldn't generate assembly.

On Thursday, 13 August 2015 at 00:54:37 UTC, JDemler wrote:
> I am not sure if I understand your idea correctly:
> The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one?
> Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?

The filesystem is a compiler issue and not a language issue, so sure, the compiler could do whatever it wants, flush everything to disk or into a SQL database or…

Keep in mind that someone might want to compile on a diskless computer from an in-memory zip-file or similar.

I think the important file generation that are relevant to the language would be for the output bundle, so that you e.g. can have a config file in D and generate platform specific files (.ini, .xml etc).