> I've attached a part of how concurrency.d could look like translated to my suggested syntax. It probably contains a lot of errors because did a quick translation and I had some trouble understanding the mixins.
>
Yes, even I couldn't understand them even a week later. Maintenance is a real problem.

My initial reaction is that the proposed syntax helps a bit, but isn't really a game-changer. I will let you know if I change my mind on closer inspection.

-- 
Graham St Jack

Hi,

I'm one of Nemerle developers.

> From an implementation point of view, the differences between Nemerle macros and string mixins are mostly syntactic.

You are deeply mistaken!
I'd advise to study the Nemerle macro-system.

You missed: pattern matching, quasi-quotation, running of fully functional Nemerle code in compile time, access to compiler API in macros and IntelliSense support.

>The one thing about them that I find really impressive is the IDE integration, especially that they got syntax highlighting to work. I don't know they've done that.

"syntax highlighting" ? :)
We have full IntelliSense support: syntax highlighting, code completion, macro
expansion (in tooltips).
We simply use the Nemerle compiler to achive that.

VladD2 wrote:
> Hi,
> 
> I'm one of Nemerle developers.

Cool!

>> From an implementation point of view, the differences between Nemerle
>> macros and string mixins are mostly syntactic.
> 
> You are deeply mistaken!
> I'd advise to study the Nemerle macro-system.
> 
> You missed: pattern matching, 

Yes, you've got me there. I've assumed that pattern matching, while a major feature, is not fundamental to the Nemerle macro-system, but I may be mistaken.

One obvious difference which *is* fundamental is that Nemerle macros allow new syntax.

Am I correct in thinking that Nemerle always requires complete ASTs?
That is, given a name "x", can you access variables "x1", "x2", "x3" ?

> quasi-quotation,

No, that's present in D. It's the primary reason I say that the differences are mostly syntactic, since I see this as THE fundamental feature. You need to be able to reenter the macro system after you have left it. Once you can do that, you can do pretty much anything.

BTW if you argued that D's current syntax is quite horrible, I couldn't disagree with you.

> running of fully functional Nemerle
> code in compile time, 

Yes, but that's a different issue. In D, running code at compile time is  regarded as an aspect of constant-folding, and is not restricted to macros.

> access to compiler API in macros

In practice, has to be a library, right? Otherwise the compiler internals would be exposed? (This is an issue we're struggling with).

Another one of the big differences is that D doesn't allow compile-time code to call external functions. Although it could certainly be done, it raises some big issues. Eg, we cannot assume that the target CPU is the same as the one we're running on. With a JIT compiler, you don't have that problem.

> and IntelliSense support.

I did mention that...

>> The one thing about them that I find really impressive is the IDE
>> integration, especially that they got syntax highlighting to work.
>> I don't know they've done that.
> 
> "syntax highlighting" ? :)
> We have full IntelliSense support: syntax highlighting, code completion, macro
> expansion (in tooltips).

I meant Intellisense, not just syntax highlighting. Though note that you can't do syntax highlighting *perfectly* unless the IDE understands the code.

> We simply use the Nemerle compiler to achive that.

Doesn't leave me any less impressed. <g>

Nick Sabalausky wrote:

> "Jacob Carlborg" <doob@me.com> wrote in message news:iednio$2vj5$1@digitalmars.com...
>>
>> I can't quite visualize how the final code will look like and as you say it's hard without a formal definition of AST macros. But I think somehow it would be possible, I mean instead of combining strings one combine expressions/syntax. But I don't know if it would be possible to combine incomplete expressions.
>>
> 
> One parallel that may or may not be applicable, but might be worth considering, is dynamically building XHTML: Using a string-template system is generally found to work very well, but building it via a DOM (essentially an AST) is often considered a bit of a pain. I guess the simplest take-away would be that string-based approaches may be easier get working well. FWIW.

The newest vb.net has this feature, if I understand you correctly, via xml literals and linq expressions. It is quite convenient.

http://msdn.microsoft.com/en-us/library/bb384460.aspx

Don Wrote:

> VladD2 wrote:
> > Hi,
> > 
> > I'm one of Nemerle developers.
> 
> Cool!
> 
> >> From an implementation point of view, the differences between Nemerle macros and string mixins are mostly syntactic.
> > 
> > You are deeply mistaken!
> > I'd advise to study the Nemerle macro-system.
> > 
> > You missed: pattern matching,
> 
> Yes, you've got me there. I've assumed that pattern matching, while a major feature, is not fundamental to the Nemerle macro-system, but I may be mistaken.
> 
> One obvious difference which *is* fundamental is that Nemerle macros allow new syntax.
> 
> Am I correct in thinking that Nemerle always requires complete ASTs? That is, given a name "x", can you access variables "x1", "x2", "x3" ?
> 
>  > quasi-quotation,
> 
> No, that's present in D. It's the primary reason I say that the differences are mostly syntactic, since I see this as THE fundamental feature. You need to be able to reenter the macro system after you have left it. Once you can do that, you can do pretty much anything.
> 
> BTW if you argued that D's current syntax is quite horrible, I couldn't disagree with you.
> 
> > running of fully functional Nemerle
> > code in compile time,
> 
> Yes, but that's a different issue. In D, running code at compile time is
>   regarded as an aspect of constant-folding, and is not restricted to
> macros.
> 
> > access to compiler API in macros
> 
> In practice, has to be a library, right? Otherwise the compiler internals would be exposed? (This is an issue we're struggling with).
> 
> Another one of the big differences is that D doesn't allow compile-time code to call external functions. Although it could certainly be done, it raises some big issues. Eg, we cannot assume that the target CPU is the same as the one we're running on. With a JIT compiler, you don't have that problem.

Don, can you please elaborate on this point?

Here's my understanding:
The D compiler is run once to both 'execute' compile time code which you refer above as constant folding AND to generate the binary to execute at run-time.

Nemerle separates this into two distinct steps:
1. you compile regular code inside macro definitions into a compiler plugin.
2. when compiling the intended run-time code you need to load the compiled macros from step 1 above on the command line of the compiler.

Since you're talking above about cross-compilation and let's say we run the compiler on X and compile for Y, I see no problem to load precompiled macros for X in order to compile the code for Y. The only limit as far as I can see is that it won't be possible to load macros compiled by the such a cross compiler on X unless as you say JIT is employed. Perhaps it makes sense to disable macro definition in a cross-compilation scenario and only allow usage.

Is there something else that I'm missing here?

> 
> > and IntelliSense support.
> 
> I did mention that...
> 
> >> The one thing about them that I find really impressive is the IDE integration, especially that they got syntax highlighting to work. I don't know they've done that.
> > 
> > "syntax highlighting" ? :)
> > We have full IntelliSense support: syntax highlighting, code completion, macro
> > expansion (in tooltips).
> 
> I meant Intellisense, not just syntax highlighting. Though note that you can't do syntax highlighting *perfectly* unless the IDE understands the code.
> 
> > We simply use the Nemerle compiler to achive that.
> 
> Doesn't leave me any less impressed. <g>

foobar wrote:
> Don Wrote:

>> Another one of the big differences is that D doesn't allow compile-time code to call external functions. Although it could certainly be done, it raises some big issues. Eg, we cannot assume that the target CPU is the same as the one we're running on. With a JIT compiler, you don't have that problem.
> 
> Don, can you please elaborate on this point? 
> 
> Here's my understanding:
> The D compiler is run once to both 'execute' compile time code which you refer above as constant folding AND to generate the binary to execute at run-time. 

> 
> Nemerle separates this into two distinct steps:
> 1. you compile regular code inside macro definitions into a compiler plugin. 2. when compiling the intended run-time code you need to load the compiled macros from step 1 above on the command line of the compiler.
> 
> Since you're talking above about cross-compilation and let's say we run the compiler on X and compile for Y, I see no problem to load precompiled macros for X in order to compile the code for Y. The only limit as far as I can see is that it won't be possible to load macros compiled by the such a cross compiler on X unless as you say JIT is employed. Perhaps it makes sense to disable macro definition in a cross-compilation scenario and only allow usage.

Suppose the pre-compiled code, when run, asks what CPU it's on. What's the answer? Is it X or Y?

Don Wrote:
> Suppose the pre-compiled code, when run, asks what CPU it's on. What's the answer? Is it X or Y?

Current: X
Target: Y

Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic.
In any case macro is a meta-programm wich generate or/and transform code.

VladD2 wrote:
> Don Wrote:
>> Suppose the pre-compiled code, when run, asks what CPU it's on. What's the answer? Is it X or Y?
> 
> Current: X
> Target: Y
> 
> Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic.
> In any case macro is a meta-programm wich generate or/and transform code. 

Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time. My understanding of Nemerle (which is quite likely to be wrong!) is that at least some functions are callable only at compile-time.

I'm also scared of the implications of allowing arbitrary code execution during compilation. Make a typo in your program, and then compilation may wipe files from your hard disk, or corrupt an external database, etc... On some platforms you may be able to sandbox it, but since it's running as part of the compilation process, rather than with the permissions it will eventually have, it just seems like a security nightmare.

All these problems would be reduced somewhat if it were only permissible to call pure functions at compile time. Although even that has issues, since not all compiler platforms may allow calling of shared libraries. If we build that into the language, we'd be cutting ourselves off from those platforms.

I'm not certain that there are any unsurmountable problems, but these issues make me really cautious.

Don Wrote:

> VladD2 wrote:
> > Don Wrote:
> >> Suppose the pre-compiled code, when run, asks what CPU it's on. What's the answer? Is it X or Y?
> > 
> > Current: X
> > Target: Y
> > 
> > Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic.
> > In any case macro is a meta-programm wich generate or/and transform code.
> 
> Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time. My understanding of Nemerle (which is quite likely to be wrong!) is that at least some functions are callable only at compile-time.
> 

I don't see how there needs to be different code to accomplish the above use case. You have a function that tests the hardware it's being run on. When this function is called in the compiler context it would return X, when it's called from the target executable it returns Y.

> I'm also scared of the implications of allowing arbitrary code execution during compilation. Make a typo in your program, and then compilation may wipe files from your hard disk, or corrupt an external database, etc... On some platforms you may be able to sandbox it, but since it's running as part of the compilation process, rather than with the permissions it will eventually have, it just seems like a security nightmare.
> 

This is a void argument since templates are Turing complete.
It's *already* possible to do all of the above in D at compile time, it's just a matter of how much code is required to accomplish this.

> All these problems would be reduced somewhat if it were only permissible to call pure functions at compile time. Although even that has issues, since not all compiler platforms may allow calling of shared libraries. If we build that into the language, we'd be cutting ourselves off from those platforms.
> 
> I'm not certain that there are any unsurmountable problems, but these issues make me really cautious.

On 2010-12-17 00:34, Graham St Jack wrote:
>
>> I've attached a part of how concurrency.d could look like translated
>> to my suggested syntax. It probably contains a lot of errors because
>> did a quick translation and I had some trouble understanding the mixins.
>>
> Yes, even I couldn't understand them even a week later. Maintenance is a
> real problem.
>
> My initial reaction is that the proposed syntax helps a bit, but isn't
> really a game-changer. I will let you know if I change my mind on closer
> inspection.

No it's basically just syntactic sugar.

-- 
/Jacob Carlborg