On 10/10/2009 00:36, Christopher Wright wrote:
> Yigal Chripun wrote:
>> On 09/10/2009 00:38, Christopher Wright wrote:
>>> It makes macros highly compiler-specific, or requires the compiler's AST
>>> to be part of the language.
>>>
>>> Nemerle took the nuclear option, and its macros are all-powerful. That's
>>> a reasonable way of doing things. I'd be happy with a more restricted
>>> system that's easier to standardize, especially if it got rid of all the
>>> hacky string manipulation in current D metaprogramming. (Seriously, even
>>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
>>
>> It doesn't have to be compiler specific. all is needed is a
>> standardized API to the compiler.
>
> Right. It adds something huge that's normally compiler-specific to the
> language. This makes me uncomfortable. It greatly increases the
> difficulty of implementation.
>
I disagree - a properly designed compiler will have such an API anyway. Look at how Clang is designed - it's a modular compiler where each part has its own library. you can combine its libs in different ways to provide different options: a full command-line compiler, semantic analysis for IDE, incremental builder for an IDE, etc..
that design obviously requires APIs for the different components.


>> What's so hackish about that?
>
> Reread. Current D metaprogramming is hackish. Nemerle's isn't.

I was referring to what Don said that providing a hook into the compiler is hackish.
>
>> many large modular systems do exactly that: eclipse, firefox, even the
>> OS itself. Unix provides syscalls which *are* an API to the OS.
>>
>> a properly designed API doesn't have to expose internal implementation
>> details.
>>
>> btw, in Nemerle they have syntax to compose/decompose AST specifically
>> so they don't need to expose the internal structure of the AST.
>
> So they have a separate object model for the syntax tree that macros can
> affect. This is what I would recommend for D.

What do you mean by object model?

they have a synax to manipulate AST:
<[ some code ]>  would be parsed by the compiler as the AST of "some code" and would be represented internally by the compiler specific AST representation.

This syntax also handles hygiene and provides means to break it when needed.

Yigal Chripun wrote:
> On 10/10/2009 00:36, Christopher Wright wrote:
>> Yigal Chripun wrote:
>>> On 09/10/2009 00:38, Christopher Wright wrote:
>>>> It makes macros highly compiler-specific, or requires the compiler's AST
>>>> to be part of the language.
>>>>
>>>> Nemerle took the nuclear option, and its macros are all-powerful. That's
>>>> a reasonable way of doing things. I'd be happy with a more restricted
>>>> system that's easier to standardize, especially if it got rid of all the
>>>> hacky string manipulation in current D metaprogramming. (Seriously, even
>>>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
>>>
>>> It doesn't have to be compiler specific. all is needed is a
>>> standardized API to the compiler.
>>
>> Right. It adds something huge that's normally compiler-specific to the
>> language. This makes me uncomfortable. It greatly increases the
>> difficulty of implementation.
>>
> I disagree - a properly designed compiler will have such an API anyway. 

Not if you have compilers from different vendors. And that's one of the key problems with making such an API part of language -- the potential for vendor lock-in.

>>> What's so hackish about that?
>>
>> Reread. Current D metaprogramming is hackish. Nemerle's isn't.
> 
> I was referring to what Don said that providing a hook into the compiler is hackish.

I stand by that.
Look, I was Forth guy back in the day. Forth and Lisp both have hack-free macros. Particularly in the case of Forth, the language is largely defined in the library; you can even make the case that the compiler is part of the library. So there's no problem with the library extending the language.
But in the case of Nemerle, it's a conventional compiler with hooks for library code.

I just feel that Nermele's approach is diametrically opposed to Forth/Lisp.
It's personal opinion. To me, that looks like a hack.

To make one thing clear:
D's compile-time reflection is a hack. And that makes most current 'D macros' hackish. I just feel that most of the problems lie on the reflection side.

Yigal Chripun wrote:
> On 10/10/2009 00:36, Christopher Wright wrote:
>> Yigal Chripun wrote:
>>> On 09/10/2009 00:38, Christopher Wright wrote:
>>>> It makes macros highly compiler-specific, or requires the compiler's AST
>>>> to be part of the language.
>>>>
>>>> Nemerle took the nuclear option, and its macros are all-powerful. That's
>>>> a reasonable way of doing things. I'd be happy with a more restricted
>>>> system that's easier to standardize, especially if it got rid of all the
>>>> hacky string manipulation in current D metaprogramming. (Seriously, even
>>>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
>>>
>>> It doesn't have to be compiler specific. all is needed is a
>>> standardized API to the compiler.
>>
>> Right. It adds something huge that's normally compiler-specific to the
>> language. This makes me uncomfortable. It greatly increases the
>> difficulty of implementation.
>>
> I disagree - a properly designed compiler will have such an API anyway. Look at how Clang is designed - it's a modular compiler where each part has its own library. you can combine its libs in different ways to provide different options: a full command-line compiler, semantic analysis for IDE, incremental builder for an IDE, etc..
> that design obviously requires APIs for the different components.
> 
> 
>>> What's so hackish about that?
>>
>> Reread. Current D metaprogramming is hackish. Nemerle's isn't.
> 
> I was referring to what Don said that providing a hook into the compiler is hackish.
>>
>>> many large modular systems do exactly that: eclipse, firefox, even the
>>> OS itself. Unix provides syscalls which *are* an API to the OS.
>>>
>>> a properly designed API doesn't have to expose internal implementation
>>> details.
>>>
>>> btw, in Nemerle they have syntax to compose/decompose AST specifically
>>> so they don't need to expose the internal structure of the AST.
>>
>> So they have a separate object model for the syntax tree that macros can
>> affect. This is what I would recommend for D.
> 
> What do you mean by object model?
> 
> they have a synax to manipulate AST:
> <[ some code ]>  would be parsed by the compiler as the AST of "some code" and would be represented internally by the compiler specific AST representation.

I looked up nemerle macros after this. There are a couple parts.

1. AST Mixins
It's a lot like string mixins with builtin string formatting and automatic conversion of arguments to their string form. Syntactic sugar on top of this. That's all that macros are. Yes, you can manipulate the AST, but at this stage, it's entirely opaque.

2. Compiler plugins
You can define a compiler module that does arbitrary things to the AST. Many modules will make use of macros. The Nemerle compiler might attempt to conflate plugins with macros, but if there were an alternate implementation of Nemerle, the difference would become very apparent very quickly.

AST mixins are sexy. Compiler plugins are also sexy[1], but targeted toward a much different audience. D could benefit from both, but the latter is far lower on the list than a decent compile-time reflection system. And of this, only compiler plugins have the issues that I mentioned earlier.


[1] Unless you're Richard Stallman. Onoz, someone could use a proprietary plugin with GCC!

On 10/10/2009 10:50, Don wrote:
> Yigal Chripun wrote:
>> On 10/10/2009 00:36, Christopher Wright wrote:
>>> Yigal Chripun wrote:
>>>> On 09/10/2009 00:38, Christopher Wright wrote:
>>>>> It makes macros highly compiler-specific, or requires the
>>>>> compiler's AST
>>>>> to be part of the language.
>>>>>
>>>>> Nemerle took the nuclear option, and its macros are all-powerful.
>>>>> That's
>>>>> a reasonable way of doing things. I'd be happy with a more restricted
>>>>> system that's easier to standardize, especially if it got rid of
>>>>> all the
>>>>> hacky string manipulation in current D metaprogramming. (Seriously,
>>>>> even
>>>>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
>>>>
>>>> It doesn't have to be compiler specific. all is needed is a
>>>> standardized API to the compiler.
>>>
>>> Right. It adds something huge that's normally compiler-specific to the
>>> language. This makes me uncomfortable. It greatly increases the
>>> difficulty of implementation.
>>>
>> I disagree - a properly designed compiler will have such an API anyway.
>
> Not if you have compilers from different vendors. And that's one of the
> key problems with making such an API part of language -- the potential
> for vendor lock-in.

if each compiler has its own API than you're correct but what I was talking about was a standard API that is part of the stdlib which the different vendors need to implement in order to be considered compliant to the language spec. the compiler internals need not be identical only the API as defined in the spec.

>
>>>> What's so hackish about that?
>>>
>>> Reread. Current D metaprogramming is hackish. Nemerle's isn't.
>>
>> I was referring to what Don said that providing a hook into the
>> compiler is hackish.
>
> I stand by that.
> Look, I was Forth guy back in the day. Forth and Lisp both have
> hack-free macros. Particularly in the case of Forth, the language is
> largely defined in the library; you can even make the case that the
> compiler is part of the library. So there's no problem with the library
> extending the language.
> But in the case of Nemerle, it's a conventional compiler with hooks for
> library code.

I don't know how deep you looked into Nemerle, but from my understanding that description is false. Nemerle is much closer to your description of Forth than you'd think.
Nemerle supports syntax extensions and parts of the language are already implemented as macros.
They are now considering to generalize this construct further so they could implement more of Nemerle as Macros.

>
> I just feel that Nermele's approach is diametrically opposed to Forth/Lisp.
> It's personal opinion. To me, that looks like a hack.
>
> To make one thing clear:
> D's compile-time reflection is a hack. And that makes most current 'D
> macros' hackish. I just feel that most of the problems lie on the
> reflection side.

Yigal Chripun wrote:
> On 10/10/2009 10:50, Don wrote:
>> Yigal Chripun wrote:
>>> On 10/10/2009 00:36, Christopher Wright wrote:
>>>> Yigal Chripun wrote:
>>>>> On 09/10/2009 00:38, Christopher Wright wrote:
>>>>>> It makes macros highly compiler-specific, or requires the
>>>>>> compiler's AST
>>>>>> to be part of the language.
>>>>>>
>>>>>> Nemerle took the nuclear option, and its macros are all-powerful.
>>>>>> That's
>>>>>> a reasonable way of doing things. I'd be happy with a more restricted
>>>>>> system that's easier to standardize, especially if it got rid of
>>>>>> all the
>>>>>> hacky string manipulation in current D metaprogramming. (Seriously,
>>>>>> even
>>>>>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
>>>>>
>>>>> It doesn't have to be compiler specific. all is needed is a
>>>>> standardized API to the compiler.
>>>>
>>>> Right. It adds something huge that's normally compiler-specific to the
>>>> language. This makes me uncomfortable. It greatly increases the
>>>> difficulty of implementation.
>>>>
>>> I disagree - a properly designed compiler will have such an API anyway.
>>
>> Not if you have compilers from different vendors. And that's one of the
>> key problems with making such an API part of language -- the potential
>> for vendor lock-in.
> 
> if each compiler has its own API than you're correct but what I was talking about was a standard API that is part of the stdlib which the different vendors need to implement in order to be considered compliant to the language spec. the compiler internals need not be identical only the API as defined in the spec.
> 
>>
>>>>> What's so hackish about that?
>>>>
>>>> Reread. Current D metaprogramming is hackish. Nemerle's isn't.
>>>
>>> I was referring to what Don said that providing a hook into the
>>> compiler is hackish.
>>
>> I stand by that.
>> Look, I was Forth guy back in the day. Forth and Lisp both have
>> hack-free macros. Particularly in the case of Forth, the language is
>> largely defined in the library; you can even make the case that the
>> compiler is part of the library. So there's no problem with the library
>> extending the language.
>> But in the case of Nemerle, it's a conventional compiler with hooks for
>> library code.
> 
> I don't know how deep you looked into Nemerle, but from my understanding that description is false. Nemerle is much closer to your description of Forth than you'd think.
> Nemerle supports syntax extensions and parts of the language are already implemented as macros.
> They are now considering to generalize this construct further so they could implement more of Nemerle as Macros.

Ah, OK. My cursory glance at Nemerle just screamed "hack". But first impressions can be misleading.
No doubt as a C-family language, they have some useful ideas.
But if Christopher's analysis is correct, the "macro" bit is different to the "plugin" bit. I think allowing the ASTs to be _modified_ by plugins is the path to madness, but a read-only ABI is OK (it's hard to see how compile-time reflection is possible without creating some kind of API).

On 12/10/2009 10:47, Don wrote:
>
> Ah, OK. My cursory glance at Nemerle just screamed "hack". But first
> impressions can be misleading.
> No doubt as a C-family language, they have some useful ideas.
> But if Christopher's analysis is correct, the "macro" bit is different
> to the "plugin" bit. I think allowing the ASTs to be _modified_ by
> plugins is the path to madness, but a read-only ABI is OK (it's hard to
> see how compile-time reflection is possible without creating some kind
> of API).
>

modifying the AST is dangerous but how would you do things like making a class implement an interface without modifying the list of interfaces the class implements ?

[serialize]
class Foo {...}

the Nemerle macro above transform this into:

class Foo : Serializable { ... }

what would be your design for this?

Yigal Chripun wrote:
> On 12/10/2009 10:47, Don wrote:
>>
>> Ah, OK. My cursory glance at Nemerle just screamed "hack". But first
>> impressions can be misleading.
>> No doubt as a C-family language, they have some useful ideas.
>> But if Christopher's analysis is correct, the "macro" bit is different
>> to the "plugin" bit. I think allowing the ASTs to be _modified_ by
>> plugins is the path to madness, but a read-only ABI is OK (it's hard to
>> see how compile-time reflection is possible without creating some kind
>> of API).
>>
> 
> modifying the AST is dangerous but how would you do things like making a class implement an interface without modifying the list of interfaces the class implements ?
> 
> [serialize]
> class Foo {...}
> 
> the Nemerle macro above transform this into:
> 
> class Foo : Serializable { ... }
> 
> what would be your design for this?

I don't think it should be done that way. Either it should be intrusive (eg, require you to derive from Serializable), or else entirely external (and operate via reflection). Eg, possibly by declaring Serializable!(Foo) after the definition of Foo.
It's an excellent question though, we might need some language changes to get a good solution. But I think it's very important to enforce that the only way to modify the AST is indirect, through code.

One of the strengths I see of the string mixins, despite their syntactic ugliness, is that metaprogramming transformations are only ever syntax sugar: there is _always_ a source-code equivalent; moreover, it's trivially available, simply by expanding all of the mixins (and Descent will show it to you). It's not clear to me that that remains true if you can manipulate the AST directly. Of course, if you allow direct access to the AST, you have unlimited power. Capturing as much of the power as possible, without the danger, is the challenge.

Don Wrote:

> Yigal Chripun wrote:
> > On 12/10/2009 10:47, Don wrote:
> >>
> >> Ah, OK. My cursory glance at Nemerle just screamed "hack". But first
> >> impressions can be misleading.
> >> No doubt as a C-family language, they have some useful ideas.
> >> But if Christopher's analysis is correct, the "macro" bit is different
> >> to the "plugin" bit. I think allowing the ASTs to be _modified_ by
> >> plugins is the path to madness, but a read-only ABI is OK (it's hard to
> >> see how compile-time reflection is possible without creating some kind
> >> of API).
> >>
> > 
> > modifying the AST is dangerous but how would you do things like making a class implement an interface without modifying the list of interfaces the class implements ?
> > 
> > [serialize]
> > class Foo {...}
> > 
> > the Nemerle macro above transform this into:
> > 
> > class Foo : Serializable { ... }
> > 
> > what would be your design for this?
> 
> I don't think it should be done that way. Either it should be intrusive
> (eg, require you to derive from Serializable), or else entirely external
> (and operate via reflection). Eg, possibly by declaring
> Serializable!(Foo) after the definition of Foo.
> It's an excellent question though, we might need some language changes
> to get a good solution. But I think it's very important to enforce that
> the only way to modify the AST is indirect, through code.
> 
> One of the strengths I see of the string mixins, despite their syntactic ugliness, is that metaprogramming transformations are only ever syntax sugar: there is _always_ a source-code equivalent; moreover, it's trivially available, simply by expanding all of the mixins (and Descent will show it to you). It's not clear to me that that remains true if you can manipulate the AST directly. Of course, if you allow direct access to the AST, you have unlimited power. Capturing as much of the power as possible, without the danger, is the challenge.
> 
> 

I agree that there should be a source code equivalent for macros - otherwise it probably would be hard to debug. But I think we agree that at least a read only API would be very useful, much better than the current bag-o-hacks and without the risks.

I know the Nemerle devs are working on improving their Macro system implementation. I'll ask in their NG about this issue.