Jarrett Billingsley wrote:
> On Wed, Oct 7, 2009 at 11:21 AM, Don <nospam@nospam.com> wrote:
>> Steven Schveighoffer wrote:
>>> On Wed, 07 Oct 2009 09:17:59 -0400, Jarrett Billingsley
>>> <jarrett.billingsley@gmail.com> wrote:
>>>
>>>> It's also insanely kludgy and ugly. Bleh.
>> Ugly, yes. Kludgy, I don't think so. It's only a syntax issue. The basic
>> concept of passing meta-code to the compiler in the form of raw text is
>> simple:
>>
>> mixin() if you want to insert something into the parse step.
>>  is(typeof()) if you want to catch it again after the syntax pass.
>>  stringof if you want to catch it again after the semantic pass.
>>
>> And that's all. The syntax is ugly, but the semantics are beautifully
>> elegant.
> 
> It'd be nice if they actually worked. is(typeof()) fails for *any*
> error, and it eats those errors too, so if your code fails to compile
> for some reason other than the one you're testing for, welp, good luck
> figuring that out. And don't even get me started on .stringof.
> 
> Also, see my post on the "get template and its instantiation
> parameters" thread for my detailed opinion on them.
> 
>> By contrast, something like Nemerle macros are a kludge. The idea of
>> providing a 'hook' into the compiler is a horrible hack. It exposes all
>> kinds of compiler internals. Yes, it has nicer syntax.
> 
> I.. don't even know how to begin to respond to that.

Have you read the Nemerle extended macro tutorial? The compiler's internal structures are completely exposed. That's a hack.

On Thu, Oct 8, 2009 at 1:06 AM, Don <nospam@nospam.com> wrote:
> Jarrett Billingsley wrote:
>>
>> On Wed, Oct 7, 2009 at 11:21 AM, Don <nospam@nospam.com> wrote:
>>>
>>> Steven Schveighoffer wrote:
>>>>
>>>> On Wed, 07 Oct 2009 09:17:59 -0400, Jarrett Billingsley <jarrett.billingsley@gmail.com> wrote:
>>>>
>>>>> It's also insanely kludgy and ugly. Bleh.
>>>
>>> Ugly, yes. Kludgy, I don't think so. It's only a syntax issue. The basic concept of passing meta-code to the compiler in the form of raw text is simple:
>>>
>>> mixin() if you want to insert something into the parse step.
>>>  is(typeof()) if you want to catch it again after the syntax pass.
>>>  stringof if you want to catch it again after the semantic pass.
>>>
>>> And that's all. The syntax is ugly, but the semantics are beautifully elegant.
>>
>> It'd be nice if they actually worked. is(typeof()) fails for *any* error, and it eats those errors too, so if your code fails to compile for some reason other than the one you're testing for, welp, good luck figuring that out. And don't even get me started on .stringof.
>>
>> Also, see my post on the "get template and its instantiation parameters" thread for my detailed opinion on them.
>>
>>> By contrast, something like Nemerle macros are a kludge. The idea of providing a 'hook' into the compiler is a horrible hack. It exposes all kinds of compiler internals. Yes, it has nicer syntax.
>>
>> I.. don't even know how to begin to respond to that.
>
> Have you read the Nemerle extended macro tutorial? The compiler's internal structures are completely exposed. That's a hack.

It seems macros are implemented as compiler extensions.  You compile your macros into DLLs first, that then get loaded into the compiler as plugins.  On the plus side, doing things that way you really do have access to any API you need at compile-time, using the same syntax as run-time.  All of .NET can be used at compile-time in your macros.  No more "can't CTFE that" gotchas.

But it does raise security concerns.  I wonder if they have some way to prevent macros from running malicious code.  I guess you better run your web-based compiler service in a tightly partitioned VM.

Overall it seems pretty nifty to me, really.  Giving macros access to an actual compiler API seems less hackish than throwing in a smattering of diverse functionality under the heading of __traits. And less prone to gotchas than trying to create a separate compile-time D interpreter that runs inside the D compiler.

What do you see as the down sides?  Just that some rogue macro might mess up the AST?

--bb

On Thu, Oct 8, 2009 at 4:00 AM, Don <nospam@nospam.com> wrote:

>> So it looks to me like the mechanics of it are basically identical. Just Nemerle's syntax is nicer.
>
> Only with trivial examples. With more complicated examples they look less identical. I'm basing my views on pages like this:
>
> http://nemerle.org/Macros_-_extended_course._Part_2
>
> Unless I'm totally misunderstanding this, it looks to me as though Nemerle
> macros are implemented as compiler plugins.
> All the advanced facilities are obtained by exposing the compiler's API!

Well they're not.. "plugins" per se as much as "compiled modules." Okay yes that's basically a plugin :P But it's not that different from D, where you use compile-time executed functions to do the same sorts of things. It's just that you precompile those functions instead of having the compiler "compile" them on every compilation.

But really, I don't see how this is significantly different from hooking into the D compiler's internals with __traits, .stringof, .mangleof and the like. So it uses an object-oriented API to access those things instead of ad-hoc hacks. And? I don't know how you can trash Nemerle's approach while leaving D's unmentioned.

Jarrett Billingsley wrote:
> On Thu, Oct 8, 2009 at 4:00 AM, Don <nospam@nospam.com> wrote:
> 
>>> So it looks to me like the mechanics of it are basically identical.
>>> Just Nemerle's syntax is nicer.
>> Only with trivial examples. With more complicated examples they look less
>> identical. I'm basing my views on pages like this:
>>
>> http://nemerle.org/Macros_-_extended_course._Part_2
>>
>> Unless I'm totally misunderstanding this, it looks to me as though Nemerle
>> macros are implemented as compiler plugins.
>> All the advanced facilities are obtained by exposing the compiler's API!
> 
> Well they're not.. "plugins" per se as much as "compiled modules."
> Okay yes that's basically a plugin :P But it's not that different from
> D, where you use compile-time executed functions to do the same sorts
> of things. It's just that you precompile those functions instead of
> having the compiler "compile" them on every compilation.

No. CTFE is simply taking constant-folding to its logical conclusion.

> But really, I don't see how this is significantly different from
> hooking into the D compiler's internals with __traits, .stringof,
> .mangleof and the like. So it uses an object-oriented API to access
> those things instead of ad-hoc hacks. And? 

The thing I think is elegant about D's approach, ugly as the syntax currently is, is the complete separation of the lex - parse - semantic - codegen phases. And I think CTFE is fantastic (and I plan to fix it so it works properly). Think about how easy it is to explain.

I'm not a fan of is(typeof()) .stringof and __traits in their current form. They are hackish indeed, and weren't originally intended for macro development. (Actually .stringof isn't hackish, just buggy and unspecified). BUT they demonstrate the benefit of the seperate compilation phases. The fundamentals are strong.

> I don't know how you can trash Nemerle's approach while leaving D's unmentioned.

What do you mean, 'unmentioned'? Hey, you started this by trashing D's approach!

On Thu, Oct 8, 2009 at 11:25 AM, Don <nospam@nospam.com> wrote:
> Jarrett Billingsley wrote:
>>
>> On Thu, Oct 8, 2009 at 4:00 AM, Don <nospam@nospam.com> wrote:
>>
>>>> So it looks to me like the mechanics of it are basically identical. Just Nemerle's syntax is nicer.
>>>
>>> Only with trivial examples. With more complicated examples they look less identical. I'm basing my views on pages like this:
>>>
>>> http://nemerle.org/Macros_-_extended_course._Part_2
>>>
>>> Unless I'm totally misunderstanding this, it looks to me as though
>>> Nemerle
>>> macros are implemented as compiler plugins.
>>> All the advanced facilities are obtained by exposing the compiler's API!
>>
>> Well they're not.. "plugins" per se as much as "compiled modules." Okay yes that's basically a plugin :P But it's not that different from D, where you use compile-time executed functions to do the same sorts of things. It's just that you precompile those functions instead of having the compiler "compile" them on every compilation.
>
> No. CTFE is simply taking constant-folding to its logical conclusion.

All the Nemerle *implementation* is doing differently here is having you precompile the functions that are to be executed at compile-time. That's it. The resultant semantics are utterly the same. You are running code at compile-time, it doesn't matter if that code is running on the hardware or in a VM in the compiler.

>> But really, I don't see how this is significantly different from hooking into the D compiler's internals with __traits, .stringof, .mangleof and the like. So it uses an object-oriented API to access those things instead of ad-hoc hacks. And?
>
> The thing I think is elegant about D's approach, ugly as the syntax currently is, is the complete separation of the lex - parse - semantic - codegen phases. And I think CTFE is fantastic (and I plan to fix it so it works properly). Think about how easy it is to explain.

And Nemerle's macros are *hard* to explain? They're a function that executes at compile time. Oh, wait! That's exactly the same as CTFE.

>> I don't know how you can trash Nemerle's approach while leaving D's unmentioned.
>
> What do you mean, 'unmentioned'? Hey, you started this by trashing D's approach!

I'm not talking about me. I'm talking about you. I don't know how you can trash a language with macros that were designed *from the bottom up* to be used as such and which are treated as first-class citizens, while not admitting the hilariously ad-hoc nature of a language where macros fall out as a consequence of a number of other, ill-defined poorly-implemented unorthogonal features.

Sigh, I'm done.

Jarrett Billingsley wrote:
> On Thu, Oct 8, 2009 at 11:25 AM, Don <nospam@nospam.com> wrote:
>> Jarrett Billingsley wrote:
>>> On Thu, Oct 8, 2009 at 4:00 AM, Don <nospam@nospam.com> wrote:
>>>
>>>>> So it looks to me like the mechanics of it are basically identical.
>>>>> Just Nemerle's syntax is nicer.
>>>> Only with trivial examples. With more complicated examples they look less
>>>> identical. I'm basing my views on pages like this:
>>>>
>>>> http://nemerle.org/Macros_-_extended_course._Part_2
>>>>
>>>> Unless I'm totally misunderstanding this, it looks to me as though
>>>> Nemerle
>>>> macros are implemented as compiler plugins.
>>>> All the advanced facilities are obtained by exposing the compiler's API!
>>> Well they're not.. "plugins" per se as much as "compiled modules."
>>> Okay yes that's basically a plugin :P But it's not that different from
>>> D, where you use compile-time executed functions to do the same sorts
>>> of things. It's just that you precompile those functions instead of
>>> having the compiler "compile" them on every compilation.
>> No. CTFE is simply taking constant-folding to its logical conclusion.
> 
> All the Nemerle *implementation* is doing differently here is having
> you precompile the functions that are to be executed at compile-time.
> That's it. The resultant semantics are utterly the same. You are
> running code at compile-time, it doesn't matter if that code is
> running on the hardware or in a VM in the compiler.
> 
>>> But really, I don't see how this is significantly different from
>>> hooking into the D compiler's internals with __traits, .stringof,
>>> .mangleof and the like. So it uses an object-oriented API to access
>>> those things instead of ad-hoc hacks. And?
>> The thing I think is elegant about D's approach, ugly as the syntax
>> currently is, is the complete separation of the lex - parse - semantic -
>> codegen phases. And I think CTFE is fantastic (and I plan to fix it so it
>> works properly). Think about how easy it is to explain.
> 
> And Nemerle's macros are *hard* to explain? They're a function that
> executes at compile time. Oh, wait! That's exactly the same as CTFE.
> 
>>> I don't know how you can trash Nemerle's approach while leaving D's
>>> unmentioned.
>> What do you mean, 'unmentioned'? Hey, you started this by trashing D's
>> approach!
> 
> I'm not talking about me. I'm talking about you. I don't know how you
> can trash a language with macros that were designed *from the bottom
> up* to be used as such and which are treated as first-class citizens,
> while not admitting the hilariously ad-hoc nature of a language where
> macros fall out as a consequence of a number of other, ill-defined
> poorly-implemented unorthogonal features.
> 
> Sigh, I'm done.

I agree with Jarrett here. And also seeing how some things are implemented in D using CTFE and .stringof and it's parsing is very complex to understand. I mean, I read the code and it's very hard for me to understand what's going on. Specially because it's mostly all of the time parsing strings and extracting information which I don't know in what format it comes in the first place, it's just guess and work around it.

Bill Baxter wrote:
> It seems macros are implemented as compiler extensions.  You compile
> your macros into DLLs first, that then get loaded into the compiler as
> plugins.  On the plus side, doing things that way you really do have
> access to any API you need at compile-time, using the same syntax as
> run-time.  All of .NET can be used at compile-time in your macros.  No
> more "can't CTFE that" gotchas.
> 
> But it does raise security concerns.  I wonder if they have some way
> to prevent macros from running malicious code.  I guess you better run
> your web-based compiler service in a tightly partitioned VM.

C# has security levels. If you run the Nemerle compiler in low trust mode, fewer bad things can happen.

> Overall it seems pretty nifty to me, really.  Giving macros access to
> an actual compiler API seems less hackish than throwing in a
> smattering of diverse functionality under the heading of __traits.
> And less prone to gotchas than trying to create a separate
> compile-time D interpreter that runs inside the D compiler.
> 
> What do you see as the down sides?  Just that some rogue macro might
> mess up the AST?

It makes macros highly compiler-specific, or requires the compiler's AST to be part of the language.

Nemerle took the nuclear option, and its macros are all-powerful. That's a reasonable way of doing things. I'd be happy with a more restricted system that's easier to standardize, especially if it got rid of all the hacky string manipulation in current D metaprogramming. (Seriously, even __traits returns string arrays for a lot of stuff. It's ridiculous.)

On 09/10/2009 00:38, Christopher Wright wrote:
> Bill Baxter wrote:
>> It seems macros are implemented as compiler extensions. You compile
>> your macros into DLLs first, that then get loaded into the compiler as
>> plugins. On the plus side, doing things that way you really do have
>> access to any API you need at compile-time, using the same syntax as
>> run-time. All of .NET can be used at compile-time in your macros. No
>> more "can't CTFE that" gotchas.
>>
>> But it does raise security concerns. I wonder if they have some way
>> to prevent macros from running malicious code. I guess you better run
>> your web-based compiler service in a tightly partitioned VM.
>
> C# has security levels. If you run the Nemerle compiler in low trust
> mode, fewer bad things can happen.
>
>> Overall it seems pretty nifty to me, really. Giving macros access to
>> an actual compiler API seems less hackish than throwing in a
>> smattering of diverse functionality under the heading of __traits.
>> And less prone to gotchas than trying to create a separate
>> compile-time D interpreter that runs inside the D compiler.
>>
>> What do you see as the down sides? Just that some rogue macro might
>> mess up the AST?
>
> It makes macros highly compiler-specific, or requires the compiler's AST
> to be part of the language.
>
> Nemerle took the nuclear option, and its macros are all-powerful. That's
> a reasonable way of doing things. I'd be happy with a more restricted
> system that's easier to standardize, especially if it got rid of all the
> hacky string manipulation in current D metaprogramming. (Seriously, even
> __traits returns string arrays for a lot of stuff. It's ridiculous.)

It doesn't have to be compiler specific. all is needed is a standardized API to the compiler.
What's so hackish about that?
many large modular systems do exactly that: eclipse, firefox, even the OS itself. Unix provides syscalls which *are* an API to the OS.

a properly designed API doesn't have to expose internal implementation details.

btw, in Nemerle they have syntax to compose/decompose AST specifically so they don't need to expose the internal structure of the AST.

On 08/10/2009 17:25, Don wrote:
> Jarrett Billingsley wrote:
>> On Thu, Oct 8, 2009 at 4:00 AM, Don <nospam@nospam.com> wrote:
>>
>>>> So it looks to me like the mechanics of it are basically identical.
>>>> Just Nemerle's syntax is nicer.
>>> Only with trivial examples. With more complicated examples they look
>>> less
>>> identical. I'm basing my views on pages like this:
>>>
>>> http://nemerle.org/Macros_-_extended_course._Part_2
>>>
>>> Unless I'm totally misunderstanding this, it looks to me as though
>>> Nemerle
>>> macros are implemented as compiler plugins.
>>> All the advanced facilities are obtained by exposing the compiler's API!
>>
>> Well they're not.. "plugins" per se as much as "compiled modules."
>> Okay yes that's basically a plugin :P But it's not that different from
>> D, where you use compile-time executed functions to do the same sorts
>> of things. It's just that you precompile those functions instead of
>> having the compiler "compile" them on every compilation.
>
> No. CTFE is simply taking constant-folding to its logical conclusion.
>
>> But really, I don't see how this is significantly different from
>> hooking into the D compiler's internals with __traits, .stringof,
>> .mangleof and the like. So it uses an object-oriented API to access
>> those things instead of ad-hoc hacks. And?
>
> The thing I think is elegant about D's approach, ugly as the syntax
> currently is, is the complete separation of the lex - parse - semantic -
> codegen phases. And I think CTFE is fantastic (and I plan to fix it so
> it works properly). Think about how easy it is to explain.
>
What about Nemerle's macro system design (I'm not talking about their specific implementation of it) conflicts with D's complete separation of lex -> parse -> semantic phases?
The phases can still be completely separate *but* extensible by the macro system.

BTW, regarding this design aspect of DMD and D
The dragon book (second edition) says on page 966:
<quote>
Object-Oriented Versus Phase-Oriented
with an object oriented approach, all the code for a construct is collected in the class for the construct. Alternatively, with a phase-oriented approach the code is grouped by phase so a type checking procedure would have a case for each construct and a code generation procedure would have a case for each construct, and so on.
the tradeoff is that an object-oriented approach makes it easier to change or add a construct, such as "for" statements, and a phase-oriented approach makes it easier to change or add a phase, such as type checking.
</quote>


> I'm not a fan of is(typeof()) .stringof and __traits in their current
> form. They are hackish indeed, and weren't originally intended for macro
> development. (Actually .stringof isn't hackish, just buggy and
> unspecified). BUT they demonstrate the benefit of the seperate
> compilation phases. The fundamentals are strong.
>
>  > I don't know how you can trash Nemerle's approach while leaving D's
> unmentioned.
>
> What do you mean, 'unmentioned'? Hey, you started this by trashing D's
> approach!

Yigal Chripun wrote:
> On 09/10/2009 00:38, Christopher Wright wrote:
>> It makes macros highly compiler-specific, or requires the compiler's AST
>> to be part of the language.
>>
>> Nemerle took the nuclear option, and its macros are all-powerful. That's
>> a reasonable way of doing things. I'd be happy with a more restricted
>> system that's easier to standardize, especially if it got rid of all the
>> hacky string manipulation in current D metaprogramming. (Seriously, even
>> __traits returns string arrays for a lot of stuff. It's ridiculous.)
> 
> It doesn't have to be compiler specific. all is needed is a standardized API to the compiler.

Right. It adds something huge that's normally compiler-specific to the language. This makes me uncomfortable. It greatly increases the difficulty of implementation.

> What's so hackish about that?

Reread. Current D metaprogramming is hackish. Nemerle's isn't.

> many large modular systems do exactly that: eclipse, firefox, even the OS itself. Unix provides syscalls which *are* an API to the OS.
> 
> a properly designed API doesn't have to expose internal implementation details.
> 
> btw, in Nemerle they have syntax to compose/decompose AST specifically so they don't need to expose the internal structure of the AST.

So they have a separate object model for the syntax tree that macros can affect. This is what I would recommend for D.