On Sun, 11 Jul 2010 07:01:49 -0400, Don <nospam@nospam.com> wrote:

> retard wrote:
>> Thu, 08 Jul 2010 21:43:57 -0400, Robert Jacques wrote:
>>
>>> Check out Walter's slides and/or talk from the D conference.
>>> (http://www.digitalmars.com/webnews/newsgroups.php?
>> art_group=digitalmars.D.announce&article_id=12555)
>>> AST does stand for abstract syntax tree and they are much more like Lisp
>>> macros as opposed to the C preprocessor.
>>  Too bad nothing happened. He promised AST macros, but it will probably still take several years before we have a preliminary implementation and maybe 10-20 years before a written spec & almsot bug-free implementation.
>
> Part of what happened was that at the conference, I showed that the functionality of string mixins was a superset of what was planned for AST macros. Since that time, there has not been any kind of proposal, or even a concept. So it's a bit meaningless to talk about "AST macros" right now as if they were a planned-but-not-yet-implemented feature.

The problem I see with mixins is they are ugly.  The difference between mixins and most other parts of the language is the syntax is still super-basic.  It would be the equivalent of having to call functions with callfunction(funcname, params).  What we need is a way to make mixins look like normal expressions.

I proposed at one time to use macro to define easily callable mixins.  It was something along the lines of:

macro log(level, x) mixin("if(" ~ level ~ " == log.level){log.output(" ~ x ~ ")}");

which makes it possible to have lazy logging without the lazy keyword and without the ugliness of mixins at the call site.

If people agree that AST macros are superseded by mixins, then why not take over the macro keyword (which if restricted to AST macros is effectively a dead keyword), and make programmer's lives and generic programming easier?

-Steve

Steven Schveighoffer:
> If people agree that AST macros are superseded by mixins,

String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros.

Bye,
bearophile

On Mon, 12 Jul 2010 08:12:31 -0400, bearophile <bearophileHUGS@lycos.com> wrote:

> Steven Schveighoffer:
>> If people agree that AST macros are superseded by mixins,
>
> String mixins are a hack, just a bit better than C preprocessor (because they are scoped), they are not a replacement of clean macros.

Interesting statement, can you back it up? :)  What can you do with a macro that you can't do with a mixin?

-Steve

On 07/12/2010 07:30 AM, Steven Schveighoffer wrote:
> On Mon, 12 Jul 2010 08:12:31 -0400, bearophile
> <bearophileHUGS@lycos.com> wrote:
>
>> Steven Schveighoffer:
>>> If people agree that AST macros are superseded by mixins,
>>
>> String mixins are a hack, just a bit better than C preprocessor
>> (because they are scoped), they are not a replacement of clean macros.
>
> Interesting statement, can you back it up? :) What can you do with a
> macro that you can't do with a mixin?
>
> -Steve

I think the big thing about macros is you don't have to worry about lexing and parsing.

if <A> is of the form (Assignable, Assignable, ... ),
and <B> of the form (AssignExp, AssignExp, ... )

how do you readily rewrite

mixin("<A> = <B>;");

to

(<A>[0] = <B>[0], <A>[1] = <B>[1], ... )

without a metric shit ton of library support?

With the most bare bones of AST macros (maybe a ctfe function that gets passed a AST* and returns a AST* or something equivalent), it's pretty trivial. And you don't need to wrap it in a string literal.

On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer <ellery-newcomer@utulsa.edu> wrote:

> On 07/12/2010 07:30 AM, Steven Schveighoffer wrote:
>> On Mon, 12 Jul 2010 08:12:31 -0400, bearophile
>> <bearophileHUGS@lycos.com> wrote:
>>
>>> Steven Schveighoffer:
>>>> If people agree that AST macros are superseded by mixins,
>>>
>>> String mixins are a hack, just a bit better than C preprocessor
>>> (because they are scoped), they are not a replacement of clean macros.
>>
>> Interesting statement, can you back it up? :) What can you do with a
>> macro that you can't do with a mixin?
>>
>> -Steve
>
> I think the big thing about macros is you don't have to worry about lexing and parsing.
>
> if <A> is of the form (Assignable, Assignable, ... ),
> and <B> of the form (AssignExp, AssignExp, ... )
>
> how do you readily rewrite
>
> mixin("<A> = <B>;");
>
> to
>
> (<A>[0] = <B>[0], <A>[1] = <B>[1], ... )
>
> without a metric shit ton of library support?
>
> With the most bare bones of AST macros (maybe a ctfe function that gets passed a AST* and returns a AST* or something equivalent), it's pretty trivial. And you don't need to wrap it in a string literal.

But is that common?  Is there a common need to parse mixin strings and replace with other strings?  It seems like the means that AST macros operate, but is that the ultimate goal?  Even if it is, can't a string mixin system do exactly that?

You say a ton of library support is bad, but why is that?  If the end result is the same, what does it matter if you implement it via a library vs. the compiler?  Also, to me, it's trivial to understand string manipulation where the end result is the language I already know, vs. having to learn how the compiler represents syntax.

I haven't really used AST macros.  Does anyone have a good practical example of how it's used in other languages to compare with string mixins?  Your example seems trivially solved via simple library functions such as split or using regex.

-Steve

== Quote from Steven Schveighoffer (schveiguy@yahoo.com)'s article
> On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer <ellery-newcomer@utulsa.edu> wrote:
[...]
> > I think the big thing about macros is you don't have to worry about lexing and parsing.

> > if <A> is of the form (Assignable, Assignable, ... ),
> > and <B> of the form (AssignExp, AssignExp, ... )

> > how do you readily rewrite

> > mixin("<A> = <B>;");

> > to

> > (<A>[0] = <B>[0], <A>[1] = <B>[1], ... )

> > without a metric shit ton of library support?

> > With the most bare bones of AST macros (maybe a ctfe
> > function that gets passed a AST* and returns a AST*
> > or something equivalent), it's pretty trivial. And
> > you don't need to wrap it in a string literal.

> But is that common? Is there a common need to parse mixin strings and replace with other strings?

In my experience with Lisp macros (which are fundamentally AST macros) doing that sort of thing is fairly common when it's easy.

> Even if it is, can't a string
> mixin system do exactly that?

I think it can, actually. Having library functions pick up the slack may not be a terrible idea at all. Parsing a string into an AST is just a function, transforming that AST into another AST is a function, and transforming the resulting AST back into a string that gets mixed in is a third function.

I see two problems with this plan, both which may be addressable without too much trouble.

One problem is that string mixins look terrible, what with them being string literals. Having some sort of syntactic support for using macros would be a major improvement. I liked your "macro" keyword suggestion from elsethread.

It requires a lot of straight-up hackery. Consider retard's suggested way for creating a unique name. Being able to do this is crucial to writing macros that simultaneously do non-trivial things and are robust, and the inability to create unique names is one reason C preprocessor macros are so lame. It would be far nicer if you could use some sort of __traits form to get a unique name (and then that could be wrapped in a string-mixin macro)!

> Also, to me, it's trivial to understand string manipulation where the end result is the language I already know, vs. having to learn how the compiler represents syntax.

String manipulation scales very poorly, it's error prone, and it's tedious.

Cheers,
Pillsy

On 07/12/2010 09:57 AM, pillsy wrote:
> == Quote from Steven Schveighoffer (schveiguy@yahoo.com)'s article
>> On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer
>> <ellery-newcomer@utulsa.edu>  wrote:
> [...]
>>> I think the big thing about macros is you don't have to
>>> worry about lexing and parsing.
>
>>> if<A>  is of the form (Assignable, Assignable, ... ),
>>> and<B>  of the form (AssignExp, AssignExp, ... )
>
>>> how do you readily rewrite
>
>>> mixin("<A>  =<B>;");
>
>>> to
>
>>> (<A>[0] =<B>[0],<A>[1] =<B>[1], ... )
>
>>> without a metric shit ton of library support?
>
>>> With the most bare bones of AST macros (maybe a ctfe
>>> function that gets passed a AST* and returns a AST*
>>> or something equivalent), it's pretty trivial. And
>>> you don't need to wrap it in a string literal.
>
>> But is that common? Is there a common need to parse mixin
>> strings and replace with other strings?
>
> In my experience with Lisp macros (which are fundamentally AST
> macros) doing that sort of thing is fairly common when it's easy.
>
>> Even if it is, can't a string
>> mixin system do exactly that?
>
> I think it can, actually. Having library functions pick up the
> slack may not be a terrible idea at all. Parsing a string into an
> AST is just a function, transforming that AST into another AST is a
> function, and transforming the resulting AST back into a string
> that gets mixed in is a third function.
>
> I see two problems with this plan, both which may be addressable
> without too much trouble.
>
> One problem is that string mixins look terrible, what with them
> being string literals. Having some sort of syntactic support for
> using macros would be a major improvement. I liked your "macro"
> keyword suggestion from elsethread.
>
> It requires a lot of straight-up hackery. Consider retard's
> suggested way for creating a unique name. Being able to do this is
> crucial to writing macros that simultaneously do non-trivial things
> and are robust, and the inability to create unique names is one
> reason C preprocessor macros are so lame. It would be far nicer if
> you could use some sort of __traits form to get a unique name (and
> then that could be wrapped in a string-mixin macro)!
>
>> Also, to me, it's trivial to understand string manipulation
>> where the end result is the language I already know, vs.
>> having to learn how the compiler represents syntax.
>
> String manipulation scales very poorly, it's error prone, and it's
> tedious.
>
> Cheers,
> Pillsy

Though I sympathise with all of the above, I should add that I have looked at languages that feature AST macros (Lisp, Scheme, Dylan) and such macros are difficult to create and read as well. We're not looking at day and night improvement there.

Andrei

Mon, 12 Jul 2010 14:57:35 +0000, pillsy wrote:

> == Quote from Steven Schveighoffer (schveiguy@yahoo.com)'s article
>> On Mon, 12 Jul 2010 09:33:34 -0400, Ellery Newcomer <ellery-newcomer@utulsa.edu> wrote:
> [...]
>> > I think the big thing about macros is you don't have to worry about lexing and parsing.
> 
>> > if <A> is of the form (Assignable, Assignable, ... ), and <B> of the
>> > form (AssignExp, AssignExp, ... )
> 
>> > how do you readily rewrite
> 
>> > mixin("<A> = <B>;");
> 
>> > to
> 
>> > (<A>[0] = <B>[0], <A>[1] = <B>[1], ... )
> 
>> > without a metric shit ton of library support?
> 
>> > With the most bare bones of AST macros (maybe a ctfe function that gets passed a AST* and returns a AST* or something equivalent), it's pretty trivial. And you don't need to wrap it in a string literal.
> 
>> But is that common? Is there a common need to parse mixin strings and replace with other strings?
> 
> In my experience with Lisp macros (which are fundamentally AST macros) doing that sort of thing is fairly common when it's easy.
> 
>> Even if it is, can't a string
>> mixin system do exactly that?
> 
> I think it can, actually. Having library functions pick up the slack may not be a terrible idea at all. Parsing a string into an AST is just a function, transforming that AST into another AST is a function, and transforming the resulting AST back into a string that gets mixed in is a third function.

Yes, this would be possible if the dmd compiler didn't crash, leak memory etc. The quality is so shitty that I'd say that implementing a compile time AST parser for D 2 is almost impossible. And what's worse is that the language is changing constantly so the AST parser meta-library would need to be updated once in a while. This is great news for all the fans of NIH syndrome.

> 
> I see two problems with this plan, both which may be addressable without too much trouble.
> 
> One problem is that string mixins look terrible, what with them being string literals. Having some sort of syntactic support for using macros would be a major improvement. I liked your "macro" keyword suggestion from elsethread.
> 
> It requires a lot of straight-up hackery. Consider retard's suggested way for creating a unique name. Being able to do this is crucial to writing macros that simultaneously do non-trivial things and are robust, and the inability to create unique names is one reason C preprocessor macros are so lame. It would be far nicer if you could use some sort of __traits form to get a unique name (and then that could be wrapped in a string-mixin macro)!
> 
>> Also, to me, it's trivial to understand string manipulation where the end result is the language I already know, vs. having to learn how the compiler represents syntax.
> 
> String manipulation scales very poorly, it's error prone, and it's tedious.

Like Andrei said, real AST macros can also be hairy, but they _are_ an improvement over string based macros in many cases. Lisp users have 50+ years of experience with meta-programming. You're mostly talking with amateurs here. Of course they don't see the value of true macros. One fundamental prerequisite when discussing macros is that you've actually used them in at least ONE other language.

On 07/13/2010 10:43 AM, retard wrote:
> Yes, this would be possible if the dmd compiler didn't crash, leak memory
> etc. The quality is so shitty that I'd say that implementing a compile
> time AST parser for D 2 is almost impossible.

Well it should be said that the quality of the compiler has dramatically improved over the past months. I think it would be worth writing a tokenizer for starters.

> And what's worse is that
> the language is changing constantly so the AST parser meta-library would
> need to be updated once in a while. This is great news for all the fans
> of NIH syndrome.

The rate of language changes has slowed down very visibly since TDPL has come out.

>> String manipulation scales very poorly, it's error prone, and it's
>> tedious.
>
> Like Andrei said, real AST macros can also be hairy, but they _are_ an
> improvement over string based macros in many cases. Lisp users have 50+
> years of experience with meta-programming. You're mostly talking with
> amateurs here. Of course they don't see the value of true macros. One
> fundamental prerequisite when discussing macros is that you've actually
> used them in at least ONE other language.

Agreed. Reminds me of the Go programmers who don't need generics :o). The Sapir-Whorf theory may or may not apply to natural language, but has a visible presence in programming languages.


Andrei

Sorry if this is "re-opening" an old thread, but did anything come from this and DIP50? It seems like a really interesting concept and this thread was one of the first results for a Google search.

Thanks.