On Friday, April 06, 2018 20:33:10 Chris Katko via Digitalmars-d wrote:
> Sorry if this is "re-opening" an old thread, but did anything come from this and DIP50? It seems like a really interesting concept and this thread was one of the first results for a Google search.

D does not have any kind of AST macros and likely never will. Walter is completely against the idea - though I'd have to go digging through newsgroup's history to find posts where he talked about it to give the exact reasons. It's been a while since they were discussed last, and I don't recall them at the moment. Either way, DIP 50 was part of the old DIP process where DIPs were not always really reviewed like they should have been, so it was never officially accepted or rejected. I don't know if Walter or Andrei have even ever looked at it. And for any DIP to be accepted now, it must go through the new DIP process: https://github.com/dlang/DIPs

So, regardless of what Walter thinks of AST macros, for them to make it into the language, someone will have to submit a DIP through the new DIP process, and no one has done that yet. However, based on what Walter has said in the past, I would fully expect such a DIP to be rejected.

- Jonathan M Davis

On Friday, 6 April 2018 at 21:23:00 UTC, Jonathan M Davis wrote:
> D does not have any kind of AST macros and likely never will. Walter is completely against the idea - though I'd have to go digging through newsgroup's history to find posts where he talked about it to give the exact reasons. It's been a while since they were discussed last, and I don't recall them at the moment.

I think Walter's reason was that such macros would hide too many idiosyncrasies in how they were programmed, such that a lot of code which seems simple on the surface will actually obfuscate complicated and arbitrary macro-programming patterns. Thus, code that uses them will become much less maintainable, because it is liable to do so many different and hidden things. Also, the tasks for which AST-macros would typically be used are already largely accommodated by templates and other features. Thus, the real need for them isn't that high.

On Friday, 6 April 2018 at 21:45:45 UTC, Zach Tollen wrote:
>
> I think Walter's reason was that such macros would hide too many idiosyncrasies in how they were programmed, such that a lot of code which seems simple on the surface will actually obfuscate complicated and arbitrary macro-programming patterns. Thus, code that uses them will become much less maintainable, because it is liable to do so many different and hidden things. Also, the tasks for which AST-macros would typically be used are already largely accommodated by templates and other features. Thus, the real need for them isn't that high.

I think it's time to revisit this.

The reason being that templates are only well suited to very specific types of meta-programs which have a low degree of parameterization and a low degree of abstraction.

Using templates to introspect and manipulate types is like using a hammmer's flat back to remove a nail.
It _can_ be done but with an absurd amount of work.
You just have to remove all of the wall around the nail by pounding it until the wall has around the nail is disintegrated :)

This is not an exaggeration.

Templates used for introspection (or anything else really that's modestly complex) are equally hard to reason about for compilers and for programmers. I guess programmers have an advantage when it comes to _efficient_ pattern recognition.

On Monday, 9 April 2018 at 15:30:33 UTC, Stefan Koch wrote:
> On Friday, 6 April 2018 at 21:45:45 UTC, Zach Tollen wrote:
>>
>> I think Walter's reason was that such macros would hide too many idiosyncrasies in how they were programmed, such that a lot of code which seems simple on the surface will actually obfuscate complicated and arbitrary macro-programming patterns. Thus, code that uses them will become much less maintainable, because it is liable to do so many different and hidden things. Also, the tasks for which AST-macros would typically be used are already largely accommodated by templates and other features. Thus, the real need for them isn't that high.
>
> I think it's time to revisit this.
>
> The reason being that templates are only well suited to very specific types of meta-programs which have a low degree of parameterization and a low degree of abstraction.
>
> Using templates to introspect and manipulate types is like using a hammmer's flat back to remove a nail.
> It _can_ be done but with an absurd amount of work.
> You just have to remove all of the wall around the nail by pounding it until the wall has around the nail is disintegrated :)
>
> This is not an exaggeration.
>
> Templates used for introspection (or anything else really that's modestly complex) are equally hard to reason about for compilers and for programmers. I guess programmers have an advantage when it comes to _efficient_ pattern recognition.

Wouldn't AST macros require either to standardize (and freeze) AST representation within the compiler or to maintain an equally frozen alternate representation to be exposed? I can't see how that wouldn't make the compiler's development slower since all of a sudden changing the internal representation would impact user code, especially given the low number of breaking changes that are accepted today.

I have little experience with AST macros outside Lisp where its homoiconicity avoids the issue almost completely so pardon the naive question.

On 2018-04-09 18:39, Cym13 wrote:
> On Monday, 9 April 2018 at 15:30:33 UTC, Stefan Koch wrote:
>> On Friday, 6 April 2018 at 21:45:45 UTC, Zach Tollen wrote:
>>>
>>> I think Walter's reason was that such macros would hide too many idiosyncrasies in how they were programmed, such that a lot of code which seems simple on the surface will actually obfuscate complicated and arbitrary macro-programming patterns. Thus, code that uses them will become much less maintainable, because it is liable to do so many different and hidden things. Also, the tasks for which AST-macros would typically be used are already largely accommodated by templates and other features. Thus, the real need for them isn't that high.
>>
>> I think it's time to revisit this.
>>
>> The reason being that templates are only well suited to very specific types of meta-programs which have a low degree of parameterization and a low degree of abstraction.
>>
>> Using templates to introspect and manipulate types is like using a hammmer's flat back to remove a nail.
>> It _can_ be done but with an absurd amount of work.
>> You just have to remove all of the wall around the nail by pounding it until the wall has around the nail is disintegrated :)
>>
>> This is not an exaggeration.
>>
>> Templates used for introspection (or anything else really that's modestly complex) are equally hard to reason about for compilers and for programmers. I guess programmers have an advantage when it comes to _efficient_ pattern recognition.
> 
> Wouldn't AST macros require either to standardize (and freeze) AST representation within the compiler or to maintain an equally frozen alternate representation to be exposed?

Yes. It can have two separate ASTs, one internal for the compiler and one external that are used together with the AST macros. They would probably quite similar but they don't need to be exactly the same.

> I can't see how that wouldn't make the compiler's development slower since all of a sudden changing the internal representation would impact user code, especially given the low number of breaking changes that are accepted today.

We already have three clients using the AST: LDC, GDC and the DMD Dub package.

-- 
/Jacob Carlborg

On Monday, 9 April 2018 at 15:30:33 UTC, Stefan Koch wrote:
> [snip]
>
> Using templates to introspect and manipulate types is like using a hammmer's flat back to remove a nail.
> It _can_ be done but with an absurd amount of work.
> You just have to remove all of the wall around the nail by pounding it until the wall has around the nail is disintegrated :)
>
> This is not an exaggeration.
>
> Templates used for introspection (or anything else really that's modestly complex) are equally hard to reason about for compilers and for programmers. I guess programmers have an advantage when it comes to _efficient_ pattern recognition.

Could AST macros replace things like @safe/@nogc and enable the user to create their own (like a @supersafe that disallows @trusted)?

On Friday, 13 April 2018 at 00:37:39 UTC, jmh530 wrote:
>
> Could AST macros replace things like @safe/@nogc and enable the user to create their own (like a @supersafe that disallows @trusted)?

Short Answer: Yes. If the AST-Macro facilities are built with that use in mind.

On Friday, 6 April 2018 at 20:33:10 UTC, Chris Katko wrote:
> Sorry if this is "re-opening" an old thread, but did anything come from this and DIP50? It seems like a really interesting concept and this thread was one of the first results for a Google search.
>
> Thanks.

Thanks for reminding me about this thread, I thought I would see how close i could get to having this work now that I know D's edges better:

On Sunday, 11 July 2010 at 13:29:36 UTC, Michel Fortin wrote:
>
> That said, I don't feel like I'm cheating when using string mixins. I find them a quite good substitute to AST macros. And perhaps string mixins are simpler too: you don't have to learn a new macro syntax, you just manipulate strings. Though I'm still waiting for the day we can use string mixins in expressions to do things like this:
>
> 	int num = 1;
> 	string result = substitute!"Number: $num";
> 	assert(result == "Number: 1");

So what i came up with was:

--- substituter.d
module substituter;

string substituteForMixin(string input){

    input ~= ` `;
    string output = ``;
    size_t l= input.length-1;

    for(size_t i=0; i<l; i++){
        if(input[i]=='#' && input[i+1]=='{'){
            output ~= (i==0) ? `to!string(` : `"~to!string(`;
            i+=2;
            while(input[i] != '}'){
                output ~= input[i++];
            }
            output ~= (i==l-1) ? `)` : `)~"`;
        }else{
            if(i==0) output ~= `"`;
            output ~= input[i];
            if(i==l-1) output ~= `"`;
        }
    }
    return output;
}

mixin template Substituter(){
    string substitute(alias string ident)(){
        import std.conv : to;
        mixin(`return `~substituteForMixin(ident)~`;`);
    };
}

@safe unittest{

    mixin Substituter;

    int number = 42;

    assert(
        substitute!("number is #{number}")
        == "number is 42"
    );
    assert(
        substitute!("#{number+1} is the result of number+1")
        == "43 is the result of number+1"
    );
    assert(
        substitute!("you get #{number*number} when you square number")
        == "you get 1764 when you square number"
    );
}
---

I think thats as clean as I can make it currently. Also I'm using #{} syntax to allow basic expressions and converting to!string on everything.

Not sure if this syntax is documented but you can use enum like:

enum wrapmixin(alias string s) = {mixin(s)};
enum wrapmixin(alias string s){mixin(s)};

And the mixin() is run at the time the enum is called. The only problem with this though is the lambda/function is defined in the scope enum was defined not where the enum was called. So I think its just sugar for:

template wrapmixin(alias string s){alias wrapmixin = {mixin(s);}}
template wrapmixin(alias string s){auto wrapmixin(){mixin(s);}}

To work around this I used a mixin template to work around the scope issue.

Now to bring us back to 2018, I think that this enum version of the code should inject the function/lambda into the scope of it's caller. That would allow us to do a lot of cool things.

Also the other idea I had was to have mixin functions that only take compiletime args (they are UFCS-able though, unlike templates) and mixin themselves when called like:

---
mixin add1Xtimes(alias int a, alias int t){
    uint i=t;
    do{a++;}while(--i);
    return a;
}
@safe unittest{
    uint n=0;
    n.add1Xtimes!(4).add1Xtimes!(6);
    assert(n==10);
}
---

And that becomes:

---
@safe unittest{
    uint n=0;
    uint __m0_i=4;
    do{n++;}while(--__m0_i);
    uint __m1_i=6;
    do{n++;}while(--__m1_i);
    assert(n==10);
}
---

Notice that if the return is never used it's stripped out.

I haven't really given much thought to the implementation of this though.

On Friday, 13 April 2018 at 11:54:12 UTC, David Bennett wrote:
> Also the other idea I had was to have mixin functions that only take compiletime args (they are UFCS-able though, unlike templates) and mixin themselves when called like:
>
> ---
> mixin add1Xtimes(alias int a, alias int t){
>     uint i=t;
>     do{a++;}while(--i);
>     return a;
> }
> @safe unittest{
>     uint n=0;
>     n.add1Xtimes!(4).add1Xtimes!(6);
>     assert(n==10);
> }
> ---
>
> [snip]
>
> I haven't really given much thought to the implementation of this though.

With more thought thats probably not a good solution as its different from the rest of the language and would be harder then it needs to be in order to integrate with the front-end.

What could work though is having a new AutoMixinDecloration that was inited with an expression. This expression would contain a call chain to regular functions, templates, other AutoMixinDecloration etc. Then when it's called it dups the contents of these functions into the callers scope.

---
auto ref incXTimesAndCount(string cident, alias uint csym, uint times)(auto ref uint a=0){
    uint t = times;
    do{
        mixin(cident~"++;");
        csym++;
        a++;
    }while(--t);
    return a;
}

auto mixin incXAndXTimesAndCount(string cident, alias uint csym, uint t1, uint t2)
                = incXTimesAndCount!(cident, csym, t1).incXTimesAndCount!(cident, csym, t2);

@safe unittest{
    uint n=10;
    uint c1=0;
    uint c2=0;
    uint r = n.incXAndXTimesAndCount!("c1", c2, 4, 6);
    assert(n==20);
    assert(c1==10);
    assert(c2==10);
    assert(r==20);
}
---

And it lowers to this in the front end:

---
@safe unittest{
    uint n=10;
    uint c1=0;
    uint c2=0;
    uint __m0_t=4;
    do{n++; c1++; c2++;}while(--__m0_t);
    uint __m1_t=6;
    do{n++; c1++; c2++;}while(--__m1_t);
    uint r = n;
    assert(n==20);
    assert(c1==10);
    assert(c2==10);
    assert(r==20);
}
---

Or it might be even possible to make the current TemplateMixinDeclaration to act like this if you add the auto keyword to the declaration with a symbol with the same name in the scope and the above was just sugar to this:

---
auto mixin template incXAndXTimesAndCount(string cident, alias uint csym, uint t1, uint t2)
{
    alias incXAndXTimesAndCount
            = incXTimesAndCount!(cident, csym, t1).incXTimesAndCount!(cident, csym, t2);
}
---