On 11/12/2013 3:06 AM, Don wrote:
> On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:
>> I forgot to mention that "expression templates" can be used in D in an
>> equivalent manner that they are used in C++,
>
> They are crippled compared to C++, because you have no control over the return
> type of opCmp and opEquals, which means that you can't have expression templates
> involving comparisons. That's extremely limiting.

It is limiting, but I don't know about extremely limiting. Eric Anderton's regex engine didn't need them. And, if you really do need them, you can always define the templates as regular names:

   lessThan(a,b)
   equals(a,b)

etc. Not the greatest, but not unworkable.

On 11/12/2013 09:36 PM, deadalnix wrote:
> The thing I'd like to be able to do it to create a async/await/yield
> like mechanism, purely as library.

That's a prime example for AST macros.

On 11/10/2013 10:20 PM, Jacob Carlborg wrote:
> I've been thinking quite long of how AST macros could look like in D.
> I've been posting my vision of AST macros here in the newsgroup a couple
> of times already. I've now been asked to create a DIP out of it, so here
> it is:
>
> http://wiki.dlang.org/DIP50
>
Thanks, looks like an insightful discussion.

On 11/12/2013 10:20 AM, Walter Bright wrote:
> I confess I have some serious reservations about AST macros in general:
>
> 1. I've seen very heavy use of such macros in macro assemblers. What
> happens is people use it to invent their own (very baroque) language on
> top of the existing assembler one. Anyone trying to read the code has to
> learn this new unique language, and given the limitations of the macro
> capability, it often comes with unfixable bizarre corner cases and
> limitations.
>
Yes, this is a big issue.
I have similar concerns about UDA, but it's easier to put the most needed UDAs into phobos making them more standardized.

> 4. AST macros, pretty much by definition, manipulate the AST. However,
> if you examine the semantic routines in DMD, a *lot* more is required to
> do more significant things. The symbol table must be consulted, etc. I
> just don't see how one could hope to implement something like function
> overloading with AST macros.  Or template type deduction. Or closure
> semantics, attribute inference, etc. Sure, some forms of foreach can be
> rewritten into for statements by manipulating the AST, but not the more
> interesting forms, such as the ones that deal with opApply().
>
That's the question we'd have to answer.
I think what we should target for is that constructs like async/await or SIMDifying of code can be implemented using AST macros.

> For example, the Linq example in the DIP is not compelling, as
> aesthetically nicer code can be written using D's ranges and algorithms:
>
>      auto data = arr.filter!(x => x > 5).array;
>
> I see no compelling advantage in trying to make D code look like C#; to
> be blunt it's like the old:
>
>      #define BEGIN {
>      #define END }
>
> macros used in old C code to make it look like Pascal.
>
True

On Tuesday, 12 November 2013 at 23:34:38 UTC, Martin Nowak wrote:
> On 11/12/2013 09:36 PM, deadalnix wrote:
>> The thing I'd like to be able to do it to create a async/await/yield
>> like mechanism, purely as library.
>
> That's a prime example for AST macros.

First of all: I haven't used macros extensively yet. Its merely a theoretical viewpoint, so don't take the arguments well-established.

Macros could be used as in ScalaCL ( I mean the latest version, which is experimental : https://github.com/ochafik/ScalaCL ). I like the idea using the host language ( or a subset of it) to program GPUs, not C, like PyOpenCL does. It reduces the complexity of the problem when you switch target of the algorithm from CPU to GPU and back.

Macros could also be used in (re)implementing some of the basic syntax - that's what developers of Nemerle did. Somebody can argue whether it was a good idea or not, but it would be interesting to disable syntax by disabling the macro that transforms it by a single line in the file ( not importing the macro ). ( Could be useful for microcontroller platforms to control the runtime's memory usage by disabling certain features - not compiling them into the executable )

In AOT compilation, there isn't any difference between macros and CTFE functions which do something with AST. That would be a way to implement them - creating a compiler API which could only be used by compile-time functions.

Yeah, and I know, it's crippling when code is ambigous. But I also think macros are the best way to define interfaces for modules. Using them can obfuscate code very easily when not enough care is taken handling them and it can make the learning curve much steeper, but there always existed ill-designed API-s and we could still use the good old tools to get the API done :) . Maybe it wouldn't be as good as a macro-ed API would be, but still works. If we choose the right tool, it can lower the complexity of the API.

Let's have a look at the feature: pointers. A very powerful, be also dangerous weapon that cal blow up the entire program suddenly. With some techniques used to fend off the problems it can cause, it can be much safer. Rust made a trial with their pointer system. I don't think it is a solution, but ignoring the problem isn't either.

The second example: the first cannons blew up, killing people. But time showed technology can evolve.
The area of macros - in my opinion - is not explored yet. Only the use of them can reveal the ways to make them better. Theoretical scientists cannot create good solutions for everyday use in the first place. It's a trial-and-error process.

By allowing usage of macros only in certain cases, language desginer can make the burden of the coders lighter ( although I like statement macros, they can be very ambigous ). Also, if designer make macros visually very distinct from other tokens, users can read the code even easier, but sense is needed when choosing a syntax which fits into the language. ( #-s are very alien from D, in my opinion ).

Personally I wouldn't like hacking on the compiler or creating a whole new parser to do the same as we could do with macros, and I suggest a trial.
Another option is to  wait until somebody comes up with an idea and when time proved if the solution is right, include it - doing this saves a lot of work. That's how cutting-edge technologies work ( even if AST macros are an old idea ).

On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu wrote:
> On 11/12/13 7:21 AM, Ellery Newcomer wrote:
>> the .NET framework can do this because it exposes an api for querying,
>> building, and compiling asts.
>>
>> D cannot do this because it doesn't. (and I have tried to make it work)
>
> Maybe the problem needs to be reformulated for D. I think an SQL mixin that either stays unchanged (for DB engines) or translates to a D expression (for native D data types) would be doable, nontrivial, interesting, and instantly usable for people who already know SQL without any extra learning. In other words... actually better than Linq.
>
> Andrei

I was using linq as an example of what we could do with this DIP or what it should be able to do. It wasn't a suggestion of how we should implement such a library.

I also identified with it that having scoped variables from the initiation call is required for safety as well as pragmas for error. Later Jacob alerted me to warning as well. Will submit pull once warning is done.

I also suggested a generic lexer with rules to make it well defined. With it also if's like templates have for macro over loading but nobody was keen for this hence not on wiki.

Yes this can be done already in theory however it doesnt make it not a valid use case.

On 2013-11-12 10:20, Walter Bright wrote:

> I confess I have some serious reservations about AST macros in general:
>
> 1. I've seen very heavy use of such macros in macro assemblers. What
> happens is people use it to invent their own (very baroque) language on
> top of the existing assembler one. Anyone trying to read the code has to
> learn this new unique language, and given the limitations of the macro
> capability, it often comes with unfixable bizarre corner cases and
> limitations.

There's a word for that, it's called DSL (Domain Specific Language) :)

> This got so bad that in some cases I know of, the poor sap who is given
> the job of making improvements resorted to running the assembler to
> generate an object file, then disassembling the object file back into
> source code! Something went very wrong for this to be necessary. I know
> in my own assembler work I have abandoned all use of macros.

DSL's are so popular so many languages are specifically designed to make it easy to create DLS's.

> (Note that macros in typical assemblers are far more powerful than C's
> macro language, which stands out for being pathetically underpowered.)
>
> 2. Macros are something that should be used very sparingly. However,
> this is not what happens. I used to be perplexed that even top shelf
> C/C++ programmers tend to write what I not so humbly consider to be
> pretty abusive use of macros. Now it just saddens me.

It depends on what kind of macros we're talking about. For C preprocessor macros, absolutely. But just because to different features have the same word "macro", you shouldn't put the in the same box. I guess I shouldn't have used the word "macro" at all in the DIP.

> 3. Lisp is a language that encourages users to write macros to pretty
> much invent a custom language for the task at hand. Heavy use of such
> makes the code unrecognizable as being Lisp code. I believe a language
> ought to have some "anchors" that the person reading the code can
> reliably recognize. (GO has taken this idea pretty far.)

Lisp doesn't have any syntax to begin with.

> 4. AST macros, pretty much by definition, manipulate the AST. However,
> if you examine the semantic routines in DMD, a *lot* more is required to
> do more significant things. The symbol table must be consulted, etc. I
> just don't see how one could hope to implement something like function
> overloading with AST macros. Or template type deduction. Or closure
> semantics, attribute inference, etc. Sure, some forms of foreach can be
> rewritten into for statements by manipulating the AST, but not the more
> interesting forms, such as the ones that deal with opApply().

Macros are not for implementing function overloading or template type deduction, we already have those features. I don't consider these easy to implement and they're not lowered in the same way as "scope" or "foreach" are.

> There are some statements in the DIP about the Context parameter and
> it'll provide semantic information, but I don't see how this can work
> without making it an ever-expanding collection of arbitrary methods.

We'll just put in what we want and need there. At the worst case you'll get access to the whole compiler. Which is what we want any way, at least the front end, to be usable as a library.

> 5. We've said "no" in the past to things like user-defined tokens, which
> are closely related.

No, they're not.

> 6. Note that D has a limited form of AST macros with mixin templates.

Yes, I do know. Note that they are too limiting compared to AST macros.

> To sum up, any AST macro proposal has some tough barriers to get over.
> Within its necessarily severe semantic limitations, it has to deliver a
> pretty compelling set of solutions, while avoiding the morass of
> incomprehensibility that far too many macro systems become in practice.

Note that AST macros (according to the DIP) is not anything like C preprocessor macros. Basically the only thing they have in common is the name, "macro". It's not a valid comparison.

C macros are textual macros. They can create completely new syntax. AST macros cannot. They work at the semantic level in the compiler, just as templates. They can add new semantic meaning to existing syntax. Just like templates, macros need to lex and parse correctly. Templates do not need to be semantically correct if they're not instantiated.

-- 
/Jacob Carlborg

On 2013-11-12 11:03, Walter Bright wrote:

> Here's a project to add Linq support to C++:
>
> http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language
>
>
> There's also:
>
> http://pfultz2.github.io/Linq/
>
> https://github.com/hjiang/linqxx/wiki

Again, operator overloading in D is too limiting to implement something Linq like.

-- 
/Jacob Carlborg

On 2013-11-12 19:01, Rob T wrote:

> My personal main need for macros at this point, is to make up for a lack
> of better reflection, but as has been pointed out, it's not clear how
> macros would help anyway without access to the sort of reflection that
> I'm currently lacking.

Macros would need a reflection API to be usable.

> What we seem to be discussing, is the need to unify within D the
> abilities to perform better reflection along with a means to automate
> code generation during compile time. The current way reflection and
> mixins are done, are totally different, almost like two different
> languages with D, so it would be nice to have a more unified system that
> more closely resembles D itself.
>
> BTW, I agree with Walters concerns about AST macros. It would be ugly to
> work with code that is not comprehensible due to over use of macros that
> attempt to redefine the language into something else. OTOH there's also
> significant advantages to be able to add certain things into the
> language that are currently lacking. For example, I've experimented with
> a method of implementing co-routines that use switch statements, but
> it's very ugly to do without macro support to the point of being
> impractical.
>
> --rt


-- 
/Jacob Carlborg

On 2013-11-12 21:03, Joseph Cassman wrote:

> After using string mixins for a while I have come to feel much like your
> statement in (1). I like the power and flexibility of string mixins.
> Text and strings are understandable and straightforward to manipulate.
> This is a clear win in my opinion.
>
> There are also some things I find difficult at the moment.
>
>   - One is the process of what to do when things do not go as expected.
> To debug I currently use pragma statements and try to visualize what is
> going on in my mind. This is do-able. But I feel it could be better.

If we got macros we should really have a good way to debug them. At least a flag to shows how the macros are lowered.

>   - It would be nice to be able to use some more D language concepts to
> construct the string mixin inside the template. For example, sometimes I
> would like to use a loop to construct the result.
>
> If these two items could be addressed I would be pretty satisfied with
> using string mixins as they are.
>
> One win that the AST macro concept provides that string mixins does not
> is manipulation of code syntax trees (either at compile or run time). It
> would be nice to combine the various code manipulation ideas that have
> been tossed around before into a single module, say std.meta or
> std.reflection. In addition to combining the various current code
> manipulation functionality into a single api (e.g. __traits,
> std.traits), it could provide functionality to work with code-in-use.
> For example, it would be pretty cool to be able to do the following (the
> actual syntax for constructing the code is not that important to me, but
> the underlying functionality for manipulating code is).
>
> sample.d
>
>      import std.meta;
>      void main(string[] args) {
>          auto e = construct("a + %s", args[1]);
>          writefln("%s", e(5));
>          auto f = construct(e + "b");
>          writefln("%s", f(?, 10).toDebugString);
>          writefln("%s", f(5, 10));
>      }

Yes, but the problem with string mixins is that you need to reparse a lot of code. With string mixins it becomes:

1. The compiler parses the code and creates an AST (fine so far)
2. Does type checking an other stuff
3. Library code parses a string into an AST
4. Library code manipulates the AST
5. Library code converts the AST back to a string
6. The string is interested with a string mixin at the calling context
7. The compiler parses the code and creates an AST
8. Does type checking an other stuff

With AST macaros you can avoid many of these steps:

1. The compiler parses the code and creates an AST
2. Does type checking an other stuff
3. Library code (macro) manipulates the AST
4. Compiler inserts the manipulate AST into the calling context
5. Does type checking an other stuff

Instead of eight steps we get five steps. We remove the unnecessary conversions between AST's and strings.

-- 
/Jacob Carlborg