On 14/12/2019 2:31 AM, rikki cattermole wrote:
> 
> I've talked about this before with async, but I'm partial towards having a specialized string like q"" that will always be mixed in i.e.

That should be q{} of course, damn widget rendering! I'm even forgetting basic tokens.

On Friday, 13 December 2019 at 13:17:24 UTC, Jacob Carlborg wrote:
> Yet again we have an issue that AST macros can solve but Walter and company prefers to have specialized features instead of generic ones.

This is a valid Forth program:

  : rotchar ( c -- c' )
    dup  [char] a [ char m 1+ ] literal within 13 and
    over [char] A [ char M 1+ ] literal within 13 and +
    over [char] n [ char z 1+ ] literal within -13 and +
    over [char] N [ char Z 1+ ] literal within -13 and + + ;

  : rot13 ( -- )
    begin
      begin 0 parse dup while
        2dup bounds do i c@ rotchar i c! loop evaluate
      repeat 2drop
    refill 0= until ;

  .( hello) rot13 fcnpr .( jbeyq)
  pe olr

Output:

  hello world

Once evaluation hits ROT13, subsequent code is only evaluated
after getting rot13'd. So that FCNPR is actually SPACE ,etc.
Through PARSE and REFILL , ROT13 takes over compiling the rest
of the program. You can even do it recursively,

  rot13 .( ebg13 jbeyg) pe
  ebg13 .( rotated all the way back again!) cr

Common Lisp has similar capabilities with the rarely used
reader macros. Yes, even the "programmable programming
language" has features its adherents are loathe to use.

So generic solutions are not that hard to think of, and the
idea of languages where normal user words can participate in
compilation to this degree have been around for a long time.
The problem is that nobody wants to even think about touching
"other people's code" in the resulting family of languages.

I can't deny that you have some legitimate pattern or gripe but
I think the stated bias is fine.

On Friday, 13 December 2019 at 13:53:00 UTC, mipri wrote:
> So generic solutions are not that hard to think of, and the
> idea of languages where normal user words can participate in
> compilation to this degree have been around for a long time.
> The problem is that nobody wants to even think about touching
> "other people's code" in the resulting family of languages.

That has nothing to do with AST manipulation. Forth was designed for taking minimal space and was edited in non-scrollable screens of ~1000 characters. You can't be more terse. Postscript is very close to Forth though, and you can write clean code in Postscript.

Lisp was also designed as an exercise in minimalism. Lisp and Forth are basically the two of the oldest languages still in use. If anything that shows that they got something right... but arcane obviously.

If you want to discuss modern "AST" based languages you should take a look at modern Term Rewriting languages:

http://maude.cs.illinois.edu/w/index.php/The_Maude_System

https://agraef.github.io/pure-lang/

On Friday, 13 December 2019 at 14:29:48 UTC, Ola Fosheim Grøstad wrote:
> On Friday, 13 December 2019 at 13:53:00 UTC, mipri wrote:
>> So generic solutions are not that hard to think of, and the
>> idea of languages where normal user words can participate in
>> compilation to this degree have been around for a long time.
>> The problem is that nobody wants to even think about touching
>> "other people's code" in the resulting family of languages.
>
> That has nothing to do with AST manipulation.

"generic solutions" vs. "specific solutions" has everything to
do with AST manipulation vs. a specific string interpolation
method.

> Forth was designed for taking minimal space and was edited in non-scrollable screens of ~1000 characters. You can't be more terse. Postscript is very close to Forth though, and you can write clean code in Postscript.

What has absolutely nothing to do with anything, is this.

> Lisp was also designed as an exercise in minimalism. Lisp and Forth are basically the two of the oldest languages still in use. If anything that shows that they got something right... but arcane obviously.

And this.

> If you want to discuss modern "AST" based languages you should take a look at modern Term Rewriting languages:
>
> http://maude.cs.illinois.edu/w/index.php/The_Maude_System
>
> https://agraef.github.io/pure-lang/

And, amazingly, this.

Please try harder to read what people say instead of
responding to keywords like 'Forth' and 'Lisp.

That's impolite. I think what you've done here is also impolite.

On Friday, 13 December 2019 at 14:39:41 UTC, mipri wrote:
> That's impolite. I think what you've done here is also impolite.

What is impolite is to try to shoot down the relevance of having AST macros by pointing to Forth and Lisp. What are you trying to achieve by pointing to ancient languages that use cryptic identifiers for historic reasons?

AST macros are no worse than templates.

And certainly a lot better than string mixins!

On Friday, 13 December 2019 at 14:52:27 UTC, Ola Fosheim Grøstad wrote:
> What is impolite is to try to shoot down the relevance of
> having AST macros by pointing to Forth and Lisp.

What I am trying to do is state that a bias towards specialized
features rather than generic ones is a very reasonable bias for
a language designer to have. That's it. If a metaprogramming
DIP came along I wouldn't object that "Lisp tried this and it
was bad".

> What are you
> trying to achieve by pointing to ancient languages that use
> cryptic identifiers for historic reasons?

I mentioned PARSE and REFILL from Forth, and reader macros from
Common Lisp. The age and identifiers of these languages has
nothing to do with the capabilities of those features. What
they represent is a kind of 'final DIP'. If any DIP can be
objected to with "this can be done in a library", then the
final DIP is the one that lets you do anything at all in a
library. If you *don't* have a bias towards specific solutions
then I think you'll eventually come up with a solution like
this, and I propose that it -- not "AST macros" -- is a bad
idea.

> AST macros are no worse than templates.

They really are, though. They're invariably much uglier and
harder to maintain than templates. I think that if you want to
have AST macros, you should certainly have a feature like
templates as both A) a stopgap, so that not everything has to
be done with AST macros, and B) a destination for AST macros.

It's a very different 'template', but consider this Nim:

  import strutils

  func maybeQuote(str: string): string =
    if {' ', ','} in str:
      '\'' & str & '\''
    else:
      str

  template `-->`(str: typed, field: untyped): untyped =
    str & ':' & maybeQuote($field)

  echo "hi" --> "hi"

Output: hi:hi

With a macro you could introduce a syntax

  echo --- hi

With the same output of "hi:hi"

How should you do that? Well, you can do it entirely in the
ugly macro system, or you can have 1-2 lines of code that just
emits `"hi" --> "hi"`, using the template. The macro becomes a
way of augmenting the template with a stringification power,
and the combination of template * macro is much easier to read
and understand than would be a macro generating the final Nim.

I imagine I'd come to the same conclusions about how best to
employ AST macros in D.

> And certainly a lot better than string mixins!

On Friday, 13 December 2019 at 15:24:32 UTC, mipri wrote:
> On Friday, 13 December 2019 at 14:52:27 UTC, Ola Fosheim Grøstad wrote:
>> What is impolite is to try to shoot down the relevance of
>> having AST macros by pointing to Forth and Lisp.
>
> What I am trying to do is state that a bias towards specialized
> features rather than generic ones is a very reasonable bias for
> a language designer to have. That's it. If a metaprogramming
> DIP came along I wouldn't object that "Lisp tried this and it
> was bad".

It was reasonable for D1, which was a simple language!

But when they went full on meta-programming with D2, and stated that this was their focus then they should try to do a lot better than other competing languages in that department.

You have to cut down bloat somewhere. So you have to choose:

1. have lots of special features

2. enable libraries to implement it and have lots of meta programing features

If you do both, you end up with an umanagable mess that goes nowhere.

> I mentioned PARSE and REFILL from Forth, and reader macros from
> Common Lisp. The age and identifiers of these languages has
> nothing to do with the capabilities of those features. What

Well, it kinda does, but regardless. Nobody has stated what AST manipulation in D would look like, so what outdated languages have done isn't really relevant unless someone propose exactly the same.

> they represent is a kind of 'final DIP'. If any DIP can be
> objected to with "this can be done in a library", then the
> final DIP is the one that lets you do anything at all in a
> library. If you *don't* have a bias towards specific solutions
> then I think you'll eventually come up with a solution like
> this, and I propose that it -- not "AST macros" -- is a bad
> idea.

Golden rule in  programming language design:

1. implement it as a library construct first

2. enhance the language so it can be done as a library construct

3. if it still is too heavy on syntax add syntactic sugar for it

4. if it cannot be done at all, then most reluctantly consider a language change, but resist, resist resist…

5. if you find yourself adding more and more features. Freeze the language. Start over from scratch, build a new language, because what you started with has aggregated too much cruft and is in danger of becoming unmanagble. That means there was something insufficient in the language premises.

> How should you do that? Well, you can do it entirely in the
> ugly macro system, or you can have 1-2 lines of code that just
> emits `"hi" --> "hi"`, using the template.

Nobody has suggested that D should add anything ugly.

Did you look at Term Rewriting?

> I imagine I'd come to the same conclusions about how best to
> employ AST macros in D.

Why would you imagine anything about a design nobody has even proposed???

On Friday, 13 December 2019 at 15:38:14 UTC, Ola Fosheim Grøstad wrote:
>> I mentioned PARSE and REFILL from Forth, and reader macros from
>> Common Lisp. The age and identifiers of these languages has
>> nothing to do with the capabilities of those features. What
>
> Well, it kinda does, but regardless. Nobody has stated what AST manipulation in D would look like, so what outdated languages have done isn't really relevant unless someone propose exactly the same.

Walter and Andrei's proposal from 2012 [1] looked like this:

// definition
macro debugPrint(expr) {
    writefln("%s(%s): %s == %s", __FILE__, __LINE__, expr.stringof, expr)
}

// usage
debugPrint(2 + 2);

You'll notice that there's no direct manipulation of the AST--everything is done by substitution.

[1] https://www.youtube.com/watch?v=FRfTk44nuWE&t=1h5m37s

On Friday, 13 December 2019 at 16:07:01 UTC, Paul Backus wrote:
> You'll notice that there's no direct manipulation of the AST--everything is done by substitution.

Yes, and there are many ways to do substitution.

Full term rewriting is extremely powerful, to the extent that it is used for formal proofs and verifiable programming. That is going a bit far.

On the other end of the spectrum you have very limited pattern matching languages like XSLT that compensate for the simplicity by providing entrypoints for scripting (javascript).

So it is possible that something like XSLT (substitution) with CTFE where XSLT use javascript could work.

Pattern matching transforms with some CTFE mixed in can provide elegant AST manipulation. In my experience it leads to code that is relatively easy to read and understand and feels familiar for people with FP background.

On Friday, 13 December 2019 at 16:44:17 UTC, Ola Fosheim Grøstad wrote:
> On Friday, 13 December 2019 at 16:07:01 UTC, Paul Backus wrote:
>> You'll notice that there's no direct manipulation of the AST--everything is done by substitution.
>
> Yes, and there are many ways to do substitution.

The youtube link in my previous post has a time code that will bring you directly to the start of the segment on macros. In that segment, Walter explains in detail how macros would be expanded, with worked examples on the whiteboard. Pattern matching is also discussed.