On 11/12/13 9:03 AM, Dicebot wrote:
> On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:
>> I don't see how AST reflection is needed for generating correct D code
>> from an SQL expression. Dmitry did exactly that with ctRegex. Please
>> illuminate.
>
> Other way around. Generating arbitrary output (including but not limited
> to SQL expressions) from D code.

Yah. That I agree with. That's why I said we should reformulate the problem that LINQ solves.

> As I have already mentioned, it is the
> very same thing we do now with __traits and UDA's but not limited to
> only declarations.

Interesting.


Andrei

On Tuesday, 12 November 2013 at 16:14:57 UTC, John Colvin wrote:
> On Tuesday, 12 November 2013 at 15:21:16 UTC, Ellery Newcomer wrote:
>> On 11/12/2013 06:38 AM, John Colvin wrote:
>>> On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg wrote:
>>>> auto person = Person.where(e => e.name == "John");
>>>>
>>>> Translates to:
>>>>
>>>> select * from person where name = 'John'
>>>>
>>>
>>> for those of us entirely unfamiliar with linq, what is this supposed to
>>> do? Select people with name "John" from a collection of people, like in
>>> sql? It seems trivial to do this using filter, or am I missing
>>> something...?
>>
>> linq provides an interface to query any collection, but what is interesting in this case is
>>
>> Person.where(e => e.name == "John")
>>
>> invokes an engine that builds the necessary sql to issue an equivalent query to your sql database and assembles the results into a range of Person. And it can do this for any arbitrary predicate (well, almost. It doesn't handle function calls to arbitrary code too well).
>>
>> the .NET framework can do this because it exposes an api for querying, building, and compiling asts.
>>
>> D cannot do this because it doesn't. (and I have tried to make it work)
>
> oh, I see. Would AST macros really be enough to make this work in D? "Arbitrary code" is a huge feature space in D, including much that doesn't map well to anything outside of a relatively low-level language, let alone SQL.
> I can see it quickly becoming a nightmare that would be worse than just issuing the predicate as an sql string or some generic equivalent.

Actually,

s/arbitrary code/simple to moderately complex expressions that don't call functions that aren't explicitly handled by the translation engine/

and this is a better representation of what linq can do in e.g. entity framework.

and you could go the route of django models, e.g.

Place.objects.filter(name="Bob's Cafe")

as long as you had enough metadata set up. you could probably get this to work in D, but

A. it looks like crap
B. it in no way resembles sql
C. it would look like even worse crap in D (no named parameters!)
D. it doesn't readily permit you to make queries with complex expression trees (idk, maybe something like linq's

Places.Where(p => p.Elev < 200 && Addresses.Any(a => a.State == p.State))

why am I arguing against this? nobody wants this, nobody asked for this.

Also, sql is to databases what assembly is to cpus. It's low level, [moderately] untyped, and it differs on every single platform. provide access to it, sure, but please don't use it to build your abstractions.

On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:
> On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:
>> I forgot to mention that "expression templates" can be used in D in an equivalent manner that they are used in C++,
>
> They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting.
>
>
> My feeling with AST macros is:
>
> (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus.
>
> Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros.
>
>
> (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals.
>
> The question I find particularly interesting is, "if we had macros, would reflection look different?"

My personal main need for macros at this point, is to make up for a lack of better reflection, but as has been pointed out, it's not clear how macros would help anyway without access to the sort of reflection that I'm currently lacking.

What we seem to be discussing, is the need to unify within D the abilities to perform better reflection along with a means to automate code generation during compile time. The current way reflection and mixins are done, are totally different, almost like two different languages with D, so it would be nice to have a more unified system that more closely resembles D itself.

BTW, I agree with Walters concerns about AST macros. It would be ugly to work with code that is not comprehensible due to over use of macros that attempt to redefine the language into something else. OTOH there's also significant advantages to be able to add certain things into the language that are currently lacking. For example, I've experimented with a method of implementing co-routines that use switch statements, but it's very ugly to do without macro support to the point of being impractical.

--rt

> There is significant regret about the way macros are defined in Scala. Probably not an example to follow.
>
> Andrei

Not an example to follow, but it would definitely
be worth looking into how they defined macros and
what aspect of that caused problems, so we can avoid
making the same mistakes.

On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:
> On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:
>> I forgot to mention that "expression templates" can be used in D in an equivalent manner that they are used in C++,
>
> They are crippled compared to C++, because you have no control over the return type of opCmp and opEquals, which means that you can't have expression templates involving comparisons. That's extremely limiting.
>
>
> My feeling with AST macros is:
>
> (1) AST macros are basically syntax sugar for string mixins. The one and only thing that string mixins have in their favour, is that they have almost no syntax. But that is actually a *huge* plus.
>
> Because it's syntax sugar, an AST macro syntax would be most convincing if you took some existing string mixins from various real projects, and show how beautiful they would become with macros.
>
>
> (2) The big functionality we don't have, is comprehensive reflection. I'd like to see more thought in that area. It seems challenging to define reflection in a way that doesn't expose compiler internals.
>
> The question I find particularly interesting is, "if we had macros, would reflection look different?"

After using string mixins for a while I have come to feel much like your statement in (1). I like the power and flexibility of string mixins. Text and strings are understandable and straightforward to manipulate. This is a clear win in my opinion.

There are also some things I find difficult at the moment.

 - One is the process of what to do when things do not go as expected. To debug I currently use pragma statements and try to visualize what is going on in my mind. This is do-able. But I feel it could be better.

 - It would be nice to be able to use some more D language concepts to construct the string mixin inside the template. For example, sometimes I would like to use a loop to construct the result.

If these two items could be addressed I would be pretty satisfied with using string mixins as they are.

One win that the AST macro concept provides that string mixins does not is manipulation of code syntax trees (either at compile or run time). It would be nice to combine the various code manipulation ideas that have been tossed around before into a single module, say std.meta or std.reflection. In addition to combining the various current code manipulation functionality into a single api (e.g. __traits, std.traits), it could provide functionality to work with code-in-use. For example, it would be pretty cool to be able to do the following (the actual syntax for constructing the code is not that important to me, but the underlying functionality for manipulating code is).

sample.d

    import std.meta;
    void main(string[] args) {
        auto e = construct("a + %s", args[1]);
        writefln("%s", e(5));
        auto f = construct(e + "b");
        writefln("%s", f(?, 10).toDebugString);
        writefln("%s", f(5, 10));
    }

-----

> ./sample 2
> 7
> Numerical Expression: a + 2 + 10
> 17

Implementing the "expression tree" side of AST macros seems like it could be done as a library, which would be a win for maintainability and cross-compiler portability. There is probably required, compiler-related functionality that I cannot see making this unfeasable or possibly difficult to do. Still, looks good to me on paper at least. Perhaps going this route could gain the good functionality of AST macros while also making use of the already implemented features?

Seems to me like some of these ideas tie into previous discussions. Here are two I remember. I am sure there are others.

http://forum.dlang.org/thread/juf7sk$16rl$1@digitalmars.com
http://forum.dlang.org/thread/mailman.1716.1340388570.24740.digitalmars-d@puremagic.com

Joseph

On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu wrote:
>> Other way around. Generating arbitrary output (including but not limited
>> to SQL expressions) from D code.
>
> Yah. That I agree with. That's why I said we should reformulate the problem that LINQ solves.

I had an impression it was exactly the context in which linq was originally mentioned, no idea why discussion has moved from that to expressing DSL's (which is not really a problem in D) :)

>> As I have already mentioned, it is the
>> very same thing we do now with __traits and UDA's but not limited to
>> only declarations.
>
> Interesting.

Reason why I am doing this comparison is that some of the cases why UDA's are so useful applies to AST reflection capabilities too. Most importantly, it allows to have the code which is both valid (and used) D code on its own and used to generate some co-product from it in automatic mode. Similar declarative approach has been used to huge success in vibe.d code (rest, forms) and I see no reasons why it can't be so for imperative one.

On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu wrote:
>> As I have already mentioned, it is the
>> very same thing we do now with __traits and UDA's but not limited to
>> only declarations.
>
> Interesting.
>

The thing I'd like to be able to do it to create a async/await/yield like mechanism, purely as library.

@generator
uint foo() {
    writeln("generating !");
    return 0;
    return 1;

    writeln("and the last one");
    return 2;
}

=>

auto foo() {
    enum States {
        S0,
        S1,
        S2,
    }

    struct Generator {
        State s;
        uint current;

        @property front() {
            return current;
        }

        void popFront() {
            final switch(s) with(State) {
                case S0:
                    current = 1;
                    s = S1;
                    return;
                case S1:
                    writeln("and the last one");
                    current = 2;
                    s = S2;
                    return;
                case S2:
                    assert(0, "Nothing more in here");
            }
        }

        @property empty() {
            return s == State.S2;
        }
    }

    auto g = Generator();

    writeln("generating !");
    g.current = 0;
    g.state = State.S0;

    return g;
}

12-Nov-2013 21:03, Dicebot пишет:
> On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:
>> I don't see how AST reflection is needed for generating correct D code
>> from an SQL expression. Dmitry did exactly that with ctRegex. Please
>> illuminate.
>
> Other way around. Generating arbitrary output (including but not limited
> to SQL expressions) from D code. As I have already mentioned, it is the
> very same thing we do now with __traits and UDA's but not limited to
> only declarations.

Actually I couldn't shake off the feeling that macros are just CTFE functions on Ast objects. How objects are created and converted back to source code is a separate question.

If we just had:

//this would invoke compiler's parser at CTFE
auto ast = "x = y;".astof

and have it work at CTFE to return sensible AST (a big if btw).

And then (after some manipulations):
ast.toString() //get back a string of D code

It may help code generation of D --> DSL.

Alternatively one can implement D parser that works at CTFE and watch it crawl :)

-- 
Dmitry Olshansky

On Tuesday, 12 November 2013 at 20:56:34 UTC, Dmitry Olshansky wrote:
> If we just had:
>
> //this would invoke compiler's parser at CTFE
> auto ast = "x = y;".astof
>
> and have it work at CTFE to return sensible AST (a big if btw).
>
> And then (after some manipulations):
> ast.toString() //get back a string of D code
>
> It may help code generation of D --> DSL.

Add there "foo.codeof" in the toolset and it will pretty much give core needed stuff.

However, it will destroy compilation times if not coupled with compiler internal parsing/semantic phase.

Actual "macro" keyword syntax sugar is hardly important to me.

On Tue, Nov 12, 2013 at 9:55 PM, Dmitry Olshansky <dmitry.olsh@gmail.com> wrote:

> If we just had:
>
> //this would invoke compiler's parser at CTFE
> auto ast = "x = y;".astof
>
> and have it work at CTFE to return sensible AST (a big if btw).
>
> And then (after some manipulations):
> ast.toString() //get back a string of D code

Which you still have to mixin, btw. Which means any tree manipulation
you do must be done at compile-time.
Unless, of course,  you then put the resulting code into another file,
to be compiled and loaded afterwards.


> It may help code generation of D --> DSL.
>
> Alternatively one can implement D parser that works at CTFE and watch it crawl :)

*cough cough*

But, as other have said, these would be very un-hygienic macros, with
no knowledge of the surrounding environment.
Unless said CT-compatible parser could also somehow determine the
local symbols (you could pass it 'by hand', but who would do that,
realistically?)