Walter Bright wrote:
> Sean Kelly wrote:
>> Please note that I'm not criticizing in-language DSL parsing as a general idea so much as questioning whether this is truly the best example for the usefulness of such a feature.
> 
> Compile time DSL's will really only be useful for relatively small languages. For a complex DSL, a separate compilation tool will be probably much more powerful and much more useful.

True, but then there's also the scavenging approach: a small DSL can be a reduced interpretation of a complex language. For example, one can imagine parsing an HTML file to extract stuff of interest, while literally skipping over the complexity of HTML (and javascript).

And I've already given the example of parsing SQL code to generate a D mapping. I've actually done this in 1998: starting from CREATE VIEW statements, I was generating C++ code with one struct per view.

> I don't know anything about database languages, so I'm no help there.
> 
> One example of a highly useful compile time DSL is the regex package that Don Clugston and Eric Anderton put together. With better metaprogramming support, this kind of thing will become much simpler to write.
> 
> There's often a need for custom 'little languages' for lots of projects. Most of the time, people just make do without them because they aren't worth the effort to create. I hope to make it so easy to create them, that all kinds of unforeseen uses will be made of them.
> 
> I'll give an example: I often have a need to create parallel tables of data. C, C++, and D have no mechanism to do that directly (though I've used a macro trick to do it in C). With a DSL, this becomes easy.

And parallel hierarchies required by some design patterns are a bitch. With code generation it will be easy to rein in.


Andrei

Bill Baxter wrote:
> Wow, this is sounding sillier and sillier.
> It seems pretty clear to me that the answer is simply that Andrei doesn't really know enough about RoR give a concrete example of how better metaprogramming would be useful for DeRailed.  He pretty much said as much in the last mail.  But it would be good if he gave some more practical, concrete examples of places where it would help.

For the record, I know next to nothing about RoR except that:

1) RoR has clearly been the "killer app" that has launched Ruby into one of the top languages in use today.

2) I thought RoR was a DSL addon to Ruby. Perhaps such supposition was a huge misunderstanding on my part.

I think it's way cool that you guys are working on DeRailed. If better compiler DSL technology won't help that, well, rats! But I don't know enough about the problem DeRailed is trying to solve to suggest any way in which a DSL might be used with it.

> Note that what's going on here is *talk* about features that may or may not get into DMD any time soon.  In fact you could say this whole discussion has been about *preventing* features from getting introduced.  At least in an ad-hoc manner. This meat of this metaprogramming discussion started with Walter saying he was thinking of adding compile time regexps to the language.  Without any discussion about whether that's a good thing or not and what the ramifications are, then it's just going to happen, whether it's good for D or not.

I beg to differ on that. The reason I started this thread was to not post a fait accompli, but to elicit discussion, especially since builtin regex has been thoroughly reviled in the past.

And it's pretty clear that it's going down in flames again <g>. I think there's a better way now, and this discussion has helped find it.

> So the question becomes what should D look like?  Rather than add hoc features, what do we really want D's metaprogramming to look like?

Yes, indeed.

What Kris brings to the table is he's building key foundation libraries for D. What Andrei brings is the academic rigor that I lack.

I know from these messages it seems that all we talk about is metaprogramming, but actually most of our discussions (and most of Andrei's work on D) are about filling in mundane gaps in the language, like the lack of a proper const. And I know Kris really wants a useful const <g>.

Kirk McDonald wrote:
> Never mind what this actually does. The problem at hand is somehow generating a class like this at compile-time, possibly given only the class Foo.

Tell me exactly what you need.

Walter Bright wrote:
> For the record, I know next to nothing about RoR except that:
> 
> 1) RoR has clearly been the "killer app" that has launched Ruby into one of the top languages in use today.
> 
> 2) I thought RoR was a DSL addon to Ruby. Perhaps such supposition was a huge misunderstanding on my part.
> 

Ruby, as a language, is well-suited to defining DSLs within itself. The strength of RoR (which speaks to Ruby's power) is that it is both merely a Ruby library and a DSL.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org

Walter Bright wrote:
> Kirk McDonald wrote:
>> Never mind what this actually does. The problem at hand is somehow generating a class like this at compile-time, possibly given only the class Foo.
> 
> Tell me exactly what you need.

Given a class Foo, allow _during compilation_ enumeration of all of its embedded symbols (fields, methods, types, aliases), with full access to their type information (notably return and argument types for methods), transitively. Allow arbitrary code generation using all of that information.

Andrei

Andrei Alexandrescu (See Website For Email) wrote:
> Walter Bright wrote:
>> Kirk McDonald wrote:
>>> Never mind what this actually does. The problem at hand is somehow generating a class like this at compile-time, possibly given only the class Foo.
>>
>> Tell me exactly what you need.
> 
> Given a class Foo, allow _during compilation_ enumeration of all of its embedded symbols (fields, methods, types, aliases), with full access to their type information (notably return and argument types for methods), transitively. Allow arbitrary code generation using all of that information.

That will eventually happen, but I was hoping there was some subset that Kirk needed which I can provide relatively quickly.

Walter Bright wrote:
> I know from these messages it seems that all we talk about is metaprogramming, but actually most of our discussions (and most of Andrei's work on D) are about filling in mundane gaps in the language, like the lack of a proper const. 

Ehm. Somehow I thought it's actually interesting. Reminds me of Tom Sawyer's neighbors painting the fence for him :o).

Andrei

Walter Bright wrote:
> Andrei Alexandrescu (See Website For Email) wrote:
>> Walter Bright wrote:
>>> Kirk McDonald wrote:
>>>> Never mind what this actually does. The problem at hand is somehow generating a class like this at compile-time, possibly given only the class Foo.
>>>
>>> Tell me exactly what you need.
>>
>> Given a class Foo, allow _during compilation_ enumeration of all of its embedded symbols (fields, methods, types, aliases), with full access to their type information (notably return and argument types for methods), transitively. Allow arbitrary code generation using all of that information.
> 
> That will eventually happen, but I was hoping there was some subset that Kirk needed which I can provide relatively quickly.

Without knowing the details, my guess is that nonstatic methods with their return and argument types should be enough.

Andrei

Andrei Alexandrescu (See Website For Email) schrieb am 2007-02-10:
> Bill Baxter wrote:
>> As for Andrei having Walter's ear.  I think Andrei has Walter's ear mostly because Andrei is interested in the same kinds of things that interest Walter.  I think everyone can tell by now that Walter pretty much works on solving the problems that interest him.   Right now (and pretty much ever since 'static if') the thing that seems to interest him most is metaprogramming.  Hopefully some day he'll get back to being interested in reflection.  But if he's really got the metaprogramming bug, then that may not be until after he's got D's compile time framework to a point where he feels it's "done".  But only Walter knows.
>
> There is a deeper connection between runtime reflection and compile-time reflection than it might appear.
>
> In the runtime reflection scenario, the compiler must generate, for each user-defined type, an amount of boilerplate code that allows symbolic inspection from the outside, and code execution from the outside with, say, untyped (or dynamically-typed) arguments.
>
> The key point is that the code is *boilerplate* and as such its production can be confined to a code generation task, which would keep the compiler simple. The availability of compile-time introspection effectively enables implementation of run-time introspection in a library.
>
> For example:
>
> class Widget
> {
>    ... data ...
>    ... methods ...
> }
>
> mixin Manifest!(Widget);
>
> If compile-time introspection is available, the Manifest template can generate full-blown run-time introspection code for Widget, with stubs for dynamic invocation, the whole nine yards.
>
> This is nicer than leaving the task to the compiler because it relieves the compiler writer from being the bottleneck.

Going the compile-time introspection -> boilerplate way is possible but not the only solution for runtime reflection. Combining "dmd -L/DETAILEDMAP ...", Flectioned's Symbols.scanOptlinkMap and existing TypeInfo solves 80% of the boiler plate issue. The only remaining issue are non-static non-function aggregate members.

I'm not arguing against powerful compile-time reflection, in fact I love it.

Thomas

Walter Bright wrote:
> Kirk McDonald wrote:
>> Never mind what this actually does. The problem at hand is somehow generating a class like this at compile-time, possibly given only the class Foo.
> 
> Tell me exactly what you need.

Given a class, I need a way to get a list of all of its member functions at compile-time. One might naively think a tuple of aliases to the class's member functions would do, except that this does not provide enough information in the case of overloaded functions. In other words:

class Foo {
    void foo() {}
    void bar() {}
    void bar(int i) {}
}

The mechanism I suggest would get us this tuple:

methodtuple!(Foo) => Tuple!(Foo.foo, Foo.bar);

(Particularly, I do not think this tuple should include inherited methods.)

However, there is no way to distinguish the two forms of bar at compile-time. Currently, this is only enough information to get the first form with the empty parameter list, since it is lexically first.

This turns out to be a more general problem, which I will now recast in terms of global functions.

void foo() {}
void foo(int i) {}
void foo(char[] s) {}

The second mechanism needed would give us a tuple of the function types shared by this symbol:

functiontuple!(foo) => Tuple!(void function(), void function(int), void function(char[]))

functiontuple!(Foo.bar) => Tuple!(void function(), void function(int))

The third thing needed is the ability to detect function parameter storage classes, as you have discussed on this NG in the past.

Fourth is some way of detecting whether a function has default arguments, and how many it has. This can be done with templates (as std.bind shows), but direct language support would be a great deal cleaner.

Fifth is actually satisfied in a way by the new mixins. This is the ability to call an alias of a member function on an instance of the class or struct. This is very much like C++ pointers to member functions, with the difference that it happens entirely at compile-time. (Meaning that type mismatches, viz. trying to call a method of one class on an instance of another, can be detected.) I am not sure what the syntax for this should be, however. And, as I said, the new mixins can handle it, when combined with Don Clugston's nameof.

The first three are absolutely necessary. The other two can be done with templates in current D, and would merely be nice.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org