Georg Wrede wrote:
> Back to the issue at hand:
> 
> What I'm still saying, is that (even though Walter and Don have made completely kick-ass progress here), there's a glass ceiling waiting for us at this rate. By the time we have evolved the D meta business into something that has cleared all major obstacles, it will be a hard to use, cumbersome, and unobvious metalanguage. (( but still a hell of a lot nicer and of course much more powerful than C++ templates ever ))

I'm not sure about this. What do you see as the major obstacles?
I had a long list of them once, it's very short now.
Note that almost all of the improvements which have been made, have made
metaprogramming *more* similar to ordinary D, not less.

The main reason D metaprogramming is so much simpler than C++, is that we have "static if" instead of using partial template specialisation.

I'll just summarise the changes I proposed that Walter has implemented:

* strings and reals as template value arguments.
 ---> makes metafunctions able to have a greater variety of parameters,
      and therefore more like normal D.
* Improved compile-time constant folding, most notably [] and [..]
 ---> makes compile-time arrays more similar to runtime arrays.
* template quirks fixed. Most notably scope of static if.
 ---> less weird workarounds
* .mangleof property.
 OK, this one's a bit weird. But, it does establish a closer link between the two languages: it means that any type can be expressed in the run-time language.

The remaining changes that are accepted for D 1.0
* implicit function template instantiation
--> makes invoking a template function look the same as invoking a normal function.

and beyond D 1.0
* array literals
 ---> makes compile-time arrays more similar to runtime arrays.

The only other metaprogramming functionality I'd really like, but don't have, is full compile-time reflection, and possibly the ability to convert strings into variables.

I've spent a few months pushing the boundaries of what's achievable with D, and frankly, I'm running out of ideas. Practically everything I've wished I could do in C++ seems to already be possible in D, with the exceptions listed above.

In C++, metaprogramming feels like programming in an extremely impoverished functional programming language. But, in D, it feels like programming in a subset of D -- (you can only assign to variables at construction, you must use recursion for looping) -- a metafunction can be trivially converted to a runtime function (change 'static if' to 'if', remove all of the "const"s from the variable declarations, remove all the "!", and change  "template abc(char [] str)" to abc(char [] str)"). Normally you don't use recursion much in a D runtime program, but it will work.

So, I don't see any signs of D metaprogramming becoming more cumbersome than it is now. I really don't see grounds for pessimism.
BUT...
it's probable we could do better. I feel, though, that we should be aiming for a metalanguage that is more similar to ordinary D, rather than try to completely separate them.

> Thus, it would behoove us to start examining the features and properties of a from-ground-up designed meta level language for D. (Definitely not ever even intended to anything earlier than D2.0!)
> 
> A year ago this was a pipe dream. But now, with the experience we gain from the existing meta language, and (foremost) since I'm not alone anymore in thinking that such an undertaking is actually within reach, we really could start outlining it now!
> 
> ---
> 
> So I suggest that we:
> 
>  - continue developing the current D metalanguage
>  - continue to have both Don's and Phobos' regexps
>  - in this NG start the development of requirements for KBDM
> 
> KBDM being Kick-Butt D Metalanguage. :-)

I'll be interested to hear any further ideas you have.

Craig Black wrote:
> Keeping all this at an academic level, what do you have in mind for the syntax of this meta-language?

Dunno, if it was in the OP, but my intention is to avoid taking up specifics about the grammar, for as long as ever possible!

The reasoning being: most of us (who have to do with computers and/or programming) tend to (unconsciously) adhere to the Nike attitude. That being: "Just do it!"

Ever been a group leader at a software developer? Any time a boss is half-way outlining a problem, five of the Cool Guys run to their keyboards and start banging away.

If there ever was _one_ place where that slogan didn't fit, it's software development.

---

So, our _number_one_ priority is to _not_ even remotely get entangled with syntax, semantics, practices, or theory. First we need to become familiar with the problem domain. Right? !!

Don Clugston wrote:
> Georg Wrede wrote:
> 
>> Back to the issue at hand:
>>
>> What I'm still saying, is that (even though Walter and Don have made completely kick-ass progress here), there's a glass ceiling waiting for us at this rate. By the time we have evolved the D meta business into something that has cleared all major obstacles, it will be a hard to use, cumbersome, and unobvious metalanguage. (( but still a hell of a lot nicer and of course much more powerful than C++ templates ever ))
> 
> 
> I'm not sure about this. What do you see as the major obstacles?
> I had a long list of them once, it's very short now.
> Note that almost all of the improvements which have been made, have made
> metaprogramming *more* similar to ordinary D, not less.
> 
> The main reason D metaprogramming is so much simpler than C++, is that we have "static if" instead of using partial template specialisation.
> 
> I'll just summarise the changes I proposed that Walter has implemented:
> 
> * strings and reals as template value arguments.
>  ---> makes metafunctions able to have a greater variety of parameters,
>       and therefore more like normal D.
> * Improved compile-time constant folding, most notably [] and [..]
>  ---> makes compile-time arrays more similar to runtime arrays.
> * template quirks fixed. Most notably scope of static if.
>  ---> less weird workarounds
> * .mangleof property.
>  OK, this one's a bit weird. But, it does establish a closer link between the two languages: it means that any type can be expressed in the run-time language.
> 
> The remaining changes that are accepted for D 1.0
> * implicit function template instantiation
> --> makes invoking a template function look the same as invoking a normal function.
> 
> and beyond D 1.0
> * array literals
>  ---> makes compile-time arrays more similar to runtime arrays.
> 
> The only other metaprogramming functionality I'd really like, but don't have, is full compile-time reflection, and possibly the ability to convert strings into variables.
> 
> I've spent a few months pushing the boundaries of what's achievable with D, and frankly, I'm running out of ideas. Practically everything I've wished I could do in C++ seems to already be possible in D, with the exceptions listed above.
> 
> In C++, metaprogramming feels like programming in an extremely impoverished functional programming language. But, in D, it feels like programming in a subset of D -- (you can only assign to variables at construction, you must use recursion for looping) -- a metafunction can be trivially converted to a runtime function (change 'static if' to 'if', remove all of the "const"s from the variable declarations, remove all the "!", and change  "template abc(char [] str)" to abc(char [] str)"). Normally you don't use recursion much in a D runtime program, but it will work.
> 
> So, I don't see any signs of D metaprogramming becoming more cumbersome than it is now. I really don't see grounds for pessimism.
> BUT...
> it's probable we could do better. I feel, though, that we should be aiming for a metalanguage that is more similar to ordinary D, rather than try to completely separate them.
> 
>> Thus, it would behoove us to start examining the features and properties of a from-ground-up designed meta level language for D. (Definitely not ever even intended to anything earlier than D2.0!)
>>
>> A year ago this was a pipe dream. But now, with the experience we gain from the existing meta language, and (foremost) since I'm not alone anymore in thinking that such an undertaking is actually within reach, we really could start outlining it now!
>>
>> ---
>>
>> So I suggest that we:
>>
>>  - continue developing the current D metalanguage
>>  - continue to have both Don's and Phobos' regexps
>>  - in this NG start the development of requirements for KBDM
>>
>> KBDM being Kick-Butt D Metalanguage. :-)
> 
> 
> I'll be interested to hear any further ideas you have.

Not least due to a few recent "experiences", I'll refrain from answering your _valid_ questions, for now.

Before that, and mostly because you seemed pretty happy with the existing meta-stuff, I think I'll "do my homework" (i.e. "better know what we have now, before touting for changes").

---

While I'm at it, probably I'll use this NG for "meta newbie" questions. My method of learning (which has thoroughly frustrated, ehhh, a number of people) is to linger at the "dumb questions level" until I really feel that I've got a Crystal Clear picture.

georg

PS, I haven't skipped the "glass ceiling idea" yet.....

Georg Wrede wrote:

<snip>
> So I suggest that we:
> 
>  - continue developing the current D metalanguage
>  - continue to have both Don's and Phobos' regexps
>  - in this NG start the development of requirements for KBDM
> 
> KBDM being Kick-Butt D Metalanguage. :-)

This is a very good idea, as code-which-generates-code is the heart of some powerful techniques(Lisp/Scheme macros among them).

To make them truly work however one of the following is really needed:

1. Runtime evaluation, as in scripting type languages.  In this case all you need is a function which returns a string.

2. A _uniform_ way to represent all the constructs of the language. This works in Lisp because everything is a List(actually a CONS cell), and you can simply walk the tree checking for symbols and generating code as needed.

D is much simpler than C++, I will agree wholeheartedly. However, I fear that such a language strapped on to D would be rife with kludges to accomodate its (relatively)complex syntax.

Cheers.
-DavidM

David Medlock wrote:
> Georg Wrede wrote:
> 
> <snip>
>> So I suggest that we:
>>
>>  - continue developing the current D metalanguage
>>  - continue to have both Don's and Phobos' regexps
>>  - in this NG start the development of requirements for KBDM
>>
>> KBDM being Kick-Butt D Metalanguage. :-)
> 
> This is a very good idea, as code-which-generates-code is the heart of some powerful techniques(Lisp/Scheme macros among them).
> 
> To make them truly work however one of the following is really needed:
> 
> 1. Runtime evaluation, as in scripting type languages.  In this case all you need is a function which returns a string.
> 
> 2. A _uniform_ way to represent all the constructs of the language. This works in Lisp because everything is a List(actually a CONS cell), and you can simply walk the tree checking for symbols and generating code as needed.

I'm sure that compile time reflection is the way to go.
We already have the .mangleof property, which maps any type to a string. Once we get array literals, you could have an array of strings showing all members of a given item.
eg a property
char [][] .membersof
which is valid for any type.

Then with __identifier(char []) you could walk the syntax tree.

Somewhat lacking in syntactic sugar, and doesn't deal with control structures, but it gets you to most of the interesting stuff.

I've already written templates that do:
char [] qualifiednameof!(alias x)
char [] symbolnameof!(alias x)
which work for any variable, function, type or module x,
and as a side-effect, you get loads of type info. I don't think too many additions would be required to make everything possible. But, you'd certainly want some simplified syntax.

> D is much simpler than C++, I will agree wholeheartedly. However, I fear that such a language strapped on to D would be rife with kludges to accomodate its (relatively)complex syntax.
> 
> Cheers.
> -DavidM

> I'm sure that compile time reflection is the way to go.
> We already have the .mangleof property, which maps any type to a string.
> Once we get array literals, you could have an array of strings showing all
> members of a given item.
> eg a property
> char [][] .membersof
> which is valid for any type.

Compile-time reflection would be very nice.  I'm not sure I understand why array literals would be a requirement for this.  Am I missing something?

-Craig

Craig Black wrote:
>> I'm sure that compile time reflection is the way to go.
>> We already have the .mangleof property, which maps any type to a string. Once we get array literals, you could have an array of strings showing all members of a given item.
>> eg a property
>> char [][] .membersof
>> which is valid for any type.
> 
> Compile-time reflection would be very nice.  I'm not sure I understand why array literals would be a requirement for this.  Am I missing something?

It's because we can't have compile-time constant folding of [], except for char arrays, until we get array literals. Compile-time reflection _could_ be done without using arrays, but I think it's the most natural way of doing it, anything else would require new syntax or be a bit clunky?

Don Clugston wrote:
> Craig Black wrote:
>>
>> Compile-time reflection would be very nice.  I'm not sure I understand why array literals would be a requirement for this.  Am I missing something?
> 
> It's because we can't have compile-time constant folding of [], except for char arrays, until we get array literals. Compile-time reflection _could_ be done without using arrays, but I think it's the most natural way of doing it, anything else would require new syntax or be a bit clunky?
>

Why would it requite clunky syntax? I don't see why array literals would affect this. How exactly are you thinking compile-time reflection should work? (in terms of usage)
In fact I don't see how compile-time reflection could work at all (regardless of array literals). I tried to convert my runtime reflection example in the Reflection thread, but some things wouldn't work, one of which being that one cannot use a TypeInfo as a type (for instance in a declaration or something).

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to be... unnatural."

In article <du7kl1$288j$1@digitaldaemon.com>, Bruno Medeiros says...
>
>Don Clugston wrote:
>> Craig Black wrote:
>>>
>>> Compile-time reflection would be very nice.  I'm not sure I understand why array literals would be a requirement for this.  Am I missing something?
>> 
>> It's because we can't have compile-time constant folding of [], except for char arrays, until we get array literals. Compile-time reflection _could_ be done without using arrays, but I think it's the most natural way of doing it, anything else would require new syntax or be a bit clunky?
> >
>
>Why would it requite clunky syntax? I don't see why array literals would
>affect this. How exactly are you thinking compile-time reflection should
>work? (in terms of usage)
>In fact I don't see how compile-time reflection could work at all
>(regardless of array literals). I tried to convert my runtime reflection
>example in the Reflection thread, but some things wouldn't work, one of
>which being that one cannot use a TypeInfo as a type (for instance in a
>declaration or something).
>
>-- 
>Bruno Medeiros - CS/E student
>"Certain aspects of D are a pathway to many abilities some consider to
>be... unnatural."

I dont see why it would require any syntax beyond what D currently uses, just a way to integrate it... a simple metaprotocol...

you could write all of your 'metacode' in D which gets compiled into small shared libraries which are dynamically loaded (or compiled, i suppose) at compile/translation time to process your code, via a protcol of metaobjects representing your code (reflection data on steroids). your metacode, in D, would just manipulate these metaobjects before being sent to the compiler....

Its all a fairly straightforward process. ive currently been playing with this using an experimental metacompiler hooked up with dparser, and it works QUITE WELL. For example, i can take source modules, and translate them back into source replacing all class references with (assert(ref !is null), ref) and instantly you have auto-asserting class references :D the only real haslte is using the #line directive to synch up debugging data between your translated code and your original source. But in the end metacode expands just like macros in c++, only much much cooler :D

Bruno Medeiros wrote:
> Don Clugston wrote:
>> Craig Black wrote:
>>>
>>> Compile-time reflection would be very nice.  I'm not sure I understand why array literals would be a requirement for this.  Am I missing something?
>>
>> It's because we can't have compile-time constant folding of [], except for char arrays, until we get array literals. Compile-time reflection _could_ be done without using arrays, but I think it's the most natural way of doing it, anything else would require new syntax or be a bit clunky?
>  >
> 
> Why would it requite clunky syntax? I don't see why array literals would affect this. How exactly are you thinking compile-time reflection should work? (in terms of usage)

Just because an array lookup can't be done at compile time.

Suppose the compiler gave you:

const char [][] members = ["func1", "func2", "func3"];

and then you want to say
const char [] secondone = members[2];

this won't be implemented until we get array literals.

> In fact I don't see how compile-time reflection could work at all (regardless of array literals). I tried to convert my runtime reflection example in the Reflection thread, but some things wouldn't work, one of which being that one cannot use a TypeInfo as a type (for instance in a declaration or something).

You don't need TypeInfo at compile time. You already have typeof, which is better. (You also have mangleof, which lets you get function parameters, etc, which is something you can't currently do at runtime).

Compile-time reflection is much better than runtime. Think about it -- the only time you need runtime reflection is when dynamically loading an external module. If you're reflecting something that's in your own exe, you could do it at compile time.
If you had compile-time reflection, you could do run-time reflection as a library, at compile time you just collect and save all the info you want.