Currently, the DMD template system is making amazing progress!

Being an old warrior, I still haven't entirely put my original ideas 6' under. What I'm currently brewing, is a proof-of-concept precompiler. My idea is that (or actually was, before these huge leaps Walter and Don have made), with a suitable precompiler, one should be able to test different ideas about "I want this/that in D", and see for himself whether they crumble once put to actual use.

Heh, forcing this thing to everybody who claims to have "this just amazing and _essential_ idea to incorporate in D", would reduce some garbage bandwidth usage from the newsgroups -- without anybody having to shoot 'em down.

There are several different ways one could use this for the advancement of D development:

 - a "greyhound race electric rabbit", to "show what the D template syntax _could_ be capable of"

 - have the folks with Unbelievable ideas test their ideas themselves (before wasting bandwidth here)

 - a general tool to see just _how_ far one can go with a decent meta language

 - specifically, to see if we can develop a _clear_ and _obvious_ syntax to do hopefully _everything_ that's ever been done in BOOST

 - if folks find ways to do things with this that can't be done with the current (or next month's) D template syntax, this might give more rigorous food for thought (both for and against) such ideas

 - a theoretical discussion on _what_ should, could, would, be doable with a meta language -- the tool is here

Having said all that, this being a preprocessor, makes it possible for _any_ language to use _precisely this_ preprocessor to the "up to now, in that language, intractable problems". Unfortunately. Now, the aim is to incorporate any lessons from this (if there ever will be any), into the DMD compiler itself! (Of course, filtered by The Crowd, and whatever Walter consideres appropriate.) That is, the existing DMD compiler front end.  My dream is that such would give D even more power than anything achievable with preprocessor only.

---

The current state is, it's half-way between planning and early alpha. Any ideas are welcome. Even if somebody wants to shoot this down, no hard feelings!

What I'm currently looking for with this post:

 - hard problems or examples, to challenge if such can be done with this approach

 - everyday examples of "ordinary preprocessor things" you've missed from the times you used the C(++) preprocessor

 - implicit template instantiation

 - entire classes, and even whole families of classes, wrapped in a single template

 - templates parameterized on things other than the function parameter types

 - examples of "what can currently be done with C++ and not (yet) with D"

 - what would really be cool, but not even dreamable (until today, be it reasonable or not)

 - corporate internal Best Practices examples

 - any (I mean _any at all_) dreams or ideas, or even off-hand thoughts, on how the D syntax might be bettered (or worsened :-) ), now that we (RSN) have a tool to check those out with.

Such examples might include, ideas from languages currently not thought at all implementable in a strongly typed imperative language, weak references, adding methods to already compiled classes, the recent C# SQL things, a robot language (a bit like Logo), logic programming, parser design incorporated right into the language (a la Spirit), etc.

---

The ultimate test would be to

 - reprogram as much as possible from www.boost.org/libs/libraries.htm

 - getting _all_ of that to work would kill the bystanders

 - getting it to work so that an average programmer would understand some the metaprogramming involved, would kill the Boost Gurus

 - getting publicity for such, would just kill <your suggestion here>

---

I know, I know, this sounds like the writings of the deranged white haired mad scientist. (I'm not gonna argue, heck, even I am not sure of this thing. But it does look promising.)

Whatever we do, we should get things done so fast that the C++ Design-Committee-Bureaucracy won't be able to follow us until we've taken over the world.   :-)

Even if they'll use our precompiler as-is. (Which currently would be no problem, unfortunately!)

---

So, let's hear some crap, please!

Georg Wrede wrote:
> Currently, the DMD template system is making amazing progress!
> 
> Being an old warrior, I still haven't entirely put my original ideas 6' under. What I'm currently brewing, is a proof-of-concept precompiler. My idea is that (or actually was, before these huge leaps Walter and Don have made), with a suitable precompiler, one should be able to test different ideas about "I want this/that in D", and see for himself whether they crumble once put to actual use.
> 
>  - specifically, to see if we can develop a _clear_ and _obvious_ syntax to do hopefully _everything_ that's ever been done in BOOST
> 
>  - if folks find ways to do things with this that can't be done with the current (or next month's) D template syntax, this might give more rigorous food for thought (both for and against) such ideas
> 
>  - a theoretical discussion on _what_ should, could, would, be doable with a meta language -- the tool is here
> 
> Having said all that, this being a preprocessor, makes it possible for _any_ language to use _precisely this_ preprocessor to the "up to now, in that language, intractable problems". Unfortunately. Now, the aim is to incorporate any lessons from this (if there ever will be any), into the DMD compiler itself! (Of course, filtered by The Crowd, and whatever Walter consideres appropriate.) That is, the existing DMD compiler front end.  My dream is that such would give D even more power than anything achievable with preprocessor only.

I wonder how much this is possible without close integration into the language (proper language parsing). I think that the D template system is getting very close to implementing all of the functionality of the C preprocessor. So, it would need to be quite a bit more sophisticated than the C preprocessor.


> What I'm currently looking for with this post:
> 
>  - hard problems or examples, to challenge if such can be done with this approach
> 
>  - everyday examples of "ordinary preprocessor things" you've missed from the times you used the C(++) preprocessor

(1) Token pasting. Suppose there was a keyword '__identifier', with syntax:
__identifier(StringLiteral)
legal anywhere that an identifier is legal.
int __identifier("abc");
would be equivalent to:
int abc;
BUT: the string literal would need to be evaluated before the token pasting occurs, otherwise there's no point.

(2) Stringizing. The inverse of the above.

int abc;
char [] name = __nameof(abc);

is equivalent to:
char [] name = "abc";

If 'abc' is an alias, it should use the ultimate name.(Ie, it should use the name which is printed out in an error message).

>  - implicit template instantiation
>  - examples of "what can currently be done with C++ and not (yet) with D"

Using my hack to put delegates into C++ (www.codeproject.com/cpp/fastdelegate.asp).

you can retrofit polymorphism, along the lines of Chris Diggin's interface idea. See my comment in
http://www.codeproject.com/cpp/retrofitpolymorphism2.asp
My method requires implicit template instantiation.
(Would be *much* nicer in D).
But even better would be to set up proper interfaces, the way he does it with C macros. I think this is something you could do with a preprocessor such as you describe.

>  - entire classes, and even whole families of classes, wrapped in a single template
> 
>  - templates parameterized on things other than the function parameter types


> The ultimate test would be to
> 
>  - reprogram as much as possible from www.boost.org/libs/libraries.htm
> 
>  - getting _all_ of that to work would kill the bystanders
> 
>  - getting it to work so that an average programmer would understand some the metaprogramming involved, would kill the Boost Gurus

The Boost Metaprogramming Library looks like an easy target, already.
It's horribly hacky, and you can't actually do very much with it.

Other than implicit template instantiation, I don't think the C++ template system has many advantages left, compared to D.

Georg Wrede wrote:
> The current state is, it's half-way between planning and early alpha. Any ideas are welcome. Even if somebody wants to shoot this down, no hard feelings!
> 

Why would anyone wish to shoot this idea down? It is IMO a good idea, I tryed to make something like this happen too. But as Don Clugston said in his reply to your post a simple preprocesor is hardly going to be enough. If you wish to do anything more complex with syntax you should have a real parser.

How is your knowledge on language grammars? I have a working grammar-based parser as a project on dsource and a rather imperfect D-grammar (it doesn't really parse the intire D language). With a fixed grammar and a good lexer hooked up it could easily extend todays D.

I have played with it more in the past (too little time now because of the exams), and there are nice things that can be done with it.

Just as an example I implemented, a new type of statement in D that looked like this:

int[10] numbers;

(each[int index] numbers) = index;

and it translated to
foreach(int index, typeof(numbers[index]) __item; numbers)
{
  __item = index;
}

And another example was aplply-to-all-like syntax

numbers.{writefln(it); it += 2; }; //print each number and add 2 to it.


So if you feel this could help you in any way feel free to use it. It is rather hard to start using but it gets easier later.

Anyway, good luck with your project!

On 11/30/2005 02:47 AM, Ivan Senji wrote:
> Georg Wrede wrote:
> 
>> The current state is, it's half-way between planning and early alpha.
[snip]

> How is your knowledge on language grammars? I have a working grammar-based parser as a project on dsource and a rather imperfect D-grammar (it doesn't really parse the intire D language). With a fixed grammar and a good lexer hooked up it could easily extend todays D.
[snip]
> Just as an example I implemented, a new type of statement in D that looked like this:
> 
> int[10] numbers;
> 
> (each[int index] numbers) = index;
> 
> and it translated to
> foreach(int index, typeof(numbers[index]) __item; numbers)
> {
>   __item = index;
> }

What I like to see is:

(each[int index] int const[10] numbers) = index;

IOW, use it to intialize the numbers.  This would
enable the compiler to know what the values are
at each element of numbers; hence, enable better
speed optimization.

For a concrete example, take a look at little.funvec_.zip
in http://tinyurl.com/8bgy5 .  In file:

  little.funvec_constcvec.cpp

there's a vector initialized with function pointers.  The
function pointers have to be entered by the programmer.

OTOH, in file:

  little.funvec_foreach.cpp

the function pointers are calculated in a templated CTOR
which uses a boost::mpl::for_each to fill the vector.

Obviously, funvec_constcvec is faster than funvec_foreach
even though they both do *essentially* the same thing,
fill a constant vector containing functions pointers
(just like a virtual function table ).  Could this
this be done with your parser?

Speed was of prime importance to the original poster
of the thread which prompted the little.funvec_* code:

http://groups.google.com/group/comp.lang.c++.moderated/msg/23c238cdda9cd878

Larry Evans wrote:
> What I like to see is:
> 
> (each[int index] int const[10] numbers) = index;
> 
> IOW, use it to intialize the numbers.  This would
> enable the compiler to know what the values are
> at each element of numbers; hence, enable better
> speed optimization.

Interesting.

> 
> For a concrete example, take a look at little.funvec_.zip
> in http://tinyurl.com/8bgy5 .  In file:
> 
>   little.funvec_constcvec.cpp
> 
> there's a vector initialized with function pointers.  The
> function pointers have to be entered by the programmer.
> 
> OTOH, in file:
> 
>   little.funvec_foreach.cpp
> 
> the function pointers are calculated in a templated CTOR
> which uses a boost::mpl::for_each to fill the vector.
> 
> Obviously, funvec_constcvec is faster than funvec_foreach
> even though they both do *essentially* the same thing,
> fill a constant vector containing functions pointers
> (just like a virtual function table ).  

Didn't really look ad that code because I don't like
the way it is structured (unreadable, sorry) :), but...

> Could this
> this be done with your parser?

The question is "is there a way to write this in D (using
templates)?" If the answer is yes, then it can be done, the only
problem is you have to write code in the grammar definition
that describes that translation to normal D.

But it takes free time, and that I do not have :(

Larry Evans wrote:
> On 11/30/2005 02:47 AM, Ivan Senji wrote:
> 
>> Georg Wrede wrote:
>>
>>> The current state is, it's half-way between planning and early alpha.
> 
> [snip]
> 
>> How is your knowledge on language grammars? I have a working grammar-based parser as a project on dsource and a rather imperfect D-grammar (it doesn't really parse the intire D language). With a fixed grammar and a good lexer hooked up it could easily extend todays D.
> 
> [snip]
> 
>> Just as an example I implemented, a new type of statement in D that looked like this:
>>
>> int[10] numbers;
>>
>> (each[int index] numbers) = index;
>>
>> and it translated to
>> foreach(int index, typeof(numbers[index]) __item; numbers)
>> {
>>   __item = index;
>> }
> 
> 
> What I like to see is:
> 
> (each[int index] int const[10] numbers) = index;
> IOW, use it to intialize the numbers.  This would
> enable the compiler to know what the values are
> at each element of numbers; hence, enable better
> speed optimization.

This sort of thing can already be done with D templates.
-------
template square(int n)
{
  const int square = n*n;
}

const [] numbers = generateArray!(10, square);
// const int [] numbers = [0, 1, 4, 9, 16, 25, ...]

const [] smallfactorials = generateArray!(20, factorial);
// const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
-------
const T [] generateArray!(int limits, const T metafunction!(int) f )
is a template I've made, it can accept metafunctions
returning any constant type. (For example, f could return
generateArray!(5, g) where g returns a string literal).

Note that it's also possible for the array length itself to be calculated at compile time.

BTW, anyone have a better name for generateArray ?
I've also tried calling it "generateLookupTable" and "staticforeach",
neither of which seem appropriate.

Don Clugston wrote:
> 
> This sort of thing can already be done with D templates.
> -------
> template square(int n)
> {
>   const int square = n*n;
> }
> 
> const [] numbers = generateArray!(10, square);
> // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]
> 
> const [] smallfactorials = generateArray!(20, factorial);
> // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
> -------
> const T [] generateArray!(int limits, const T metafunction!(int) f )
> is a template I've made, it can accept metafunctions
> returning any constant type. (For example, f could return
> generateArray!(5, g) where g returns a string literal).
> 
Hum.. I'm somewhat confused. How exactly did you do that? Could you show the code(for generateArray)?

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to be... unnatural."

Bruno Medeiros wrote:
> Don Clugston wrote:
> 
>>
>> This sort of thing can already be done with D templates.
>> -------
>> template square(int n)
>> {
>>   const int square = n*n;
>> }
>>
>> const [] numbers = generateArray!(10, square);
>> // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]
>>
>> const [] smallfactorials = generateArray!(20, factorial);
>> // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
>> -------
>> const T [] generateArray!(int limits, const T metafunction!(int) f )
>> is a template I've made, it can accept metafunctions
>> returning any constant type. (For example, f could return
>> generateArray!(5, g) where g returns a string literal).
>>
> Hum.. I'm somewhat confused. How exactly did you do that? Could you show the code(for generateArray)?

It's not pretty, but it's trivial, really. Currently done with a hack (like a C++ preprocessor).
---------------
private import hack.hackgenerate;

template generateArray(alias generator, int n)
{
     const typeof(generator!(0)) [] generateArray = hackgenerate!(n, generator);
}
----------------
I autogenerate a file with the first few entries looking like this: (hackgenerate.d). Has entries for num=1 to 128.

I have to use this hack, because D doesn't have proper array literals yet, so arrays can't be concatentated or pulled apart at compile time.
As you can imagine, the D code for making the hackgenerate file is very short.

--------------
// Autogenerated file - do not edit

template hackgenerate(int num, alias gen)
{
  pragma(msg, "Table is too large!");
  static assert(0);
}

 template hackgenerate(int num :0, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0) ];
}

 template hackgenerate(int num :1, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0), gen!(1) ];
}

 template hackgenerate(int num :2, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0), gen!(1), gen!(2) ];
}
--------------

On 12/09/2005 01:36 AM, Don Clugston wrote:
[snip]
> 
> It's not pretty, but it's trivial, really. Currently done with a hack (like a C++ preprocessor).
> ---------------
> private import hack.hackgenerate;
> 
> template generateArray(alias generator, int n)
> {
>      const typeof(generator!(0)) [] generateArray = hackgenerate!(n, generator);
> }
> ----------------
> I autogenerate a file with the first few entries looking like this: (hackgenerate.d). Has entries for num=1 to 128.
> 
> I have to use this hack, because D doesn't have proper array literals yet, so arrays can't be concatentated or pulled apart at compile time.
> As you can imagine, the D code for making the hackgenerate file is very short.
> 
> --------------
> // Autogenerated file - do not edit

Don, since I've already done my own hack
(in PP_RECFLD.zip of
http://boost-consulting.com/vault/index.php?&directory=Template%20Metaprogramming
) I was hoping to avoid this.  I'd really like something
that can just be done in the language itself without any preprocessing.

> Don, since I've already done my own hack
> (in PP_RECFLD.zip of
> http://boost-consulting.com/vault/index.php?&directory=Template%20Metaprogramming 
> 
> ) I was hoping to avoid this.  I'd really like something
> that can just be done in the language itself without any preprocessing.

I agree entirely. I've done enough of it too. That was just a temporary workaround.

The main need for the the hack is the absence of array literals, a feature which Walter has said is coming (but not for a while, because it's tricky to implement).

Right now, it is possible to avoid the hack for the special case of strings. But unfortunately, it needs a second hack, because
const char [] ~= const char is not evaluated at compile time.
You need to convert char into char[]. I'm hoping that this limitation will be removed very soon, ie this year, so that we can at least do
one special case perfectly. Once we have that, array literals will become a more fun problem for Walter to work on, so it will probably gain increased priority.