View mode: basic / threaded / horizontal-split · Log in · Help
November 30, 2005
A 'nother way to do this
Currently, the DMD template system is making amazing progress!

Being an old warrior, I still haven't entirely put my original ideas 6' 
under. What I'm currently brewing, is a proof-of-concept precompiler. My 
idea is that (or actually was, before these huge leaps Walter and Don 
have made), with a suitable precompiler, one should be able to test 
different ideas about "I want this/that in D", and see for himself 
whether they crumble once put to actual use.

Heh, forcing this thing to everybody who claims to have "this just 
amazing and _essential_ idea to incorporate in D", would reduce some 
garbage bandwidth usage from the newsgroups -- without anybody having to 
shoot 'em down.

There are several different ways one could use this for the advancement 
of D development:

 - a "greyhound race electric rabbit", to "show what the D template 
syntax _could_ be capable of"

 - have the folks with Unbelievable ideas test their ideas themselves 
(before wasting bandwidth here)

 - a general tool to see just _how_ far one can go with a decent meta 
language

 - specifically, to see if we can develop a _clear_ and _obvious_ 
syntax to do hopefully _everything_ that's ever been done in BOOST

 - if folks find ways to do things with this that can't be done with 
the current (or next month's) D template syntax, this might give more 
rigorous food for thought (both for and against) such ideas

 - a theoretical discussion on _what_ should, could, would, be doable 
with a meta language -- the tool is here

Having said all that, this being a preprocessor, makes it possible for 
_any_ language to use _precisely this_ preprocessor to the "up to now, 
in that language, intractable problems". Unfortunately. Now, the aim is 
to incorporate any lessons from this (if there ever will be any), into 
the DMD compiler itself! (Of course, filtered by The Crowd, and whatever 
Walter consideres appropriate.) That is, the existing DMD compiler front 
end.  My dream is that such would give D even more power than anything 
achievable with preprocessor only.

---

The current state is, it's half-way between planning and early alpha. 
Any ideas are welcome. Even if somebody wants to shoot this down, no 
hard feelings!

What I'm currently looking for with this post:

 - hard problems or examples, to challenge if such can be done with 
this approach

 - everyday examples of "ordinary preprocessor things" you've missed 
from the times you used the C(++) preprocessor

 - implicit template instantiation

 - entire classes, and even whole families of classes, wrapped in a 
single template

 - templates parameterized on things other than the function parameter 
types

 - examples of "what can currently be done with C++ and not (yet) with D"

 - what would really be cool, but not even dreamable (until today, be 
it reasonable or not)

 - corporate internal Best Practices examples

 - any (I mean _any at all_) dreams or ideas, or even off-hand 
thoughts, on how the D syntax might be bettered (or worsened :-) ), now 
that we (RSN) have a tool to check those out with.

Such examples might include, ideas from languages currently not thought 
at all implementable in a strongly typed imperative language, weak 
references, adding methods to already compiled classes, the recent C# 
SQL things, a robot language (a bit like Logo), logic programming, 
parser design incorporated right into the language (a la Spirit), etc.

---

The ultimate test would be to

 - reprogram as much as possible from www.boost.org/libs/libraries.htm

 - getting _all_ of that to work would kill the bystanders

 - getting it to work so that an average programmer would understand 
some the metaprogramming involved, would kill the Boost Gurus

 - getting publicity for such, would just kill <your suggestion here>

---

I know, I know, this sounds like the writings of the deranged white 
haired mad scientist. (I'm not gonna argue, heck, even I am not sure of 
this thing. But it does look promising.)

Whatever we do, we should get things done so fast that the C++ 
Design-Committee-Bureaucracy won't be able to follow us until we've 
taken over the world.   :-)

Even if they'll use our precompiler as-is. (Which currently would be no 
problem, unfortunately!)

---

So, let's hear some crap, please!
November 30, 2005
Re: A 'nother way to do this
Georg Wrede wrote:
> Currently, the DMD template system is making amazing progress!
> 
> Being an old warrior, I still haven't entirely put my original ideas 6' 
> under. What I'm currently brewing, is a proof-of-concept precompiler. My 
> idea is that (or actually was, before these huge leaps Walter and Don 
> have made), with a suitable precompiler, one should be able to test 
> different ideas about "I want this/that in D", and see for himself 
> whether they crumble once put to actual use.
> 
>  - specifically, to see if we can develop a _clear_ and _obvious_ syntax 
> to do hopefully _everything_ that's ever been done in BOOST
> 
>  - if folks find ways to do things with this that can't be done with the 
> current (or next month's) D template syntax, this might give more 
> rigorous food for thought (both for and against) such ideas
> 
>  - a theoretical discussion on _what_ should, could, would, be doable 
> with a meta language -- the tool is here
> 
> Having said all that, this being a preprocessor, makes it possible for 
> _any_ language to use _precisely this_ preprocessor to the "up to now, 
> in that language, intractable problems". Unfortunately. Now, the aim is 
> to incorporate any lessons from this (if there ever will be any), into 
> the DMD compiler itself! (Of course, filtered by The Crowd, and whatever 
> Walter consideres appropriate.) That is, the existing DMD compiler front 
> end.  My dream is that such would give D even more power than anything 
> achievable with preprocessor only.

I wonder how much this is possible without close integration into the 
language (proper language parsing). I think that the D template system 
is getting very close to implementing all of the functionality of the C 
preprocessor. So, it would need to be quite a bit more sophisticated 
than the C preprocessor.


> What I'm currently looking for with this post:
> 
>  - hard problems or examples, to challenge if such can be done with this 
> approach
> 
>  - everyday examples of "ordinary preprocessor things" you've missed 
> from the times you used the C(++) preprocessor

(1) Token pasting. Suppose there was a keyword '__identifier', with syntax:
__identifier(StringLiteral)
legal anywhere that an identifier is legal.
int __identifier("abc");
would be equivalent to:
int abc;
BUT: the string literal would need to be evaluated before the token 
pasting occurs, otherwise there's no point.

(2) Stringizing. The inverse of the above.

int abc;
char [] name = __nameof(abc);

is equivalent to:
char [] name = "abc";

If 'abc' is an alias, it should use the ultimate name.(Ie, it should use 
the name which is printed out in an error message).

>  - implicit template instantiation
>  - examples of "what can currently be done with C++ and not (yet) with D"

Using my hack to put delegates into C++ 
(www.codeproject.com/cpp/fastdelegate.asp).

you can retrofit polymorphism, along the lines of Chris Diggin's 
interface idea. See my comment in
http://www.codeproject.com/cpp/retrofitpolymorphism2.asp
My method requires implicit template instantiation.
(Would be *much* nicer in D).
But even better would be to set up proper interfaces, the way he does it 
with C macros. I think this is something you could do with a 
preprocessor such as you describe.

>  - entire classes, and even whole families of classes, wrapped in a 
> single template
> 
>  - templates parameterized on things other than the function parameter 
> types


> The ultimate test would be to
> 
>  - reprogram as much as possible from www.boost.org/libs/libraries.htm
> 
>  - getting _all_ of that to work would kill the bystanders
> 
>  - getting it to work so that an average programmer would understand some the metaprogramming involved, would kill the Boost Gurus

The Boost Metaprogramming Library looks like an easy target, already.
It's horribly hacky, and you can't actually do very much with it.

Other than implicit template instantiation, I don't think the C++ 
template system has many advantages left, compared to D.
November 30, 2005
Re: A 'nother way to do this
Georg Wrede wrote:
> The current state is, it's half-way between planning and early alpha. 
> Any ideas are welcome. Even if somebody wants to shoot this down, no 
> hard feelings!
> 

Why would anyone wish to shoot this idea down? It is IMO a good idea, I 
tryed to make something like this happen too. But as Don Clugston said 
in his reply to your post a simple preprocesor is hardly going to be 
enough. If you wish to do anything more complex with syntax you should 
have a real parser.

How is your knowledge on language grammars? I have a working 
grammar-based parser as a project on dsource and a rather imperfect 
D-grammar (it doesn't really parse the intire D language). With a fixed 
grammar and a good lexer hooked up it could easily extend todays D.

I have played with it more in the past (too little time now because of 
the exams), and there are nice things that can be done with it.

Just as an example I implemented, a new type of statement in D that 
looked like this:

int[10] numbers;

(each[int index] numbers) = index;

and it translated to
foreach(int index, typeof(numbers[index]) __item; numbers)
{
  __item = index;
}

And another example was aplply-to-all-like syntax

numbers.{writefln(it); it += 2; }; //print each number and add 2 to it.


So if you feel this could help you in any way feel free to use it. It is 
rather hard to start using but it gets easier later.

Anyway, good luck with your project!
December 02, 2005
Re: A 'nother way to do this
On 11/30/2005 02:47 AM, Ivan Senji wrote:
> Georg Wrede wrote:
> 
>> The current state is, it's half-way between planning and early alpha.
[snip]

> How is your knowledge on language grammars? I have a working 
> grammar-based parser as a project on dsource and a rather imperfect 
> D-grammar (it doesn't really parse the intire D language). With a fixed 
> grammar and a good lexer hooked up it could easily extend todays D.
[snip]
> Just as an example I implemented, a new type of statement in D that 
> looked like this:
> 
> int[10] numbers;
> 
> (each[int index] numbers) = index;
> 
> and it translated to
> foreach(int index, typeof(numbers[index]) __item; numbers)
> {
>   __item = index;
> }

What I like to see is:

(each[int index] int const[10] numbers) = index;

IOW, use it to intialize the numbers.  This would
enable the compiler to know what the values are
at each element of numbers; hence, enable better
speed optimization.

For a concrete example, take a look at little.funvec_.zip
in http://tinyurl.com/8bgy5 .  In file:

  little.funvec_constcvec.cpp

there's a vector initialized with function pointers.  The
function pointers have to be entered by the programmer.

OTOH, in file:

  little.funvec_foreach.cpp

the function pointers are calculated in a templated CTOR
which uses a boost::mpl::for_each to fill the vector.

Obviously, funvec_constcvec is faster than funvec_foreach
even though they both do *essentially* the same thing,
fill a constant vector containing functions pointers
(just like a virtual function table ).  Could this
this be done with your parser?

Speed was of prime importance to the original poster
of the thread which prompted the little.funvec_* code:

http://groups.google.com/group/comp.lang.c++.moderated/msg/23c238cdda9cd878
December 03, 2005
Re: A 'nother way to do this
Larry Evans wrote:
> What I like to see is:
> 
> (each[int index] int const[10] numbers) = index;
> 
> IOW, use it to intialize the numbers.  This would
> enable the compiler to know what the values are
> at each element of numbers; hence, enable better
> speed optimization.

Interesting.

> 
> For a concrete example, take a look at little.funvec_.zip
> in http://tinyurl.com/8bgy5 .  In file:
> 
>   little.funvec_constcvec.cpp
> 
> there's a vector initialized with function pointers.  The
> function pointers have to be entered by the programmer.
> 
> OTOH, in file:
> 
>   little.funvec_foreach.cpp
> 
> the function pointers are calculated in a templated CTOR
> which uses a boost::mpl::for_each to fill the vector.
> 
> Obviously, funvec_constcvec is faster than funvec_foreach
> even though they both do *essentially* the same thing,
> fill a constant vector containing functions pointers
> (just like a virtual function table ).  

Didn't really look ad that code because I don't like
the way it is structured (unreadable, sorry) :), but...

> Could this
> this be done with your parser?

The question is "is there a way to write this in D (using
templates)?" If the answer is yes, then it can be done, the only
problem is you have to write code in the grammar definition
that describes that translation to normal D.

But it takes free time, and that I do not have :(
December 05, 2005
Re: A 'nother way to do this
Larry Evans wrote:
> On 11/30/2005 02:47 AM, Ivan Senji wrote:
> 
>> Georg Wrede wrote:
>>
>>> The current state is, it's half-way between planning and early alpha.
> 
> [snip]
> 
>> How is your knowledge on language grammars? I have a working 
>> grammar-based parser as a project on dsource and a rather imperfect 
>> D-grammar (it doesn't really parse the intire D language). With a 
>> fixed grammar and a good lexer hooked up it could easily extend todays D.
> 
> [snip]
> 
>> Just as an example I implemented, a new type of statement in D that 
>> looked like this:
>>
>> int[10] numbers;
>>
>> (each[int index] numbers) = index;
>>
>> and it translated to
>> foreach(int index, typeof(numbers[index]) __item; numbers)
>> {
>>   __item = index;
>> }
> 
> 
> What I like to see is:
> 
> (each[int index] int const[10] numbers) = index;
> IOW, use it to intialize the numbers.  This would
> enable the compiler to know what the values are
> at each element of numbers; hence, enable better
> speed optimization.

This sort of thing can already be done with D templates.
-------
template square(int n)
{
  const int square = n*n;
}

const [] numbers = generateArray!(10, square);
// const int [] numbers = [0, 1, 4, 9, 16, 25, ...]

const [] smallfactorials = generateArray!(20, factorial);
// const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
-------
const T [] generateArray!(int limits, const T metafunction!(int) f )
is a template I've made, it can accept metafunctions
returning any constant type. (For example, f could return
generateArray!(5, g) where g returns a string literal).

Note that it's also possible for the array length itself to be 
calculated at compile time.

BTW, anyone have a better name for generateArray ?
I've also tried calling it "generateLookupTable" and "staticforeach",
neither of which seem appropriate.
December 08, 2005
Re: A 'nother way to do this
Don Clugston wrote:
> 
> This sort of thing can already be done with D templates.
> -------
> template square(int n)
> {
>   const int square = n*n;
> }
> 
> const [] numbers = generateArray!(10, square);
> // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]
> 
> const [] smallfactorials = generateArray!(20, factorial);
> // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
> -------
> const T [] generateArray!(int limits, const T metafunction!(int) f )
> is a template I've made, it can accept metafunctions
> returning any constant type. (For example, f could return
> generateArray!(5, g) where g returns a string literal).
> 
Hum.. I'm somewhat confused. How exactly did you do that? Could you show 
the code(for generateArray)?

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to 
be... unnatural."
December 09, 2005
Re: A 'nother way to do this
Bruno Medeiros wrote:
> Don Clugston wrote:
> 
>>
>> This sort of thing can already be done with D templates.
>> -------
>> template square(int n)
>> {
>>   const int square = n*n;
>> }
>>
>> const [] numbers = generateArray!(10, square);
>> // const int [] numbers = [0, 1, 4, 9, 16, 25, ...]
>>
>> const [] smallfactorials = generateArray!(20, factorial);
>> // const ulong [] smallfactorials = [0, 1, 2, 6, 20, ... ]
>> -------
>> const T [] generateArray!(int limits, const T metafunction!(int) f )
>> is a template I've made, it can accept metafunctions
>> returning any constant type. (For example, f could return
>> generateArray!(5, g) where g returns a string literal).
>>
> Hum.. I'm somewhat confused. How exactly did you do that? Could you show 
> the code(for generateArray)?

It's not pretty, but it's trivial, really. Currently done with a hack 
(like a C++ preprocessor).
---------------
private import hack.hackgenerate;

template generateArray(alias generator, int n)
{
     const typeof(generator!(0)) [] generateArray = hackgenerate!(n, 
generator);
}
----------------
I autogenerate a file with the first few entries looking like this: 
(hackgenerate.d). Has entries for num=1 to 128.

I have to use this hack, because D doesn't have proper array literals 
yet, so arrays can't be concatentated or pulled apart at compile time.
As you can imagine, the D code for making the hackgenerate file is very 
short.

--------------
// Autogenerated file - do not edit

template hackgenerate(int num, alias gen)
{
  pragma(msg, "Table is too large!");
  static assert(0);
}

 template hackgenerate(int num :0, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0) ];
}

 template hackgenerate(int num :1, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0), gen!(1) ];
}

 template hackgenerate(int num :2, alias gen)
 {
 const typeof(gen!(0)) [] hackgenerate = [
    gen!(0), gen!(1), gen!(2) ];
}
--------------
December 09, 2005
Re: A 'nother way to do this
On 12/09/2005 01:36 AM, Don Clugston wrote:
[snip]
> 
> It's not pretty, but it's trivial, really. Currently done with a hack 
> (like a C++ preprocessor).
> ---------------
> private import hack.hackgenerate;
> 
> template generateArray(alias generator, int n)
> {
>      const typeof(generator!(0)) [] generateArray = hackgenerate!(n, 
> generator);
> }
> ----------------
> I autogenerate a file with the first few entries looking like this: 
> (hackgenerate.d). Has entries for num=1 to 128.
> 
> I have to use this hack, because D doesn't have proper array literals 
> yet, so arrays can't be concatentated or pulled apart at compile time.
> As you can imagine, the D code for making the hackgenerate file is very 
> short.
> 
> --------------
> // Autogenerated file - do not edit

Don, since I've already done my own hack
(in PP_RECFLD.zip of
http://boost-consulting.com/vault/index.php?&directory=Template%20Metaprogramming
) I was hoping to avoid this.  I'd really like something
that can just be done in the language itself without any preprocessing.
December 12, 2005
Re: A 'nother way to do this
> Don, since I've already done my own hack
> (in PP_RECFLD.zip of
> http://boost-consulting.com/vault/index.php?&directory=Template%20Metaprogramming 
> 
> ) I was hoping to avoid this.  I'd really like something
> that can just be done in the language itself without any preprocessing.

I agree entirely. I've done enough of it too. That was just a temporary 
workaround.

The main need for the the hack is the absence of array literals, a 
feature which Walter has said is coming (but not for a while, because 
it's tricky to implement).

Right now, it is possible to avoid the hack for the special case of 
strings. But unfortunately, it needs a second hack, because
const char [] ~= const char is not evaluated at compile time.
You need to convert char into char[]. I'm hoping that this limitation 
will be removed very soon, ie this year, so that we can at least do
one special case perfectly. Once we have that, array literals will 
become a more fun problem for Walter to work on, so it will probably 
gain increased priority.
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home