Georg Wrede schrieb:
> I've been thinkin hard on a meta language.

Word from the wise: Don't try to reinvent the wheel but learn from those to tried before (even if they failed)

There are loads of programming languages or add-ons do existing languages out there that allow meta-programming in some way. I don't have all the references at hand that I have read over the past years. There have been thorough scientific investigations of the matter. I doubt it is possible to outstrip those projects by starting from scratch.

The point of D is not to give it the perfect features for meta-programming but designing features in such a way that they are practical, fit nicely into the general language and don't get in the way with regular programming.

A preprocessor is not such a great idea, since it will generally split the language into two levels causing all kinds of problems. Walter had good reason to drop the concept of a preprocessor.

In general, the big pieces of the meta-language already exist in D: templates. These already contain a complete functional programming language that can be used for meta-programming. Several pieces are missing for full comfort and usability, but these are details that have to be fixed.

The meta-language formed by templates certainly is not very handy when trying to do regular calculations at compile time, but then, it has to be clear that this is not the point of a meta-language. Meta-programming has the very specific purpose of doing code transformations and for this, templates are excellently designed. It is not a general functional language, but a highly specialized language. In principle, you can do anything, but going beyond the special purpose of the meta-language will be both awkward and inefficient.

Now: if you can identify single weak points of the existing meta-language and propose clear solutions, these may have a chance. For doing a complete redesign of templates, the language is a bit too mature, though.

Ciao,
Norbert

Norbert Nemec wrote:
> Georg Wrede schrieb:
> 
>> I've been thinkin hard on a meta language.
> 
> Word from the wise: Don't try to reinvent the wheel but learn from those to tried before (even if they failed)
> 
> There are loads of programming languages or add-ons do existing languages out there that allow meta-programming in some way. I don't have all the references at hand that I have read over the past years. There have been thorough scientific investigations of the matter. I doubt it is possible to outstrip those projects by starting from scratch.

Thanks. I welcome this and other criticisms. Really! The harder, the better. (If anybody can actually convince me that this is either impossible or useless, I'll be thankful. The sooner, the more thankful. Then I could stop wasting time and bandwidth, and maybe find something else where I feel I can contribute to D. :-)  )

If you happen to find some (especially in your own opinion) good references, I'd appreciate. Both of successful things, and those explaining what or why something should not be done, or is downright proved impossible.

> The point of D is not to give it the perfect features for meta-programming but designing features in such a way that they are practical, fit nicely into the general language and don't get in the way with regular programming.

I fully agree with all of that.

If somebody wants to do ultra heavy duty meta programming, they probably would do it in Lisp anyway. (Or something.)

However, what I'm _specifically_ out to get, is a syntax that is not derived from currently existing template language solutions. (D, C++, and the like.)

I can't help feeling that path is a dead end.

While easier for trivial (or at least not too complicated) template transforms, I somehow feel (lack of Ph.D. in this area confines me to a feelings-only basis here, appologies to all), that such an approach turns counter productive already in moderately advanced contexts. (Already the  Recursive Templates example (http://www.digitalmars.com/d/template.html) seems unnecessarily complicated to me. You need to use two separate templates, and an enum. And all for something that I wish could be expressed as simply as the non-template recursive function.

Not to mention, that I (hope) I'm not the only one who has to look hard at what's being done there, even if recursion itself is very familiar to me.

Every language is like a person. It appears one way at the surface, and once you really get to know them, then you (most often) realise that some of the things you thought one could do or couldn't, actually are the other way around. (I wrote about that at length in my Master's Thesis.)

The old amongst us remember Mbasic. (That was before MS-DOS and the PC. Mbasic was by Microsoft, for the CP/M operating system.) My personal _feeling_ was that programming effort grew exponentially with program size. When I found Turbo Pascal, the effort seemed to grow logarithmically! (I know this is not so, but I'm talking about the feeling. Oh, the joy!)

With templates, I again have a feeling. And that is, effort grows too fast. (Unscientific, I admit!) Doing easy stuff is easy, even next-to-easy is next-to-easy. But already moderate is hard.

NOTE: I think meta and the existing template system, are not mutually exclusive! Even if you can do some things in both.

> A preprocessor is not such a great idea, since it will generally split the language into two levels causing all kinds of problems. Walter had good reason to drop the concept of a preprocessor.

True. For the time being, however, when we toy around with what may be done with meta language, it seems like less work to have it as a preprocessor. I fully agree that (if done at all) this should be done respecting Walter's ideas:

 - context free grammar
 - no preprocessor
 - straightforward and practical

> In general, the big pieces of the meta-language already exist in D: templates. These already contain a complete functional programming language that can be used for meta-programming. Several pieces are missing for full comfort and usability, but these are details that have to be fixed.

Yes. And my quest here should in no way slow down that work!

> The meta-language formed by templates certainly is not very handy when trying to do regular calculations at compile time, but then, it has to be clear that this is not the point of a meta-language.

Strictly speaking then, should we call this, er, "quest", something else than meta? I'd have no problem with that. (I assume this thread has given a pretty good idea to everybody of what I'm aiming at.)

> Meta-programming has the very specific purpose of doing code transformations and for this, templates are excellently designed. It is not a general functional language, but a highly specialized language. 

Code transformations, yes. But I want one to be able to do arbitrary code transformations. (Yes, yes, I know, this is reaching for the stars, a wild goose chase. And even if we pull this off, the documentation has to explicitly state that this is not _supposed_ to be used for everything. This _should_ be used sparingly, if at all. And only when templates, modules, OO, etc. fail -- and even then "don't try this at home". But at this point, know that your job _will_ get done with this.)

Of course, already one could download (Windows people, Others have these already) awk, sed, perl, scheme, and do code transformations limited only by their own imagination.

But I want more. To reach this "more", this needs to be inside a language. (Just like Walter already did with the templates!) Only then can we get some serious value-add. (( Not to mention, if this was entirely a preprocessor thing, then the Other Guys could use it on their out-dated language! :-)  ))

It's a matter of principle. This thing has to be without arbitrary limits. Just like a word processor with "unlimited line length, unlimited file size". Everybody knows that this doesn't work:

$ wget -O - -r -l 9999 "ftp://archive.loc.gov/texts" | $EDIT -

no matter how "unlimited" the editor is in theory, or what a monster mainframe you're currently using. Still it matters for many practical purposes that you have "unlimited ll/fs". Same with what I'm pursuing. Nobody wants to have the compiler invent the entire source code of the Google engine. But we don't want arbitrary limits here either.

> In principle, you can do anything, 

I assume the example applications of meta in this thread would not be included in this "anything"?

> but going beyond the special purpose of the meta-language will be both awkward and inefficient.

I wish I can prove different. (I'm not sure at all, but I'm making a hell of a try at it.)

> Now: if you can identify single weak points of the existing meta-language and propose clear solutions, these may have a chance.

I will, if I find any.

> For doing a complete redesign of templates, the language is a bit too mature, though.

Yes, let's not tear down what we already have.

>(Already the  Recursive Templates example (http://www.digitalmars.com/d/template.html) seems unnecessarily complicated to me. You need to use two separate templates, and an enum. And all for something that I wish could be expressed as simply as the non-template recursive function.

I just looked at the example and it is wrong! The factorial of 0 is 1. So it should be:

# template factorial(int n : 0)  // not 1
# {
#     enum { factorial = 1 }
# }

# template factorial(int n)
# {
#     // Note . used to find global template rather than enum
#     enum { factorial = n* .factorial!(n-1) }
# }

Well after doing some meta-programming in C++ this simple example isn't hard to read.

But how to do this in a simpler way?

# metacode metaInt factorial (metaInt n)
# {
#    if (n == 0)
#       return 1;
#    else
#       return n * factorial(n-1);
# }

Might be cool. But that is a totaly different kind of meta-programming than I thought would be discussed in this thread. I thought it would be something like "compiler-plugins" that work on the AST (abstract synthax tree), so we can define our own language constructs. I never thought about any kind of compiletime-calculation.

-- Matthias Becker

Sorry, that I don't go into the details of your mail. The points you bring up contains much that I've considered before as well. The topic meta-programming is extremely deep. The more you think about it, the less you know what way you should go. That's probably why I never really stuck with one concept long enough to really elaborate it.

In any case, what you write sounds somewhat like what is tried with the OpenC++ project:

	http://opencxx.sourceforge.net/

I never managed go beyond scratching the surface of this, and it seems like it is just far too powerful to make it into mainstream development.

There also is an OpenJava project, that takes the ideas and ports them to Java. Obviously, the concept does not depend too much on the details of the language. Maybe OpenD would be just the project that you were thinking of?

I think the best place to start reading about OpenC++ would be the programmers guide by Shigeru Chiba:

	http://www.csg.is.titech.ac.jp/~chiba/pub/SPL96-024.ps.gz

I never found the time to dig into it, but maybe you are interested?

Matthias Becker wrote:
>>(Already the  Recursive Templates example (http://www.digitalmars.com/d/template.html) seems unnecessarily complicated to me. You need to use two separate templates, and an enum. And all for something that I wish could be expressed as simply as the non-template recursive function.
....
> Well after doing some meta-programming in C++ this simple example isn't hard to read.
> 
> But how to do this in a simpler way?
> 
> # metacode metaInt factorial (metaInt n)
> # {
> #    if (n == 0)
> #       return 1;
> #    else
> #       return n * factorial(n-1);
> # }

Yes.

> Might be cool. But that is a totaly different kind of meta-programming than I
> thought would be discussed in this thread. I thought it would be something like
> "compiler-plugins" that work on the AST (abstract synthax tree), so we can
> define our own language constructs. 

Ultimately, I hope, this will be the case.

But I have to admit, if I had created D (that is, if I were Walter :-) ) it would take an army to force me to open up the AST at compile time. I'd fear this leads to intractable error messages, false errors in plainly correct code, people believing DMD is broken, and all just because of bugs in people's own meta code.

If this meta thing advances well, then that might become reality. But I wouldn't bet it'd be before D 3.0. Even with luck, 2.0 seems remote.

What I do, however, think is possible, is that a kind of non-AST-messing version might be incorporated in D 2.0. This would of course be more limited than the ultimate version, but it's ok. We really do need some time to get used to this sort of thing, see what it might be applied to, see if folks use it to do something unexpectedly cool, etc. In doing that, we'll have good time to figure out a smooth and practical syntax for the 3.0 meta thing.

(Of course not forgetting that none of this might ever even become reality! Creativity, perseverance, hard work, but most of all, a good dose of plain good luck is needed.)

> I never thought about any kind of compiletime-calculation.

Compile-time calculation opens a new field for us. It gives possibilities that are hard to reach otherwise.