I've been thinkin hard on a meta language.

I've talked with folks, amongst them prof. Jaakko Järvi, who actually met Walter at the C++ meeting last fall.

Bad news: I seem to have spilled the beans, big time. After asking his opinion about some of the core concepts I'd been cooking, it occurred to me that -- this guy's office is down the hall from Bjarne Stroustrup!

If these guys stop dwelling _within_ the C++ template jungle, and start looking at it from above, then it'll be just 18 months before one of them holds a ground breaking lecture.   :-(   :-(


D meta language is an important thing. But now it is also urgent.

Is there a mailing list for it, or should I continue on my own track?

Georg Wrede wrote:

> I've been thinkin hard on a meta language.
[...]
> D meta language is an important thing. But now it is also urgent.

What would the "D meta language" be ? Something like YACC, but for D ?
(maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)

Or something completely different ?

By definition it's "a language that can be used to describe languages"
Just wondering what your little project was, and why it was urgent...

--anders

Anders F Björklund wrote:
> Georg Wrede wrote:
> 
>> I've been thinkin hard on a meta language.
> 
> [...]
> 
>> D meta language is an important thing. But now it is also urgent.
> 
> 
> What would the "D meta language" be ? Something like YACC, but for D ?
> (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)

BTW, where do I get the D grammar itself? (Other than copy/pasting from the docs, piece by piece. ;-)  )

> Or something completely different ?
> 
> By definition it's "a language that can be used to describe languages"
> Just wondering what your little project was, and why it was urgent...

Well, the C++ preprocessor has grown into a language.

It's even Turing complete, so nobody can dispute it being a language on its own. The template system now is really a meta language. One could say that the STL is a language (of sorts), too, written with the syntax of the preprocessor/template grammar.

What I've been thinking of, is something like a preprocessor that is aware of the language. (At least, for the time being it would be a (more or less) preprocessor.) The goal is, however, to have something that is a genuine part of the compiler, and that is intimately "aware" of the "underlying" language. And could conduct two-way communication with "the rest of" the compiler.

"Preprocessor" here being merely a temporary technical solution.

The way I see the state of language development today, is that real productivity, real power of expression, real versatility, and adequate abstraction -- all should be possible using a single medium. Now, there have been numeros ambitious enterprises in this arena. Quite a few passing fads have been hyped as "the ultimate tool for _everything_".

Lisp, for example, is capable of expressing anything at all, with relative ease too (at least for the lisp gurus). But we don't see Lisp running everything. With C you can do "everything" too. But nobody uses it for abstract or high level stuff.

A single language cannot fit everything any more. The fields of application and levels of abstraction are way too disparate for a single syntax to manage, if we want it to still be usable for as much as possible. (And if we don't, the competition will!)

The time has come for languages that are genuine compounds of two separate levels of abstraction. Having two separate syntaxes "for one language" is no big deal today. C++ does it all the time. D has it (so far) in a smaller scale, if we think of D templates combined with mixins. The rules of this are different from "plain" D. Many machine code assemblers have had a two-level syntax for ages. The one I've played with, was Borland's TASM. Now, with TASM you could even do OO! The syntax and even support was there. And this was like 15 years ago.

Having two syntaxes lets each of these be tuned and optimised for its own half of the cake. Also, having two syntaxes helps keep a kind of separation within the applications written. This actually resembles what many programmers do right now: do the nitty gritty in C, make it a DLL, then do the UI or program logic in, say Python, or even Euphoria.

The D philosophy fits nicely here. D has always tried to be a language where you don't need a preprocessor. Somehow though, we seem to have a long way to go. For example, here's a snippet from the DMD distribution (written in C++, but anyway):

Expression *Parser::parseAssignExp()
{   Expression *e;
    Expression *e2;
    Loc loc;

    e = parseCondExp();
    while (1)
    {
        loc = this->loc;
        switch (token.value)
        {
#define X(tok,ector) case tok:  nextToken(); \
  e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;

            X(TOKassign,    AssignExp);
            X(TOKaddass,    AddAssignExp);
            X(TOKminass,    MinAssignExp);
            X(TOKmulass,    MulAssignExp);
            X(TOKdivass,    DivAssignExp);
            X(TOKmodass,    ModAssignExp);
            X(TOKandass,    AndAssignExp);
            X(TOKorass,     OrAssignExp);
            X(TOKxorass,    XorAssignExp);
            X(TOKshlass,    ShlAssignExp);
            X(TOKshrass,    ShrAssignExp);
            X(TOKushrass,   UshrAssignExp);
            X(TOKcatass,    CatAssignExp);

#undef X
            default:                 break;
        }
        break;
    }     return e;
}

Doing this (what Walter's done here with the preprocessor) is just one example of what I want to be easy in D, too, without a separate preprocessor.

Another example would be a mathematician. He defines a notation to use in source code, and then is able to use it. Of course, one might criticize that for illegibility to the normal programmer. But more important is that it is legible to the professionals in their own field.

Say someone really needs to have (the infamous?) unquoted regular expression syntax, which was discussed here a couple of weeks ago. He could write a definition, and then just use any regular expressions within D code.

And I can't even begin to imagine what people like prof. Järvi could conjure up, given a tool like this.

So, whether easy or hairy stuff is needed, this would give more expressive power to D on so many new levels.

Our "metalanguage" should be aware of its surroundings. __LINE__ etc. are only the beginning. It should be able to get any information a D program can get -- about files, the machine, whatever. And all at compile time.

It should make the (revered, adminred, feared, sworn at) C++ template system look like an enormously hard to use toy.

Done right, it would be easy to grasp, easy to use, and therefore less error prone.

Georg Wrede schrieb:
> I've been thinkin hard on a meta language.
> 
> I've talked with folks, amongst them prof. Jaakko Järvi, who actually met Walter at the C++ meeting last fall.
> 
> Bad news: I seem to have spilled the beans, big time. After asking his opinion about some of the core concepts I'd been cooking, it occurred to me that -- this guy's office is down the hall from Bjarne Stroustrup!
> 
> If these guys stop dwelling _within_ the C++ template jungle, and start looking at it from above, then it'll be just 18 months before one of them holds a ground breaking lecture.   :-(   :-(

Don't worry. The basic idea has been around for some time already. Within C++, it will be hard to go far beyond the current state without seriously breaking compatibility. It can only be new languages that can take the next step.

Norbert Nemec wrote:
> Georg Wrede schrieb:
> 
>> I've been thinkin hard on a meta language.
>>
>> I've talked with folks, amongst them prof. Jaakko Järvi, who actually met Walter at the C++ meeting last fall.
>>
>> Bad news: I seem to have spilled the beans, big time. After asking his opinion about some of the core concepts I'd been cooking, it occurred to me that -- this guy's office is down the hall from Bjarne Stroustrup!
>>
>> If these guys stop dwelling _within_ the C++ template jungle, and start looking at it from above, then it'll be just 18 months before one of them holds a ground breaking lecture.   :-(   :-(
> 
> 
> Don't worry. The basic idea has been around for some time already. Within C++, it will be hard to go far beyond the current state without seriously breaking compatibility. It can only be new languages that can take the next step.

Of course I worry. Guys of this caliber do things "right". And once they've figured out a replacement to their current pp/tl system, the new one will take care of the old pp/tl syntax just with a snap of fingers.

The goal, after all, once they start thinking "outside the box" is to create a universal meta language. For such a language it is a piece of cake to handle all of the current crap.

So, they'll get their Backward Compatibility, but this time as a _mere_ special case -- just one small thing among the tousands of grand things it'll handle.

One bad (for us, that is) thing that follows from this, is that once they've done this, then, given time, folks will rewrite the current STL, BOOST, etc. with the new syntax (possibly even skipping the bla<foo<bar>> notation), and this will inevitably result in a vast speed-up of compilation!!!!

So, I'm worried as hell. For D. But for Human Kind this is of course a Good Thing(TM).

-----------

If we have a smarter language to begin with (like D), then right now there is a window of opportunity for fame and reputation for D. But the window won't stay open for good. After that our meta language will be "just another thing somebody does `like what's done in C++`". Farewell, publicity, fame, hype, exponential sales. Back to our niche.

My real goof was in waking up this guy, from "dwelling withing the template jungle" to looking at it "from above", "outside the box". Men of this kind don't need more than that.

Anders F Björklund wrote:
> Georg Wrede wrote:
> 
>> I've been thinkin hard on a meta language.
> 
> [...]
> 
>> D meta language is an important thing. But now it is also urgent.
> 
> 
> What would the "D meta language" be ? Something like YACC, but for D ?
> (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)
> 
> Or something completely different ?
> 
> By definition it's "a language that can be used to describe languages"
> Just wondering what your little project was, and why it was urgent...
> 
> --anders

Copy/pasting from another thread here. But read my other answer to your post first.

############################################### begin include

Why not push to get the enum-names exposed via reflection instead? I wouldn't
wish to have to re-type them all over again, in a different manner. Doesn't make
sense.

- Kris



In article <d0onfb$1afp$1@digitaldaemon.com>, Andrew Fedoniouk says...

>>
>>Thanks a lot,
>>
>>What about this:
>>
>>enum E {
>>  ZERO,
>>  ONE,
>>  TWO
>>}
>>
>>char[] toString(E e)
>>{
>>  static char[][] map =
>>  [
>>    E.ZERO  :"primordial",
>>    E.ONE   :"first",
>>    E.TWO   :"second"
>>  ];
>>  return map[e];
>>}
>>
>>for simple continuous enums?

################################################ end include

Things like the above would be handled with the meta language real neatly:

enum E {
some_meta_language_def_here;
ZERO   primordial
ONE    first
TWO    second
undefine_the_meta_def_here;
}

With a good meta language, even this would be _at_least_ as easy as with the C preprocessor.

The "some_meta_language_def_here" has to be easy to write, easy to understand, even if the language is capable of Awesome Deep Stuff too.

You want printable enums? "Do this."
You want reversed enums? "Do that."

You want a recursive compile time generated static tree structure to hold the Universal Pointers with a Riemann-Lembowski hyperbolic naturalised reverse transform in 9 dimensions? "Well, do this, and then type the definition in from your textbook."

You want enums where more than one enum-word has the same integer value? "Write it as below:"

enum E {
some_meta_language_def_here;
FOO BAR BARF
NOO NAR NARF
NIG
NAG
NUG
BOOBOO AUX_BOOBOO NOTHER_BOOBOO
undefine_the_meta_def_here;
}

And you'll get enums where FOO == BARF, NIG != NAG.

BTW, a lot of what Matthew, others (and even I) have suggested lately could be implemented with this meta language.

Carrying the thought a little further, once this language exists, then it might be worthwhile to refactor things between D and meta. For example, the current templates, mixins, and possibly other stuff, could be moved from D itself to "defined in terms of the metalanguage".

This may make the compiler core more simple and faster, for all I know.

New D features could _all_ be created in the metalanguage, at first. If it later turns out that they fit better in D itself, then so be it. The user should not have to need to know.

But what would it look like? I really like the idea of a meta-language, I just can't come up with something that fits in D and at the same time makes it extensible..

As I see it, two issues are involved:
- being able to specify new syntax, both locally (as the X example below) and globally, where the main problem is - what is the syntax for syntax, and how do you make it self-extensible; if it's not self-extensible, it's just another layer of templates, I guess..

- being able to modify existing code, because just defining syntax expansions (like X below) is not enough. For example, if you wanted to add aspects, you'd need to define the syntax, have the compiler parse the source, then do something like "for each aspect that is defined, go through all the method and class declarations, find those that match, and insert the code from the aspect there and there if that and that". Now that's basically a program in itself, so it's a good question whether it is not better to simply open the compiler to plugins, than to try to fit this in the language and have the compiler be practically a virtual machine..

Even after you're able to do that, there are issues of collisions (what if two meta-things want to modify the same, say, class?), infinite loops (code A produces B, code B produces Aa, which becomes Ba, which becomes Aaa, ...), optimization, etc.

Especially the issue of optimization seems a big problem to me... I mean, "A x B" where x is some nice unicode symbol is not that much better than mul(A,B) that people would swallow like 2x slower running time over it, in the case the compiler won't be able to optimize it. And AFAIK, the whole optimization thing is more or less quite specific cases of what gets changed to what (I may be wrong on this, I'm no expert).

OTOH, perhaps it's not that big a deal - there could still be major gains in productivity, which can often be far more important than speed..

For a final thought, I guess a real victory would be if the meta-language was so powerful, that the entire D we have now could be expressed in it (except perhaps some really really basic constructs -- you do need something to build with in the first place).


xs0


Georg Wrede wrote:
> Anders F Björklund wrote:
> 
>> Georg Wrede wrote:
>>
>>> I've been thinkin hard on a meta language.
>>
>>
>> [...]
>>
>>> D meta language is an important thing. But now it is also urgent.
>>
>>
>>
>> What would the "D meta language" be ? Something like YACC, but for D ?
>> (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)
> 
> 
> BTW, where do I get the D grammar itself? (Other than copy/pasting from the docs, piece by piece. ;-)  )
> 
>> Or something completely different ?
>>
>> By definition it's "a language that can be used to describe languages"
>> Just wondering what your little project was, and why it was urgent...
> 
> 
> Well, the C++ preprocessor has grown into a language.
> 
> It's even Turing complete, so nobody can dispute it being a language on its own. The template system now is really a meta language. One could say that the STL is a language (of sorts), too, written with the syntax of the preprocessor/template grammar.
> 
> What I've been thinking of, is something like a preprocessor that is aware of the language. (At least, for the time being it would be a (more or less) preprocessor.) The goal is, however, to have something that is a genuine part of the compiler, and that is intimately "aware" of the "underlying" language. And could conduct two-way communication with "the rest of" the compiler.
> 
> "Preprocessor" here being merely a temporary technical solution.
> 
> The way I see the state of language development today, is that real productivity, real power of expression, real versatility, and adequate abstraction -- all should be possible using a single medium. Now, there have been numeros ambitious enterprises in this arena. Quite a few passing fads have been hyped as "the ultimate tool for _everything_".
> 
> Lisp, for example, is capable of expressing anything at all, with relative ease too (at least for the lisp gurus). But we don't see Lisp running everything. With C you can do "everything" too. But nobody uses it for abstract or high level stuff.
> 
> A single language cannot fit everything any more. The fields of application and levels of abstraction are way too disparate for a single syntax to manage, if we want it to still be usable for as much as possible. (And if we don't, the competition will!)
> 
> The time has come for languages that are genuine compounds of two separate levels of abstraction. Having two separate syntaxes "for one language" is no big deal today. C++ does it all the time. D has it (so far) in a smaller scale, if we think of D templates combined with mixins. The rules of this are different from "plain" D. Many machine code assemblers have had a two-level syntax for ages. The one I've played with, was Borland's TASM. Now, with TASM you could even do OO! The syntax and even support was there. And this was like 15 years ago.
> 
> Having two syntaxes lets each of these be tuned and optimised for its own half of the cake. Also, having two syntaxes helps keep a kind of separation within the applications written. This actually resembles what many programmers do right now: do the nitty gritty in C, make it a DLL, then do the UI or program logic in, say Python, or even Euphoria.
> 
> The D philosophy fits nicely here. D has always tried to be a language where you don't need a preprocessor. Somehow though, we seem to have a long way to go. For example, here's a snippet from the DMD distribution (written in C++, but anyway):
> 
> Expression *Parser::parseAssignExp()
> {   Expression *e;
>     Expression *e2;
>     Loc loc;
> 
>     e = parseCondExp();
>     while (1)
>     {
>         loc = this->loc;
>         switch (token.value)
>         {
> #define X(tok,ector) case tok:  nextToken(); \
>   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;
> 
>             X(TOKassign,    AssignExp);
>             X(TOKaddass,    AddAssignExp);
>             X(TOKminass,    MinAssignExp);
>             X(TOKmulass,    MulAssignExp);
>             X(TOKdivass,    DivAssignExp);
>             X(TOKmodass,    ModAssignExp);
>             X(TOKandass,    AndAssignExp);
>             X(TOKorass,     OrAssignExp);
>             X(TOKxorass,    XorAssignExp);
>             X(TOKshlass,    ShlAssignExp);
>             X(TOKshrass,    ShrAssignExp);
>             X(TOKushrass,   UshrAssignExp);
>             X(TOKcatass,    CatAssignExp);
> 
> #undef X
>             default:                 break;
>         }
>         break;
>     }     return e;
> }
> 
> Doing this (what Walter's done here with the preprocessor) is just one example of what I want to be easy in D, too, without a separate preprocessor.
> 
> Another example would be a mathematician. He defines a notation to use in source code, and then is able to use it. Of course, one might criticize that for illegibility to the normal programmer. But more important is that it is legible to the professionals in their own field.
> 
> Say someone really needs to have (the infamous?) unquoted regular expression syntax, which was discussed here a couple of weeks ago. He could write a definition, and then just use any regular expressions within D code.
> 
> And I can't even begin to imagine what people like prof. Järvi could conjure up, given a tool like this.
> 
> So, whether easy or hairy stuff is needed, this would give more expressive power to D on so many new levels.
> 
> Our "metalanguage" should be aware of its surroundings. __LINE__ etc. are only the beginning. It should be able to get any information a D program can get -- about files, the machine, whatever. And all at compile time.
> 
> It should make the (revered, adminred, feared, sworn at) C++ template system look like an enormously hard to use toy.
> 
> Done right, it would be easy to grasp, easy to use, and therefore less error prone.

Georg Wrede wrote:
> My real goof was in waking up this guy, from "dwelling withing the template jungle" to looking at it "from above", "outside the box". Men of this kind don't need more than that.

Don't worry, you haven't pointed out anything he didn't know already. Lisp has been doing this for years now.  The majority of the syntax of the Nemerle language <http://nemerle.org> is implemented as macros. (even stuff like 'if' statements)

Also, Daveed Vandevoorde had a quasi-working implementation of such an extension for C++ as far back as 2003: <http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2003/n1471.pdf>

 -- andy

xs0 wrote:
> But what would it look like? I really like the idea of a meta-language, I just can't come up with something that fits in D and at the same time makes it extensible..

Well, at this point I think the best thing to do is to carefully think about what we _want_ it to be able to do. IMHO, if we try to come up with, say a syntax, or a way of "blending into D", then we unnecessarily create restraints. Both syntaxwise, but more important, we fence us into our newly made box, and that's the end of "out of the box" thinking. Right now is the time for the latter, and the longer we keep at it, the better we understand _what_ has to be done. Once that is clear, only then can we allow ourselves to think about _how_.

> As I see it, two issues are involved:
> - being able to specify new syntax, both locally (as the X example below) and globally, where the main problem is - what is the syntax for syntax, and how do you make it self-extensible; if it's not self-extensible, it's just another layer of templates, I guess..

We shouldn't care! Just assume whatever we need to, we'll come up with in due time. First we need a good list of what we _want_. Suggestions and help, even (possibly hard) criticism is welcome too!

The C crowd _started_ with a syntax. Then kept appending to it, and we all see where that leads. They've never had a clue of where they are ultimately aiming. Well, we'gre gonna do it the other way around.

> - being able to modify existing code, because just defining syntax expansions (like X below) is not enough. 

Right!

> For example, if you wanted to add aspects, you'd need to define the syntax, have the compiler parse the source, then do something like "for each aspect that is defined, go through all the method and class declarations, find those that match, and insert the code from the aspect there and there if that and that". 

The meta language idea should be able to accommodate that. But not like we specifically thought of aspects in particular, rather like if it is what we wanted it to be, and only after that somebody invents aspects, the meta language should handle that too. Or, more to the point, the programmer who wants to implement aspects. And what he has to write in meta to make it so should not be incomprehensible to others.

> Now that's basically a program in itself, so it's a good question 

Ecactly. And that is what I've meant each time I've mentioned Turing! I see no way around it. -- The C++ preprocessor language had reached Turing level without any of the C++ honchos ever even noticing. Several _years_ later somebody just happened to discover the fact.

My view is that cpp being TC is not a coincidence. Rather, it is a requisite. And therefore _we_ take that as a starting point.

> whether it is not better to simply open the compiler to plugins, than to try to fit this in the language and have the compiler be practically a virtual machine..

I would like to know more of this plugin idea.

> Even after you're able to do that, there are issues of collisions (what if two meta-things want to modify the same, say, class?), infinite loops (code A produces B, code B produces Aa, which becomes Ba, which becomes Aaa, ...), optimization, etc.

We'll take care of them at the time. Had Walter got scared of circular imports too early -- but he didn't. Same with meta.

> Especially the issue of optimization seems a big problem to me... I mean, "A x B" where x is some nice unicode symbol is not that much better than mul(A,B) that people would swallow like 2x slower running time over it, in the case the compiler won't be able to optimize it. And AFAIK, the whole optimization thing is more or less quite specific cases of what gets changed to what (I may be wrong on this, I'm no expert).
> 
> OTOH, perhaps it's not that big a deal - there could still be major gains in productivity, which can often be far more important than speed..

Optimization I hope happens after the meta phase. Of course, that does not give us the right to make meta inefficient. :-)

> For a final thought, I guess a real victory would be if the meta-language was so powerful, that the entire D we have now could be expressed in it (except perhaps some really really basic constructs -- you do need something to build with in the first place).

Hmm. I wouldn't bet against it. OTOH, I'm not ready to have it as a goal either. Not at least right now.

> Georg Wrede wrote:
> 
>> Anders F Björklund wrote:
>>
>>> Georg Wrede wrote:
>>>
>>>> I've been thinkin hard on a meta language.
>>>
>>>
>>>
>>> [...]
>>>
>>>> D meta language is an important thing. But now it is also urgent.
>>>
>>>
>>>
>>>
>>> What would the "D meta language" be ? Something like YACC, but for D ?
>>> (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)
>>
>>
>>
>> BTW, where do I get the D grammar itself? (Other than copy/pasting from the docs, piece by piece. ;-)  )
>>
>>> Or something completely different ?
>>>
>>> By definition it's "a language that can be used to describe languages"
>>> Just wondering what your little project was, and why it was urgent...
>>
>>
>>
>> Well, the C++ preprocessor has grown into a language.
>>
>> It's even Turing complete, so nobody can dispute it being a language on its own. The template system now is really a meta language. One could say that the STL is a language (of sorts), too, written with the syntax of the preprocessor/template grammar.
>>
>> What I've been thinking of, is something like a preprocessor that is aware of the language. (At least, for the time being it would be a (more or less) preprocessor.) The goal is, however, to have something that is a genuine part of the compiler, and that is intimately "aware" of the "underlying" language. And could conduct two-way communication with "the rest of" the compiler.
>>
>> "Preprocessor" here being merely a temporary technical solution.
>>
>> The way I see the state of language development today, is that real productivity, real power of expression, real versatility, and adequate abstraction -- all should be possible using a single medium. Now, there have been numeros ambitious enterprises in this arena. Quite a few passing fads have been hyped as "the ultimate tool for _everything_".
>>
>> Lisp, for example, is capable of expressing anything at all, with relative ease too (at least for the lisp gurus). But we don't see Lisp running everything. With C you can do "everything" too. But nobody uses it for abstract or high level stuff.
>>
>> A single language cannot fit everything any more. The fields of application and levels of abstraction are way too disparate for a single syntax to manage, if we want it to still be usable for as much as possible. (And if we don't, the competition will!)
>>
>> The time has come for languages that are genuine compounds of two separate levels of abstraction. Having two separate syntaxes "for one language" is no big deal today. C++ does it all the time. D has it (so far) in a smaller scale, if we think of D templates combined with mixins. The rules of this are different from "plain" D. Many machine code assemblers have had a two-level syntax for ages. The one I've played with, was Borland's TASM. Now, with TASM you could even do OO! The syntax and even support was there. And this was like 15 years ago.
>>
>> Having two syntaxes lets each of these be tuned and optimised for its own half of the cake. Also, having two syntaxes helps keep a kind of separation within the applications written. This actually resembles what many programmers do right now: do the nitty gritty in C, make it a DLL, then do the UI or program logic in, say Python, or even Euphoria.
>>
>> The D philosophy fits nicely here. D has always tried to be a language where you don't need a preprocessor. Somehow though, we seem to have a long way to go. For example, here's a snippet from the DMD distribution (written in C++, but anyway):
>>
>> Expression *Parser::parseAssignExp()
>> {   Expression *e;
>>     Expression *e2;
>>     Loc loc;
>>
>>     e = parseCondExp();
>>     while (1)
>>     {
>>         loc = this->loc;
>>         switch (token.value)
>>         {
>> #define X(tok,ector) case tok:  nextToken(); \
>>   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;
>>
>>             X(TOKassign,    AssignExp);
>>             X(TOKaddass,    AddAssignExp);
>>             X(TOKminass,    MinAssignExp);
>>             X(TOKmulass,    MulAssignExp);
>>             X(TOKdivass,    DivAssignExp);
>>             X(TOKmodass,    ModAssignExp);
>>             X(TOKandass,    AndAssignExp);
>>             X(TOKorass,     OrAssignExp);
>>             X(TOKxorass,    XorAssignExp);
>>             X(TOKshlass,    ShlAssignExp);
>>             X(TOKshrass,    ShrAssignExp);
>>             X(TOKushrass,   UshrAssignExp);
>>             X(TOKcatass,    CatAssignExp);
>>
>> #undef X
>>             default:                 break;
>>         }
>>         break;
>>     }     return e;
>> }
>>
>> Doing this (what Walter's done here with the preprocessor) is just one example of what I want to be easy in D, too, without a separate preprocessor.
>>
>> Another example would be a mathematician. He defines a notation to use in source code, and then is able to use it. Of course, one might criticize that for illegibility to the normal programmer. But more important is that it is legible to the professionals in their own field.
>>
>> Say someone really needs to have (the infamous?) unquoted regular expression syntax, which was discussed here a couple of weeks ago. He could write a definition, and then just use any regular expressions within D code.
>>
>> And I can't even begin to imagine what people like prof. Järvi could conjure up, given a tool like this.
>>
>> So, whether easy or hairy stuff is needed, this would give more expressive power to D on so many new levels.
>>
>> Our "metalanguage" should be aware of its surroundings. __LINE__ etc. are only the beginning. It should be able to get any information a D program can get -- about files, the machine, whatever. And all at compile time.
>>
>> It should make the (revered, adminred, feared, sworn at) C++ template system look like an enormously hard to use toy.
>>
>> Done right, it would be easy to grasp, easy to use, and therefore less error prone.

I'm not sure where you want to go with this meta-language but I'm here to burst your bubble.  (-:  I always wanted a meta-langauge... until I saw one. There is one for C++ and it is indecipherable.  It seems the movement in langagues is towards cleaning up and pairing down, not making them more "tricky."

Lots of people still claim C++ is too hard precisely because you don't know what it will do under the covers for a simple assignment statement.  And keep in mind C++ and even Java and D have some basic "meta programming" with operator= (opAssign) type options.

If you really, really want to add whole new calculus to the language, well that is what I thought I wanted until I saw an example.  Most people just get disturbed by creating it and even just being forced to use the results.

Finally, I've been on a tear for the last few years that people over-use languages anyway.  I think I got the idea from a Myer's C++ book (and also that Andrescu template book).  I was following along with all the insanely detailed examples of using private inheritance with a customized new operator, etc, etc, when they started adjusting the C++ vtable in order to create some lisp-like capabilities (non-portable obviously).  That was the last straw for me.  If you have to *beat* the language, then you need to simplify your app... or suffer the ugly consequences of maintaing it.  Or writing a new language!  But at some level of power, people just drop off.

My two cents.

"Georg Wrede" <georg.wrede@nospam.org> wrote in message news:422F7E8C.7040509@nospam.org...
> Anders F Björklund wrote:
>> Georg Wrede wrote:
>>
>>> I've been thinkin hard on a meta language.
>>
>> [...]
>>
>>> D meta language is an important thing. But now it is also urgent.
>>
>>
>> What would the "D meta language" be ? Something like YACC, but for D ? (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)
>
> BTW, where do I get the D grammar itself? (Other than copy/pasting from the docs, piece by piece. ;-)  )
>
>> Or something completely different ?
>>
>> By definition it's "a language that can be used to describe languages" Just wondering what your little project was, and why it was urgent...
>
> Well, the C++ preprocessor has grown into a language.
>
> It's even Turing complete, so nobody can dispute it being a language on its own. The template system now is really a meta language. One could say that the STL is a language (of sorts), too, written with the syntax of the preprocessor/template grammar.
>
> What I've been thinking of, is something like a preprocessor that is aware of the language. (At least, for the time being it would be a (more or less) preprocessor.) The goal is, however, to have something that is a genuine part of the compiler, and that is intimately "aware" of the "underlying" language. And could conduct two-way communication with "the rest of" the compiler.
>
> "Preprocessor" here being merely a temporary technical solution.
>
> The way I see the state of language development today, is that real productivity, real power of expression, real versatility, and adequate abstraction -- all should be possible using a single medium. Now, there have been numeros ambitious enterprises in this arena. Quite a few passing fads have been hyped as "the ultimate tool for _everything_".
>
> Lisp, for example, is capable of expressing anything at all, with relative ease too (at least for the lisp gurus). But we don't see Lisp running everything. With C you can do "everything" too. But nobody uses it for abstract or high level stuff.
>
> A single language cannot fit everything any more. The fields of application and levels of abstraction are way too disparate for a single syntax to manage, if we want it to still be usable for as much as possible. (And if we don't, the competition will!)
>
> The time has come for languages that are genuine compounds of two separate levels of abstraction. Having two separate syntaxes "for one language" is no big deal today. C++ does it all the time. D has it (so far) in a smaller scale, if we think of D templates combined with mixins. The rules of this are different from "plain" D. Many machine code assemblers have had a two-level syntax for ages. The one I've played with, was Borland's TASM. Now, with TASM you could even do OO! The syntax and even support was there. And this was like 15 years ago.
>
> Having two syntaxes lets each of these be tuned and optimised for its own half of the cake. Also, having two syntaxes helps keep a kind of separation within the applications written. This actually resembles what many programmers do right now: do the nitty gritty in C, make it a DLL, then do the UI or program logic in, say Python, or even Euphoria.
>
> The D philosophy fits nicely here. D has always tried to be a language where you don't need a preprocessor. Somehow though, we seem to have a long way to go. For example, here's a snippet from the DMD distribution (written in C++, but anyway):
>
> Expression *Parser::parseAssignExp()
> {   Expression *e;
>     Expression *e2;
>     Loc loc;
>
>     e = parseCondExp();
>     while (1)
>     {
>         loc = this->loc;
>         switch (token.value)
>         {
> #define X(tok,ector) case tok:  nextToken(); \
>   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;
>
>             X(TOKassign,    AssignExp);
>             X(TOKaddass,    AddAssignExp);
>             X(TOKminass,    MinAssignExp);
>             X(TOKmulass,    MulAssignExp);
>             X(TOKdivass,    DivAssignExp);
>             X(TOKmodass,    ModAssignExp);
>             X(TOKandass,    AndAssignExp);
>             X(TOKorass,     OrAssignExp);
>             X(TOKxorass,    XorAssignExp);
>             X(TOKshlass,    ShlAssignExp);
>             X(TOKshrass,    ShrAssignExp);
>             X(TOKushrass,   UshrAssignExp);
>             X(TOKcatass,    CatAssignExp);
>
> #undef X
>             default:                 break;
>         }
>         break;
>     }     return e;
> }
>
> Doing this (what Walter's done here with the preprocessor) is just one example of what I want to be easy in D, too, without a separate preprocessor.
>
> Another example would be a mathematician. He defines a notation to use in source code, and then is able to use it. Of course, one might criticize that for illegibility to the normal programmer. But more important is that it is legible to the professionals in their own field.
>
> Say someone really needs to have (the infamous?) unquoted regular expression syntax, which was discussed here a couple of weeks ago. He could write a definition, and then just use any regular expressions within D code.
>
> And I can't even begin to imagine what people like prof. Järvi could conjure up, given a tool like this.
>
> So, whether easy or hairy stuff is needed, this would give more expressive power to D on so many new levels.
>
> Our "metalanguage" should be aware of its surroundings. __LINE__ etc. are only the beginning. It should be able to get any information a D program can get -- about files, the machine, whatever. And all at compile time.
>
> It should make the (revered, adminred, feared, sworn at) C++ template system look like an enormously hard to use toy.
>
> Done right, it would be easy to grasp, easy to use, and therefore less error prone.