In article <d5cmb4$oem$1@digitaldaemon.com>, Lionello Lunesu says...
>
>Maybe, after compiling a program, the compiler can execute any function call that has constant (compile-time evaluable) arguments and replace the call with the return value.
>
>It would simply execute the function after compiling, passing the constant arguments. This will result in another constant that will perhaps result in another function being evaluated compile-time, etc..

The most interesting aspects of meta-programming are not computing the values of constants.  If that was all there was to it, it would be pointless.  Just compute the value of the constant once at start up time, and quit yer whining type of thing.  Look again at the simple "integer big enough to hold N bits" example I mentioned.  The compiler can't "simply execute the function after compiling" because that function is computing the type of some variables in code that then need to be compiled.  So at the very least the process has to be "execute the function *before* compiling".  But even that's a little simplistic. In the example above it really needs to be executed anytime integer<N> is encountered in the code _during_ compiling.

>I think the nicest thing about meta-programming will be the possibility to introduce new operators. If we can add types, functions in code, why not also operators? You want "**" to mean "to the power of"? Just define it. "<=>" should call opCmp? Want to be able to do "&" on floats? Or maybe "^" should be "to the power of" if done on reals?

I'm not sure this is really feasible or desirable (or that it is even really meta-programming).  What on earth does "&" on floats mean, for example?  And how do you deal with precedence?  You want ** to mean power of, that's fine, but the compiler has to know somehow what the precedence of that operator is or all your math is going to come out funny.  And you better be sure that you don't define any operators that are ambiguous.  Did you mean to exponentiate "a**b" or to multiply "a * (*b)"?

>I can hardly wait!

Or maybe this is faux-enthusiasm?  Are you being sarcastic?  Sorry if I didn't
catch it.  Most oo folks strongly advise against going crazy with operator
overloading.  It may seem nifty when you're doing it, but it can make the code
totally impossible to read.  Is that "*" really a "*" or is it some funky
overloaded operator?  If so then where the heck is it defined?  And where else
is this overloaded operator being used in this code?  Oh snap! I can't grep for
it because it looks exactly like every other "*" in my code!  Etc.
"Thinking in C++" puts it well:
http://www.camtp.uni-mb.si/books/Thinking-in-C++/TIC2Vone-distribution/html/Chapter12.html#Heading349

--bb

In article <d5cmb4$oem$1@digitaldaemon.com>, Lionello Lunesu says...
>
>Maybe, after compiling a program, the compiler can execute any function call that has constant (compile-time evaluable) arguments and replace the call with the return value.
>
>It would simply execute the function after compiling, passing the constant arguments. This will result in another constant that will perhaps result in another function being evaluated compile-time, etc..

The most interesting aspects of meta-programming are not computing the values of constants.  If that was all there was to it, it would be pointless.  Just compute the value of the constant once at start up time, and quit yer whining type of thing.  Look again at the simple "integer big enough to hold N bits" example I mentioned.  The compiler can't "simply execute the function after compiling" because that function is computing the type of some variables in code that then need to be compiled.  So at the very least the process has to be "execute the function *before* compiling".  But even that's a little simplistic. In the example above it really needs to be executed anytime integer<N> is encountered in the code _during_ compiling.

>I think the nicest thing about meta-programming will be the possibility to introduce new operators. If we can add types, functions in code, why not also operators? You want "**" to mean "to the power of"? Just define it. "<=>" should call opCmp? Want to be able to do "&" on floats? Or maybe "^" should be "to the power of" if done on reals?

I'm not sure this is really feasible or desirable (or that it is even really meta-programming).  What on earth does "&" on floats mean, for example?  And how do you deal with precedence?  You want ** to mean power of, that's fine, but the compiler has to know somehow what the precedence of that operator is or all your math is going to come out funny.  And you better be sure that you don't define any operators that are ambiguous.  Did you mean to exponentiate "a**b" or to multiply "a * (*b)"?

>I can hardly wait!

Or maybe this is faux-enthusiasm?  Are you being sarcastic?  Sorry if I didn't
catch it.  Most oo folks strongly advise against going crazy with operator
overloading.  It may seem nifty when you're doing it, but it can make the code
totally impossible to read.  Is that "*" really a "*" or is it some funky
overloaded operator?  If so then where the heck is it defined?  And where else
is this overloaded operator being used in this code?  Oh snap! I can't grep for
it because it looks exactly like every other "*" in my code!  Etc.
"Thinking in C++" puts it well:
http://www.camtp.uni-mb.si/books/Thinking-in-C++/TIC2Vone-distribution/html/Chapter12.html#Heading349

--bb

Andrew Fedoniouk wrote:

> Hi, Bill,
> 
> (not so) Wild idea  : to put your sources under httpd and config it for use lets say PHP processing on D files.
This is how standard C/C++ preprocessor works (BTW, you can easily use it with D too, as well as with assembler and any other language). PHP has a lot more features, but the idea is the same. http://www.digitalmars.com/d/pretod.html explains why Walter do not like it.

-- 
          Vladimir

In article <d5d55l$140k$1@digitaldaemon.com>, Bill Baxter says...

>Ok.  Yeh, I'm not completely aware of what all is possible or in the works with D.  If D can support adding members to classes and things like that at run time

(I'm just speculating wildly here too; please don't take any of it too
seriously.)

*Adding* members at runtime - probably not feasible, unless the compiler is part of the standard runtime library. Setting or calling named members at runtime, OTOH, is perfectly possible in principle - I do it quite a bit in C#. I think it's universal in languages supporting reflection.

>What kind of attributes can you define in NET?

See http://tinyurl.com/ddjap (links to an MSDN doc page) for the system ones; users can and do define their own. Note that there's substantial crossover between attributes and "marker" interfaces a la Java; dotnet itself is rather schizophrenic on the subject. (It has both a Serializable attribute and an ISerializable interface, and they do different things.) Attributes tend to be used more for metadata, particularly interop- and tool-oriented metadata, whereas interfaces are used more for the mainline semantics. Attributes can also be parameterized, e.g.

[MyAttribute("foo", 92)] someDeclaration;

whereas interfaces obviously can't (though you might be able to do something template-y in D); attributes can also be applied to just about anything, whereas marker interfaces can only be applied to classes.

- Mike

Bill Baxter wrote:
> This comment is about template metaprogramming and metaprogramming in general.
> It's a little more pie-in-the-sky than my last comment.

I might be in the minority here, but I have a strong dislike for both meta-programming and code generation.

From time to time, I'll encounter some smug programmer bragging about the fact that he's not just "writing code" anymore. Instead, now he's using some sort of code generator, and now he's "writing code that writes code".

A code shivver goes through my spine. Like someone just walked over my grave.

Invariably, the code-generation code is far more difficult to read, understand, and debug (for anyone except the original author) than the non-code-generated code would have been.

Metaprogramming (in my opinion) is the same sort of animal. Programmers like metaprogramming because it's tricky and fun. It's the same reason why lisp programmers like to write self-modifying code. It's the same reason why perl programmers have one-liner competitions (ever heard of perl golf?)

Progammers enjoy making their code perform funky tricks.

But in the long run, I think the benefits of code-generation and meta-programming (experienced during code authorship) are cancelled out by the problems they cause later in the software lifecycle (during debugging and maintenance).

--BenjiSmith

Yes, I agree that metaprogramming type things can definitely be abused.  But a couple of responses:

First like any technique the costs must be weighed against the benefits.  Just like operator overloading, you don't just go blindly applying it anywhere and everywhere just because you can.  You apply it only if it will make the resulting code significantly easier to read and maintain.

Second, what I'm advocating is that by supporting metaprogramming more directly in the language, such code will become less obfuscated.  The majority of template metaprogramming techniques only *look* complicated because the syntax for doing it sucks so bad.  (See the example I posted earlier -- more than a page of code for a simple if-then-else).  Most metaprogramming code boils down to simple comparisons, if-then-else, and simple loops.  The horrible syntax just obscures that fact.  Of course, that's what the "smug programmers" who are abusing it like about it, so they can say to their buddies "I bet you can't figure out what my code does -- see, I'm smarter than you".  But if the syntax could be made to look like something readable, these people would hopefully be bored away, and move on to entering perl obfuscation contests or something.

Finally I agree that anything like this must be used judiciously, and if not it will cause you major headaches down the line with support, maintenance, etc.

I'm no super expert on metaprogramming, mainly because I got disgusted with what a pain it was to use and maintain in C++ after experimenting with it for a while.  But I don't think that's an intrinsic fault of the technique, I think it's an implementation issue, namely that C++ templates suck for that sort of thing.

To be honest I haven't thought about this that deeply, and maybe I'm wrong and the usage case really can't be made.  But that's why I posted the message here -- to see if a bunch of smart folks interested in improving upon C++ could help flesh out the idea more.

I think to move the discussion forward it would be helpful for me (or anyone in the know) to throw out a few more real-world examples of places where people use metaprogramming today.  I really haven't touched it for a few years (ever since I ran away from the C++ implementation of screaming :-), so I've forgotten some of the techniques that I thought seemed really cool and useful back then.  With some more concrete examples "of look what metaprogramming can do!" then the D experts can come back with how you'd approach it in D.  I'll see if I can dig some more up.  But for now the best real world examples I can think of are what scripting interface generators are doing.  Like Boost.Python (http://www.boost.org/libs/python/doc/), or luaBind (http://luabind.sourceforge.net).  So how would people go about doing that type of thing in D?

Fundamental issue: just thinking this out now, but I think one fundamental source of the obfuscation in most template metaprogramming techniques is that templates support conditionals in a round-about way via the pattern matching engine for template specialization.  This is both a benefit and a curse. Benefit in that it makes matching rules very flexible and is easy to extend by ploping down new specializations whenever and wherever you like, but a curse in that it basically becomes a giant 'if-then-else' that happens to have all its cases scattered all over the place out of order.  The whole pattern-matching aspect is a lot like some functional programming languages, e.g. ML (I'm not the first to notice that-- another similarity is that recursion is the only way to implement loops in template metaprogramms).

So to try and bite off a manageable morsel, here's a simple question: is it possible and feasible to provide an alternative, more procedural, if-else construct for the purpose of deciding on template specializations?  I.e. is it possible to provide a specialization syntax that instead of this:

template TFoo(T)        { ... } // #1
template TFoo(T : T[])  { ... } // #2
template TFoo(T : char) { ... } // #3
template TFoo(T,U,V)    { ... } // #4

Looks like this:

template TFoo(T) {
if      (T: T[])   { ... } #2
else if (T : char) { ... } // #3
else if (T,U,V)    { ... } // #4
else { ... } // #1
}

The main difference/advantage would be that this gives the developer control over matching order, so there would never be compiler errors from ambiguity.  I envision there are probably other things you could do in those if's too like test if a type has a particular method, if it's an object or a POD type (plain old data), specifying ranges of value parameters if(T: 0<T<=8), etc.  Similarly, it would be cool if you could use the same matching expressions for localizing the specialization to just the code that really needs to be specialized.  Some times the only differences between three versions of a templated function comes down to one or two lines, so then you either have to factor that code out into a different little mini-template or just go and duplicate the whole function 3 times and just change the one line that's different.

For example, say if the type parameter is an object with lock/unlock methods then I want to call those, but otherwise I'll just go ahead with my business. Eg something like:

template Frobulator(T)
{
void frobulate(out T to, T from)
{
if (hasmethod(T,lock)) { from.lock(); }
{ here's where the frobulation happens, about 50 lines of code }
if (hasmethod(T,unlock)) { from.unlock(); }
}
}

This should be something that could be done at compile time when instantiating the template.  Is that possible in D?  Or is it more of what might be allowed with these introspection features that are supposedly in the works?  A related example would be if I'm deriving from the template paramter and I want the derived class to implement lock/unlock methods if they don't already exist in the base class being derived from.  (Looks like that answer to adding a method is no... http://www.digitalmars.com/d/template.html. "limitations" part says you can't add nonstatic methods with templates).


Dang, everytime I think about this stuff it makes me wonder if I shoulda gone into systems programming rather than graphics.  :-)  As it is, I just don't have the time or training really to work effectively on this stuff (one and a half compiler courses is about all I got).  So my ulterior motive here is to hopefully light a fire under someone's butt so they'll go off and create it someday and become my hero.  I don't really expect it to happen overnight, but if I just keep bringing this up in different places where smart people hang out, surely someone will pick it up and run with it eventually... :-)

--bb
P.S. Kudos if you managed to read all the way down to this sentence, oh brave
reader.  ;-)

In article <d5dlc7$1jat$1@digitaldaemon.com>, Benji Smith says...
>
>Bill Baxter wrote:
>> This comment is about template metaprogramming and metaprogramming in general. It's a little more pie-in-the-sky than my last comment.
>
>I might be in the minority here, but I have a strong dislike for both meta-programming and code generation.
>
> From time to time, I'll encounter some smug programmer bragging about
>the fact that he's not just "writing code" anymore. Instead, now he's using some sort of code generator, and now he's "writing code that writes code".
>
>A code shivver goes through my spine. Like someone just walked over my grave.
>
>Invariably, the code-generation code is far more difficult to read, understand, and debug (for anyone except the original author) than the non-code-generated code would have been.
>
>Metaprogramming (in my opinion) is the same sort of animal. Programmers like metaprogramming because it's tricky and fun. It's the same reason why lisp programmers like to write self-modifying code. It's the same reason why perl programmers have one-liner competitions (ever heard of perl golf?)
>
>Progammers enjoy making their code perform funky tricks.
>
>But in the long run, I think the benefits of code-generation and meta-programming (experienced during code authorship) are cancelled out by the problems they cause later in the software lifecycle (during debugging and maintenance).
>
>--BenjiSmith

On Fri, 6 May 2005 02:46:53 +0000 (UTC), Bill Baxter wrote:

[snip]
>  So my ulterior motive here is to
> hopefully light a fire under someone's butt so they'll go off and create it
> someday and become my hero.  I don't really expect it to happen overnight, but
> if I just keep bringing this up in different places where smart people hang out,
> surely someone will pick it up and run with it eventually... :-)

I'm in the process of porting my macro processor to D and integrating it
with Build. That will go someway to achieving your goal.

-- 
Derek Parnell
Melbourne, Australia
http://www.dsource.org/projects/build/ v2.06 released 04/May/2005
http://www.prowiki.org/wiki4d/wiki.cgi?FrontPage
6/05/2005 1:32:44 PM

In article <d5cf7e$hh1$1@digitaldaemon.com>, Bill Baxter says...
>
>Right on, Kevin.  One thing to keep in mind, though, when you talk about giving these "compileTime{}" blocks "all the power of D", is that D is designed to be an efficient, statically-typed, *compiled* language, and to achieve that it has to sacrifice some of the niceties of more dynamic typing to achieve the desired performance.  I personally think a compile-time language should be as dynamic as possible.  It effectivly is going to be an interpreted rather than compiled thing anyway, so why not make it as dynamic as possible?  Like you said, the dream is basically to write metacode that can use introspection on classes at compile time to algorithmically generate types, classes, and the code that eventually gets compiled.  If I were going to write an external code generator, I'd definitely use a scripting language, so why not give the same degree of flexibility to the built-in meta-language?  And give it good text manipulation operators too, search, replace, munge, regex etc.  Just like you'd expect in a scripting language.

I think for the most part this is true --- some scripting like features could be included.  But, before deciding on this, I think it would be good to examine what we mean by the distinction between scripting and compiled languages.  The result is a list of tradeoffs and features.

We could divide the list three ways.  Some of those features could also be available in compiled D (even as a library); some only in meta-programming; some not included at all.  A lot of features like foreach() were common in scripting languages but now appear in D and Java 1.5.  They were a good idea, they just *look* inefficient. ;)

>One issue, however, is that that doesn't really mesh well with the concept you have of "falling back" to run-time computation if the compiler determines it can't do it at compile time.  Though, I don't know if that's such a great idea, really.  I don't know why for sure, but my gut tells me that compile-time execution and run-time execution aren't quite as similar as inline and non-inline are to each other and that's going to cause problems.  For one thing, whereas it's always possible to just not inline something, it won't always be possible to not run some code at compile-time, say if the return value of the code is in fact a type rather than a number.  If you're working with types as data, then you absolutely can't run the code at runtime.

Currently, the "varargs" concept *can* do some type processing at runtime. Typeinfos, such as is used by writef(), for example.

>From a aethetics point of view, though, I do agree that it would be nice if the meta-language were as similar to the D language as possible.  It would definitely be nice if it were easy to convert "compile-time" code to run-time code where the conversion made sense.  But still I don't want to give up the flexibility of a more dynamically-typed language.

I feel the same way, but see below, where I marked [Impossible].

>I think something like this would also really once and for all make macros obsolete.  I've heard various gurus saying that the combination of inline functions, typedefs, templates, and const variables makes macros obsolete, but there's one more case where people have historically used macros that the gurus seem to have been forgotten about: language extensions.  Anyone ever seen how you make a usable object system that works in C? Answer: lots of macros.  Anyone taken a look at the "Aspect Oriented Programming" extensions for C++?  Any guess as to how they implement it?  Yep. Bunch o' Macros (among other things).  Ever heard about the trick for making the broken for loop scope in MSVC6 work properly? Macros again.  Unfortunately macros are pretty indiscriminate in how the go and muck things up and the lack of scoping etc causes real messes.  Maybe it's possible make all those sorts of language extension types of things possible but in a cleaner way.

Yep. I think we're definitely on the same page.

>I'm just thinking out loud here, but it's definitely all related.  Templates, metaprogramming, code generation, macros.  Wouldn't it be cool to have it all unified and done right?
>
>Just a small matter of figuring out what "done right" means. :-)  Unfortunately I'm just an armchair compiler writer.  :-)
>
>Oh, one final thing to throw into the mix -- special tag keywords. Things like 'synchronized' that change some aspect of how a method runs.  Or Qt's "signal" and "slot" keywords.  Qt runs a lex/yacc parser ('moc') on header files to handle those keywords and generate boilerplate code to implement them.

Yes.  I can imagine the D concept of versioning, ie debug { ... } code, being extended to support user-designed code modification.  The details, though...

>Wouldn't
>it be nice if the language supported the ability for users to add such tags?
>Another one that would be nice to have is "script", i.e. generate wrapper code
>for binding this method with a scripting language.  Boost.python for instance
>lets you put code _elsewhere_ that says generate wrapper code for this method,
>but sometimes it would be nice if you could just flag the method itself right
>there where you declare it.
>
>--bb

Here is what I am thinking, syntax is flexible of course.

: struct xy_data {
:    userTag(mutable, pickled) {
:        int x;
:    }
:    int refcount;
:
:    userTag(pickled) {
:        double y;
:    }
: }

Once you provide this, the following new kind of "foreach" would be legal at compileTime only.  Its job is to add a reset method that resets all the "mutable" tagged members, and a "pickling" field that sends all "pickled" members to a pickler class.  Note that the "refcount" field is neither pickled nor reset.

: compileTime {
:   char[] resetCode;
:   char[] pickling;
:
:   foreach(type t; variable y; xy_data.TypeInfo.members) {
:     if (y.has_tag(mutable)) {
:       mutations = mutations ~ y.name ~ " = " ~ toString(t.init) ~ ";";
:     }
:     if (y.has_tag(pickling)) {
:       pickling = pickling ~ "S.WriteData(" ~ y.name ~ ");";
:     }
:   }
:
:   // If there are mutable members, add a reset method.
:   // It resets all fields marked "mutable" to T.init.
:
:   if (mutable.length != 0) {
:     addMethod(xy_data, "void Reset()", resetCode);
:   }
:
:   // If there are pickled members, add a pickler.
:   // It sends all fields marked "pickled" to a PickleBuffer object.
:
:   if (mutable.length != 0) {
:     addMethod(xy_data, "void Pickle(PickleStream S)", pickling);
:   }
: }

The foreach() over types changes my initial idea -- it could not be D code at
runtime.

BTW, for those worried about compile complexity, I have no problem if fabs() or cos() or "memmov" is not usable at compile time.  There would be restrictions on compile-time-able code and run-time-able code.

I do think that seperating the language into perl + D is unnecessary and requires too many conflicting idioms.


The [Impossible] dilemma:

I confess, I have a problem with the "this is impossible in a non-scripting language".

If something is considered impossible because it is too inefficient, then I would say "too inefficient compared to not having the ability?".  If it provides important, unreplaceable functionality, AND can be done in a "pay as you go" manner, then to me, it at least a candidate for inclusion.  'Slow is better than No', IMHO, as long as it doesn't make other code slower or more dangerous.

On the other hand, if we are delaying for 6 months because we are fixing bugs or hashing out the "right way" to do something, let me applaud that diligence.  I'm not asking for the perl "slap in into the language, noone will mind" approach.

(Not that I dislike Perl per se.)

Kevin

In article <d5dlc7$1jat$1@digitaldaemon.com>, Benji Smith says...
>
>Bill Baxter wrote:
>> This comment is about template metaprogramming and metaprogramming in general. It's a little more pie-in-the-sky than my last comment.
>
>I might be in the minority here, but I have a strong dislike for both meta-programming and code generation.
>
..
>
>Invariably, the code-generation code is far more difficult to read, understand, and debug (for anyone except the original author) than the non-code-generated code would have been.
..
>But in the long run, I think the benefits of code-generation and meta-programming (experienced during code authorship) are cancelled out by the problems they cause later in the software lifecycle (during debugging and maintenance).
>
>--BenjiSmith

There is a kind of code generation where you tell some program (usually a big, badly written script) that you want to write a computer game with "sprite" type animation, or a business application with such and such db tables.  The code then goes off, generates a butt-ugly gui and some windows code, and you open an editor and start digging through 200K of code with variables named like CheckBoxWidget_X1128.

In this case, yeah, I'm shivering right with ya.

But in some things, like generating ASN.1 or XML loaders from a 100 page ASN spec, the code looks at a common, standard "interface language" and writes up some stream extraction "loader" code that you never, ever, will want to edit.

In these cases, I believe that the code generation can be a benefit to maintainability and reliability.

Python can pickle, Java can do (something, what's it called?), but for fast C or C++ with ASN.1, you probably need a code generator.  Otherwise you are coding hundreds of classes that could change whenever the DTD or ASN.1 spec changes.

(Good luck if the guy that understands those classes splits for Ohio.)

Kevin

Ok, so is that what Andrew was talking about when he said "Derek already did that [compile time code generation]; it's called build.exe"?    I searched for 'build.exe' but wasn't able to find anything that looked remotely like it had anything to do with code generation or template metaprogramming.  (Maybe a less generic name might help when it comes to people finding your project via web searches.  Like dbuild or buildd or something like that).

Anyway where can I find a description of your macro processor?

--bb

In article <10s431exril0t.13imwd7sdj6zh$.dlg@40tude.net>, Derek Parnell says...
>
>On Fri, 6 May 2005 02:46:53 +0000 (UTC), Bill Baxter wrote:
>
>[snip]
>>  So my ulterior motive here is to
>> hopefully light a fire under someone's butt so they'll go off and create it
>> someday and become my hero.  I don't really expect it to happen overnight, but
>> if I just keep bringing this up in different places where smart people hang out,
>> surely someone will pick it up and run with it eventually... :-)
> 
>I'm in the process of porting my macro processor to D and integrating it with Build. That will go someway to achieving your goal.
>
>-- 
>Derek Parnell
>Melbourne, Australia
>http://www.dsource.org/projects/build/ v2.06 released 04/May/2005
>http://www.prowiki.org/wiki4d/wiki.cgi?FrontPage
>6/05/2005 1:32:44 PM