On Fri, 6 May 2005 05:32:12 +0000 (UTC), Bill Baxter wrote:

> Ok, so is that what Andrew was talking about when he said "Derek already did that [compile time code generation]; it's called build.exe"?    I searched for 'build.exe' but wasn't able to find anything that looked remotely like it had anything to do with code generation or template metaprogramming.

Good, because Build doesn't do code generation or template meta-programming ;-)

>  (Maybe a less
> generic name might help when it comes to people finding your project via web
> searches.  Like dbuild or buildd or something like that).

Well I figure if Unix can do 'make', then I can do 'build'. And have you
every tried locating 'D' using google? ;-) [Oops, I just tried and it came
up as the tenth site! out of 762,000,000 sites). However I did google
"derek parnell build" and got it on the second site.

> Anyway where can I find a description of your macro processor?

The macro processor - you can't because it is not available from the Web yet. I'm still porting it from the Euphoria language to D. There is a rather old (and incomplete) manual for it at ...

  http://www.users.bigpond.com/ddparnell/Macreu/macreu.htm

The Build utility - I usually place a hint for this tool in my signature lines below.

-- 
Derek Parnell
Melbourne, Australia
http://www.dsource.org/projects/build/ v2.06 released 04/May/2005
http://www.prowiki.org/wiki4d/wiki.cgi?FrontPage
6/05/2005 4:24:46 PM

In article <1nysb5sp8mswj$.1tw5qirnk7eyg.dlg@40tude.net>, Derek Parnell says...
>
>On Fri, 6 May 2005 05:32:12 +0000 (UTC), Bill Baxter wrote:
>
>>  (Maybe a less
>> generic name might help when it comes to people finding your project via web
>> searches.  Like dbuild or buildd or something like that).
>
>Well I figure if Unix can do 'make', then I can do 'build'.

Sure. That's fair.  Is build also a generic, language-neutral tool like 'make'? I thought build was specifically for D.

> And have you
>every tried locating 'D' using google? ;-) [Oops, I just tried and it came up as the tenth site! out of 762,000,000 sites).

Heh heh.  But "D language" or "D programming" works pretty well.

> However I did google
>"derek parnell build" and got it on the second site.

Yeh, that's how I found it eventually.  But I couldn't find it till I went and dug up the 'derek parnell' part from some newsgroup post.

> 
>> Anyway where can I find a description of your macro processor?
>
>The macro processor - you can't because it is not available from the Web yet. I'm still porting it from the Euphoria language to D. There is a rather old (and incomplete) manual for it at ...
>
>  http://www.users.bigpond.com/ddparnell/Macreu/macreu.htm

Ok. Thanks.

>The Build utility - I usually place a hint for this tool in my signature lines below.

Very clever subliminal advertising, that! ;-)

>
>-- 
>Derek Parnell
>Melbourne, Australia
>http://www.dsource.org/projects/build/ v2.06 released 04/May/2005
>http://www.prowiki.org/wiki4d/wiki.cgi?FrontPage
>6/05/2005 4:24:46 PM


--bb

In article <d5c8s4$d82$1@digitaldaemon.com>, Kevin Bealer says...
>
>In article <d5c1mu$640$1@digitaldaemon.com>, Bill Baxter says...
>>
>> [...]
>>Thoughts?  Any chances something like this could ever come to D?
>
>This has been discussed, a little, on here before.  I think Walter said he's had email discussions and sees some promise in this direction or thereabouts, but I'm not positive.

Who whoo!  I finally figured out how to search through the newsgroup archives (why no search link on the wwwnews interface? http://www.digitalmars.com/drn-bin/wwwnews?-st+digitalmars.D)

Yeh, you're right, though, this has been discussed before: http://www.google.com/custom?domains=www.digitalmars.com&q=metaprogramming&sa=Search&sitesearch=www.digitalmars.com&client=pub-5628673096434613&forid=1&ie=ISO-8859-1&oe=ISO-8859-1&safe=active&cof=GALT%3A%23008000%3BGL%3A1%3BDIV%3A%23336699%3BVLC%3A663399%3BAH%3Acenter%3BBGC%3AFFFFFF%3BLBGC%3AFFFFFF%3BALC%3A000000%3BLC%3A000000%3BT%3A0000FF%3BGFNT%3A0000FF%3BGIMP%3A0000FF%3BLH%3A50%3BLW%3A270%3BL%3Ahttp%3A%2F%2Fwww.digitalmars.com%2Fdmlogo.gif%3BS%3Ahttp%3A%2F%2F%3BFORID%3A1%3B&hl=en

This is a pretty juicy one:
http://www.digitalmars.com/d/archives/digitalmars/D/1313.html
and here too:
http://www.digitalmars.com/d/archives/digitalmars/D/4905.html
and back in 2002 when Walter was just getting templates implemented:
http://www.digitalmars.com/d/archives/7954.html
(search in page for metaprogramming to skip down to the part where that
discussion starts)
And some more:
http://www.digitalmars.com/d/archives/11603.html
http://www.digitalmars.com/d/archives/15502.html
http://www.digitalmars.com/d/archives/digitalmars/D/584.html --> and in this one
Walter says to Kevin:
"I think you're on the right track. I've been exchanging email with some C++
template exports [sic] who feel the same way you do. ... This is probably a D
2.0 feature <g>."

Cool, that's very encouraging.


Wonder what became of the likes of Norbert Nemec and Andy Friesen and some of the other big proponents of this?

Anyway, I feel my time here has been productive, even if I haven't said anything new.  I'm keeping on the radar screen at least.  ;-)


--bb

Hi..

> The most interesting aspects of meta-programming are not computing the
> values of
> constants.  If that was all there was to it, it would be pointless.  Just
> compute the value of the constant once at start up time, and quit yer
> whining
> type of thing.  Look again at the simple "integer big enough to hold N
> bits"
> example I mentioned.  The compiler can't "simply execute the function
> after
> compiling" because that function is computing the type of some variables
> in code
> that then need to be compiled.  So at the very least the process has to be
> "execute the function *before* compiling".  But even that's a little
> simplistic.
> In the example above it really needs to be executed anytime integer<N> is
> encountered in the code _during_ compiling.

I guess you're right. Still, the syntax for these meta-functions should not really have to change. The compiler would need the ability to execute the code before compiling, like you said. In can run its interpreter over the source code and use the results for the actual compile phase. A difference might be that "meta-functions" will be able to return types and not only values.

>>I think the nicest thing about meta-programming will be the possibility to introduce new operators. If we can add types, functions in code, why not also operators? You want "**" to mean "to the power of"? Just define it. "<=>" should call opCmp? Want to be able to do "&" on floats? Or maybe "^" should be "to the power of" if done on reals?
>
> I'm not sure this is really feasible or desirable (or that it is even
> really
> meta-programming).  What on earth does "&" on floats mean, for example?

It could be the bit-wise and on the floats. I don't know, it was just an example :-/

> And how
> do you deal with precedence?

That should be handled in the syntax for these custom operators.

> You want ** to mean power of, that's fine, but the
> compiler has to know somehow what the precedence of that operator is or
> all your
> math is going to come out funny.  And you better be sure that you don't
> define
> any operators that are ambiguous.  Did you mean to exponentiate "a**b" or
> to
> multiply "a * (*b)"?

True. The (meta-?) compiler will complain if it ends up with an unparsable language.

>>I can hardly wait!
>
> Or maybe this is faux-enthusiasm?  Are you being sarcastic?  Sorry if I
> didn't
> catch it.

No no, not at all. I really think it would be nice to be able to define operators the way you define classes/types/functions. Just imagine how many discussion in this list would be solved by this. You want "isnot"? Define it.

It's the old discussion all over again: should a feature be built in the compiler or in the library?

Why not define "isnot" in a "library" instead? (It won't be a .lib or anything like it, since it will just get compiled in, like templates)

> Most oo folks strongly advise against going crazy with operator
> overloading.  It may seem nifty when you're doing it, but it can make the
> code
> totally impossible to read.  Is that "*" really a "*" or is it some funky
> overloaded operator?  If so then where the heck is it defined?  And where
> else
> is this overloaded operator being used in this code?  Oh snap! I can't
> grep for
> it because it looks exactly like every other "*" in my code!  Etc.
> "Thinking in C++" puts it well:
> http://www.camtp.uni-mb.si/books/Thinking-in-C++/TIC2Vone-distribution/html/Chapter12.html#Heading349

That's a valid argument, though also applicable to function names: I can use one-character variables and functions, which won't be readable to any other coder but myself. I can already define a function "myprintf" that deletes all files in the current directory. Also, modern IDE's will be able to give custom operators some fancy color. I agree that some kind of prefix for custom operators would be helpful, though.

L.

On Fri, 6 May 2005 11:27:27 +0300, Lionello Lunesu wrote:


[snip]

> No no, not at all. I really think it would be nice to be able to define operators the way you define classes/types/functions. Just imagine how many discussion in this list would be solved by this. You want "isnot"? Define it.

Every had a look at Forth?  http://www.forth.org/

One of my very favorite languages. You can define *everything*, including operators. You can really get yourself messed up with this one ;-)

Here is how you redefine the '+' operator to mean '-'

 : + - ;

Here is 5 'plus' 4 program ...

==> 5 4 + .
==> 1 ok

-- 
Derek
Melbourne, Australia
6/05/2005 6:32:50 PM

> Every had a look at Forth?  http://www.forth.org/

Damn :-) That's really messy, especially that stack notation.
But the idea is clear: why should symbols as + - / be treated any different
from function names?

Seems to me that operator overloading is much worse than defining new operators.

New operators have no meaning so you know that custom functionality is used.

L.

In article <d5f9r4$31gf$1@digitaldaemon.com>, Lionello Lunesu says...
>
>I guess you're right. Still, the syntax for these meta-functions should not really have to change. The compiler would need the ability to execute the code before compiling, like you said. In can run its interpreter over the source code and use the results for the actual compile phase. A difference might be that "meta-functions" will be able to return types and not only values.

Needs the ability to return types, yes, but I think it also needs operators to manipulate types as data.  Like adding or removing members from a class or struct, querying what members exist in a class, comparing equality of types etc. Hmm maybe that's going too far though.  It means you could have classes in your program that aren't clearly declared anywhere:

: type generateClass(method[] methodlist) {
:    class foo {
:       int baseMember;
:    };
:    foreach (method m; methodlist) {
:       if (checksomething(m)) {
:         foo += m; // adds new method to foo
:       }
:    }
:    return foo;
: }
: generateClass(myMethods) A;

This is a hypothetical 'meta-function' that returns a class type built from a list of methods passed in. So you've now got some class A in your code but looking at the above source you will have no idea what it contains.  Is that useful?  Hmm, well maybe, actually. This isn't a totally off the wall example. I have seen almost that exact thing done with the preprocessor in C. The WINTAB tablet interface library lets you define your own custom packet struct by #defining a bunch of symbols before #including the header.  This way you can get exactly the packet type you want with no extraneous members taking up memory. If you only want pressure and x-y position, then you make a struct with just that.  If you want the stylus id and event number etc, then you can add those. The difference is that in the preprocessor version, the way it works is that every possible memeber is listed in the header, but surrounded by a #if block. So all the members _are_ listed in one place.  But the 'myMethods' list does have to be initialized somewhere.

(Incidentally, the fact that that kind of trick exists is one of the main reasons I'm skeptical whenever someone tells me "Language X makes macros completely unnecessary!".  It was said by Stroustrup with C++, and now by Walter for D.  Basically Walter's "why we don't need macros page" says "C++ wasn't supposed to need macros but they forgot about all these important cases where macros are used, but now with D I have _really_ forseen all possible cases, so now we _really_ don't need macros."  Riiiight.  Enumerating all the uses of macros is like trying to list every kind of program you can write in C.  At _best_ you can come up with a list of everything that's been done to date, but who knows what someone will think up to do with it next year?)

>>>Want to be able to do "&" on floats?
>>
>> I'm not sure this is really feasible or desirable (or that it is even
>> really
>> meta-programming).  What on earth does "&" on floats mean, for example?
>
>It could be the bit-wise and on the floats. I don't know, it was just an example :-/

Ok, but it's pretty much a perfect example of how *not* to use operator overloading.


>> And how
>> do you deal with precedence?
>
>That should be handled in the syntax for these custom operators.

Is that technically feasible when you have 5 different libraries all of which could define their own operators, and some of them possibly define the same operators with different precedence?  And how do I specify the precendence of my super-cool '<*>==' operator relative to all the other operators that could be defined by some other library in the future?  Seems like that would make it difficult for a compiler writer, because the syntax tree becomes somewhat malleable.   But I can't seem to think of a good example where the clash would really be unresolvable.  You can always put the operators in namespaces to prevent clashes, for instance.

Still it seems like a thorny nest.  And also I think a completely separate issue from metaprogramming.  Just out of curiosity, has any programming language ever offered the ability to define new infix operators with changeable precedence?  I can't think of any.

--bb

In article <d5fb6d$18v$1@digitaldaemon.com>, Lionello Lunesu says...
>Seems to me that operator overloading is much worse than defining new operators.
>
>New operators have no meaning so you know that custom functionality is used.
>

But in a math-related class, usually the standard operators do have meaning and so it makes much more sense to just be able to overload them.  Say you create a matrix class, you really want to be able to overload the + operator for matrix addition, because that's the symbol everyone uses in real life.

One thing that makes crazy operator overloading manageable in FORTH and Lisp-like languages is that there are no infix operators, and so you don't need precedence.  Every operator is pre-fix (Lisp) or post-fix (FORTH) so effectively the burden of making order of operations correct is on the user.  The result is that instead of being able to write 23+34*5 in a natural way, you have to write (+ 23 (* 34 5))  or in forth I presume something like 34 5 * 23 +.  Which is unfortunate because the point of doing operator overloading in the first place was to make the syntax more natural.

One thing about operators, though, is that usually they are trouble for IDEs. With MSVC, even with Whole Tomato's Visual Assist plug-in, I can't jump to the definition of an overloaded operator.

When you think about it, yes, overloaded operators are just like any other functions, but it is seldom that you have an ordinary function like foo() that is overloaded a billion different ways, whereas operators always have lots of overloads.  I guess my point is just that if you overload anything extensively it becomes harder to track whats going on in the code, and operator overloading is no different.

I remembered another really cool example of where metaprogramming is being used extensively to good effect.  The Sh shading language by Michael McCool. http://libsh.org/ http://www.cgl.uwaterloo.ca/Projects/rendering/Talks/metashader/aw.ppt

These days graphics hardware like the GeForce 6800 can do some really neat tricks, but programming it to do those tricks is a major pain in the butt.  The simplest "hello graphics hardware world" example is a few pages of code to set up graphics contexts, compile and load and a simple shader, and then render a quad to the screen to get something to happen.  The neat thing is that shader compilation is very quick and can be done at runtime.  So Sh is a C++ library that lets you write pretty normal looking code that just happens to get executed on the graphics hardware.

I guess all in all, it's pretty similar to the binding generation problem that Luabind or Boost.Python solve.  Just the interface is graphics hardware rather than a scripting lanuguage API.  Anyway it's pretty cool.

If D made it significantly easier to develop something like Sh I bet graphics folks would really eat it up.  I mean they've been happy to learn a new shader language like every other year to keep up with the cutting edge of graphics hardware, so why not D if it can make their lives easier?  Similarly I'm sure other folks in CS academia would use it if it meant they could get prototype systems built faster, so that they could publish results faster.

The great thing about targeting academics is that academic projects often don't have much legacy code.  So there's not a high price to pay for abandoning C++, unlike in the industry at large.  Academics were pretty quick to adopt Java also, for instance, simply because it made their lives easier, and it sped up prototype development.  And I think this was before there were even decent IDEs and whatnot.  But it's not like they don't still want speed, its just that they're willing to give up a little to get their results out faster.  So if D can give them speed of development with speed of execution, I think they'll jump on it.  And of course eventually a student who writes in D at the Uni will go out into the real world and bring his D experience with him.

I'd like to mention one more potential area in which D could excel.  Someone mentioned it before on this newsgrop, but it is numerical/scientific computing. These people are still using Fortran in part because C++ just isn't very good for numerical computing.  A C++ that was as good as Fortran (i.e. potentially D) could be a big hit.

I've mentioned Blitz++, but there are also a few linear algebra packages that use template metaprogramming techniques: MTL and TNT.  I haven't tried TNT, but I did use MTL.  It doesn't look like it's been worked on since 2001, though. This is conjecture, but I think MTL basically collapsed under its own weight. It was just too complicated to maintain all that template mess in C++, and when the original developer left it behind there was really no one else who could understand it.  But _despite_ that, somehow it managed to attract a pretty active user community.  So my point of mentioning all this is that the scientific computing world is also one in which D could definitely take hold. But it has to make the right choices. Scientific computing is still mostly fortran, and it'll probably stay that way, because nobody wants to rewrite all the millions of lines of code out there that works.  But thats fine because D can call legacy fortran routines just fine.  Today, some folks in scientific computing are starting to use C++ for new work, but I think it's still at a point where they have no major attachments to C++.  It definitely presents an oportunity for D.

--bb

Kevin, I like your proposal.  More below.

In article <d5epph$2hp9$1@digitaldaemon.com>, Kevin Bealer says...
>
>Here is what I am thinking, syntax is flexible of course.
>
>: struct xy_data {
>:    userTag(mutable, pickled) {
>:        int x;
>:    }
>:    int refcount;
>:
>:    userTag(pickled) {
>:        double y;
>:    }
>: }
>
>Once you provide this, the following new kind of "foreach" would be legal at compileTime only.  Its job is to add a reset method that resets all the "mutable" tagged members, and a "pickling" field that sends all "pickled" members to a pickler class.  Note that the "refcount" field is neither pickled nor reset.
>
>: compileTime {
>:   char[] resetCode;
>:   char[] pickling;
>:
>:   foreach(type t; variable y; xy_data.TypeInfo.members) {
>:     if (y.has_tag(mutable)) {
>:       mutations = mutations ~ y.name ~ " = " ~ toString(t.init) ~ ";";
>:     }
>:     if (y.has_tag(pickling)) {
>:       pickling = pickling ~ "S.WriteData(" ~ y.name ~ ");";
>:     }
>:   }
>:
>:   // If there are mutable members, add a reset method.
>:   // It resets all fields marked "mutable" to T.init.
>:
>:   if (mutable.length != 0) {
>:     addMethod(xy_data, "void Reset()", resetCode);
>:   }
>:
>:   // If there are pickled members, add a pickler.
>:   // It sends all fields marked "pickled" to a PickleBuffer object.
>:
>:   if (mutable.length != 0) {
>:     addMethod(xy_data, "void Pickle(PickleStream S)", pickling);
>:   }
>: }
>
>The foreach() over types changes my initial idea -- it could not be D code at
>runtime.

And of course, all that could also be put in a "compile time" function with the struct as an argument, so that you could easily perform the same augmentation on every struct that needs it with one line of code, like 'decorateStruct(xy_data)'.

I like it.

>The [Impossible] dilemma:
>
>I confess, I have a problem with the "this is impossible in a non-scripting language".

Yeh, I agree.  But there are some things you just can't do if you don't have an interpreter.  Like, it might be nice to be able to create simple executable statements out of strings at runtime and run them, but without going the big step of including an interpreter in the run-time library, that can't happen. But I guess that's not really a counter-argument.  Having a run-time interpreter around is just another "price you have to pay" if you want the functionality.

Yeh, what you say really makes a lot of sense.  I just spent a few weeks back in february figuring out how to interface a scripting language with a C++ program. Yuck.  I hear a number of people these days saying you've got to have C++ for the speed  but scriptablility is also a must (true in games and for office type apps), and so the solution is these cobbled together solutions that try to marry C++ with python or what have you.  But what's the point?  If it's generally agreed that scripting is better for some things, why not just have "one language to rule them all"?  Why can't you have a language like D provide a run-time interpreter too?  If you don't need it, you don't pay for it.  Basically it would be like having compiler-level support for generating all the gobbledy-gook glue you need to interface with a script language, with the added bonus of having a unified syntax.

So putting it all together we've now got 3 phases of code execution.  :-) Interpreted compile-time execution (metaprogramming), compiled run-time execution (the norm), and interpreted run-time execution (scripting).  Are we missing anything?  Compiled compile-time execution?  Sure, why not!  I guess this would be what you'd do if all the metaprogramming tricks got to to be so time consuming to compile that they needed to be executed natively. Let's hope that doesn't happen.  Or you could maybe think of 'compiled compile-time execution' as being a code generation phase in which metaprogramming constructs are actually physically transformed into new code.  So it's compiling metacode into regular program code.  Hmm.  Maybe that's not so useful.

But I like the idea of adding run-time interpreting capabilities right into the language.  That would be neat.  That would be even cooler than having first class support for meta-programming, I think.