kris wrote:
> Thus; shouting from the rooftops that D is all about meta-code, and DSL
> up-the-wazzoo, may well provoke a backlash from the very people who
> should be embracing the language. I'd imagine Andrei would vehemently
> disagree, but so what? The people who will ultimately be responsible for
> "allowing" D through the door don't care about fads or technical
> superiority; they care about costs. And the overwhelming cost in
> software development today, for the type of companies noted above, is
> maintenance. For them, software dev is already complex enough. In all
> the places I've worked or consulted, in mutiple countries, and since the
> time before Zortech C, pedestrian-code := maintainable-code := less
> overall cost.

Some comments:

1) D has no marketing budget. It isn't backed by a major corporation. Therefore, it needs something else to catch peoples' attention. Mundane features aren't going to do it.

2) I know Java is wildly successful. But Java ain't the language for me - because it takes too much code to do the simplest things. It isn't straightforward clarifying code, either, it looks like a lot of irrelevant bother. I'm getting older and I just don't want to spend the *time* to write all that stuff. My fingertips get sore <g>. I wouldn't use Java if it was twice as fast as any other language for that reason. I wouldn't use Java if it was twice as popular as it is now for that reason.

3) Less code == more productivity, less bugs. I don't mean gratuitously less code, I mean less code in the sense that one can write directly what one means, rather than a lot of tedious bother. For example, if I want to visit each element in an array:

	foreach(v; e)
	{...}

is more direct than:

	for (size_t i = 0; i < sizeof(e)/sizeof(e[0]); i++)
	{ T v = e[i];
	 ... }


4) The more experience I have, the more it seems that the language that got a lot right is ... Lisp. But Lisp did one thing terribly, terribly wrong - the syntax. The Lisp experts who can get past that seem to be amazingly productive with Lisp. The rest of us will remain envious of what Lisp can do, but will never use it.

5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.

6) If I think about it a certain way, it looks like what C++ Boost is doing is a desperate attempt to add Lisp-like features. By desperate I mean that C++'s core feature set is just inadequate to the task. For example, look at all the effort Boost has gone through to do a barely functioning implementation of tuples. Put tuples into the language properly, and all that nasty stuff just falls away like a roofer peeling off the old shingles.

7) A lot of companies have outlawed C++ templates, and for good reason. I believe that is not because templates are inherently bad. I think that C++ templates are a deeply flawed because they were ***never designed for the purpose to which they were put***.

8) I've never been able to create usable C++ templates. Notice that the DMD front end (in C++) doesn't use a single template. I know how they work (in intimate detail) but I still can't use them.

9) But I see what C++ templates can do. So to me, the problem is to design templates in such a way that they are as simple to write as ordinary functions. *Then*, what templates can do can be accessible and maintainable. It's like cars - they used to be very difficult to drive, but now anyone can hop in, turn the key, and go.

10) Your points about pedestrian code are well taken. D needs to do pedestrian code very, very well. But that isn't enough because lots of languages do pedestrian code well enough.

11) Today's way-out feature is tomorrow's pedestrian code. I'm old enough to remember when "structured code", i.e. while, for, switch instead of goto, was the way-out feature (70's). Then, OOP was all the rage (80's), now that's a major yawner. STL was then the way-out fad (90's), now that's pedestrian too. Now it's metaprogramming (00's), and I bet by 2015 that'll be ho-hum too, and it's D that's going to do it.

12) Take a look at what Kirk McDonald is doing with Pyd. He needs all this stuff to make it slicker than oil on ground steel. He's on the bleeding edge of stuff D needs to *make* pedestrian.

Walter Bright wrote:
> kris wrote:
> 
> 7) A lot of companies have outlawed C++ templates, and for good reason. I believe that is not because templates are inherently bad. I think that C++ templates are a deeply flawed because they were ***never designed for the purpose to which they were put***.

Very true.

At a previous employ the major reason templates where outlawed was because they caused compilation time to slow down dramatically, if you didn't know what you where doing.

The other reason was like you say, they get very complex to understand quickly for all but the most simple cases.  Unfortunately without templates you generally end up writing more code or at the least, less efficient code.  Which is a maintenance headache.

-Joel

Walter Bright wrote:
> kris wrote:
>  > Thus; shouting from the rooftops that D is all about meta-code, and DSL
>  > up-the-wazzoo, may well provoke a backlash from the very people who
>  > should be embracing the language. I'd imagine Andrei would vehemently
>  > disagree, but so what? The people who will ultimately be responsible for
>  > "allowing" D through the door don't care about fads or technical
>  > superiority; they care about costs. And the overwhelming cost in
>  > software development today, for the type of companies noted above, is
>  > maintenance. For them, software dev is already complex enough. In all
>  > the places I've worked or consulted, in mutiple countries, and since the
>  > time before Zortech C, pedestrian-code := maintainable-code := less
>  > overall cost.
> 
> Some comments:
> 
> 1) D has no marketing budget. It isn't backed by a major corporation. Therefore, it needs something else to catch peoples' attention. Mundane features aren't going to do it.

It's already /stacked/ with attention grabbing features :)


> 
> 2) I know Java is wildly successful. But Java ain't the language for me - because it takes too much code to do the simplest things. It isn't straightforward clarifying code, either, it looks like a lot of irrelevant bother. I'm getting older and I just don't want to spend the *time* to write all that stuff. My fingertips get sore <g>. I wouldn't use Java if it was twice as fast as any other language for that reason. I wouldn't use Java if it was twice as popular as it is now for that reason.
> 

A lot of people feel that way. D can potentially tap that vast market.


> 3) Less code == more productivity, less bugs. I don't mean gratuitously less code, I mean less code in the sense that one can write directly what one means, rather than a lot of tedious bother. For example, if I want to visit each element in an array:
> 
>     foreach(v; e)
>     {...}
> 
> is more direct than:
> 
>     for (size_t i = 0; i < sizeof(e)/sizeof(e[0]); i++)
>     { T v = e[i];
>      ... }
> 

Yep, that's great! One of the reasons I like D so much, along with array slicing.


> 
> 4) The more experience I have, the more it seems that the language that got a lot right is ... Lisp. But Lisp did one thing terribly, terribly wrong - the syntax. The Lisp experts who can get past that seem to be amazingly productive with Lisp. The rest of us will remain envious of what Lisp can do, but will never use it.
> 

True.


> 5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.

Yet, Lisp will always remain a niche language. You have to wonder why.


> 
> 6) If I think about it a certain way, it looks like what C++ Boost is doing is a desperate attempt to add Lisp-like features. By desperate I mean that C++'s core feature set is just inadequate to the task. For example, look at all the effort Boost has gone through to do a barely functioning implementation of tuples. Put tuples into the language properly, and all that nasty stuff just falls away like a roofer peeling off the old shingles.

Boost seems more like a language within a language <g> The very thing that increases long-term costs.

> 
> 7) A lot of companies have outlawed C++ templates, and for good reason. I believe that is not because templates are inherently bad. I think that C++ templates are a deeply flawed because they were ***never designed for the purpose to which they were put***.

Agreed. But the issue is not about how badly they're flawed. Instead, it's the non-standard "language" problem. The MyDSL problem :)

> 
> 8) I've never been able to create usable C++ templates. Notice that the DMD front end (in C++) doesn't use a single template. I know how they work (in intimate detail) but I still can't use them.

Same here.

> 
> 9) But I see what C++ templates can do. So to me, the problem is to design templates in such a way that they are as simple to write as ordinary functions. *Then*, what templates can do can be accessible and maintainable. It's like cars - they used to be very difficult to drive, but now anyone can hop in, turn the key, and go.

Templates are certainly a useful tool in D. Tango has some very simple usage for handling char[]/wchar[]/dchar[] implementation.

I also look at some D templates for ages, and still can't figure out just how they work. Don Clugston is known around here as the Template Ninja -- the very name itself shouts out "Here Dwell Demons!" :-D

That kind of thing is a point of diminishing return, anywhere where cost is king. Relates to #7


> 
> 10) Your points about pedestrian code are well taken. D needs to do pedestrian code very, very well. But that isn't enough because lots of languages do pedestrian code well enough.
> 
> 11) Today's way-out feature is tomorrow's pedestrian code. I'm old enough to remember when "structured code", i.e. while, for, switch instead of goto, was the way-out feature (70's). Then, OOP was all the rage (80's), now that's a major yawner. STL was then the way-out fad (90's), now that's pedestrian too. Now it's metaprogramming (00's), and I bet by 2015 that'll be ho-hum too, and it's D that's going to do it.

The danger is, of course, the language-within-a-language that leads to #7. In structured and OOP, if you learned the language you could maintain the code (we disregard the fact that anyone can write truly unmaintainable code if they try to).

Yet, this could be described as 'contained' within the language. Quite different than what causes #7 :)


> 12) Take a look at what Kirk McDonald is doing with Pyd. He needs all this stuff to make it slicker than oil on ground steel. He's on the bleeding edge of stuff D needs to *make* pedestrian.

Certainly :)

What Kirk has been doing (much awesometude there) is one of those things that fit into the "narrow focus" or "speciality" field that /can/ benefit in some manner. But it's a black-box. When it works, nobody will fuss with the insides unless they /really/ have to. That's not how most commerical software is done today, regardless of all the efforts to make it more like circuit-design.


- Kris

Great post;  all of this is interesting, and I think I agree with all of the individual points.

Walter Bright wrote:
> kris wrote:
...
> 
> Some comments:
...
> 4) The more experience I have, the more it seems that the language that got a lot right is ... Lisp. But Lisp did one thing terribly, terribly wrong - the syntax. The Lisp experts who can get past that seem to be amazingly productive with Lisp. The rest of us will remain envious of what Lisp can do, but will never use it.
> 
> 5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.

Yes - I've often wished for Ocaml or LISP with C syntax, but LISP has another real issue to me, which is that 'mutable' algorithms are hard.

> 6) If I think about it a certain way, it looks like what C++ Boost is doing is a desperate attempt to add Lisp-like features. By desperate I mean that C++'s core feature set is just inadequate to the task. For example, look at all the effort Boost has gone through to do a barely functioning implementation of tuples. Put tuples into the language properly, and all that nasty stuff just falls away like a roofer peeling off the old shingles.
...
> 9) But I see what C++ templates can do. So to me, the problem is to design templates in such a way that they are as simple to write as ordinary functions. *Then*, what templates can do can be accessible and maintainable. It's like cars - they used to be very difficult to drive, but now anyone can hop in, turn the key, and go.
...
> 11) Today's way-out feature is tomorrow's pedestrian code. I'm old enough to remember when "structured code", i.e. while, for, switch instead of goto, was the way-out feature (70's). Then, OOP was all the rage (80's), now that's a major yawner. STL was then the way-out fad (90's), now that's pedestrian too. Now it's metaprogramming (00's), and I bet by 2015 that'll be ho-hum too, and it's D that's going to do it.
...

Code in C++ wants to be iterative/imperative, and code in LISP wants to be recursive and functional.  But the 'macro system' in both cases is written as a functional / recursive system.  In LISP, this was by design and works with the language, in C++ its because recursion happened to fit through a keyhole in the template grammer.

LISP makes it hard / awkward to work with large mutable structures. Everything is functional so a functional macro system can do everything.

C++ templates were great (by today's standards, adequate) for container classes, but that task could almost be done just with text substitution.

So...

In my view, to make metaprogramming work in a way that non-LISP people understand in something like C++ or D, we need to see programs as a series of iterations:

1. A program with all the static code, but main() is replaced by calls
   to functions in each module that manufacture code (code factories).

2. Same as 1, with the newly manufactured code added, along with new
   calls to code factories.

3. Same as 2, etc.

N. A program with all static (non-template) code.

Each step is run to produce the next step -- in most cases, this consists of running one code generator after another -- the individual steps are not actually explicit, but it is important for one code generator to be able to use the results of another when it needs to.

All steps except N are optional, and usually you would have just 1 and N.  It might be a good idea at first to require that.

The difference between this world order and the current (C++) way, is that in C++ the factories are defined as recursive templates that expand like fractals.

Rationale:

LISP macros have access to essentially all of LISP.  If they didn't have access to (for instance) mapcar, the LISP macro system would be measurably harmed.

D programs need the same consideration --- if a 'regular D' programmer had to do without std.string, they would feel the absence every time he needed find().  Similarly, every time I try to write a meta-program in D today, I feel the same absence of std.string.

I think to really write clean code for the 'code factory' step, I need to be able to write nested loops over the user input, accumulate code in strings with "~=", and so on without writing it recursively.

My first thought on "how" is to build an interpreter in the compiler that can run the parse tree of a D function.  Which brings us fairly close to the LISP domain, the primary difference being that this interpreter would not necessarily be included in the compiled program (i.e. its not 'eval()', yet.)

(Without having written a real working compiler, I don't know how practical this is; I think I've heard of parse-tree based interpreters in relation to perl.  A reasonable restriction might be "no C calls".)

Kevin

kris wrote:
> Walter Bright wrote:
>> 3) Less code == more productivity, less bugs. I don't mean gratuitously less code, I mean less code in the sense that one can write directly what one means, rather than a lot of tedious bother. For example, if I want to visit each element in an array:
>>
>>     foreach(v; e)
>>     {...}
>>
>> is more direct than:
>>
>>     for (size_t i = 0; i < sizeof(e)/sizeof(e[0]); i++)
>>     { T v = e[i];
>>      ... }
>>
> 
> Yep, that's great! One of the reasons I like D so much, along with array slicing.

The C++ version is even *worse* than the C one (for wordiness bother):

	for (std::vector<T>::const_iterator i = e.begin(); i != e.end(); i++)
	{  T v = *i;
	... }

I mean I know the reasons for every bit of the syntax there, and in isolation they make sense, but put it all together and it seems to go backwards.


>> 5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.
> 
> Yet, Lisp will always remain a niche language. You have to wonder why.

I'm pretty sure it's the syntax.


>> 7) A lot of companies have outlawed C++ templates, and for good reason. I believe that is not because templates are inherently bad. I think that C++ templates are a deeply flawed because they were ***never designed for the purpose to which they were put***.
> 
> Agreed. But the issue is not about how badly they're flawed. Instead, it's the non-standard "language" problem. The MyDSL problem :)

I disagree with that. When you write a program using classes and functions, you *are* creating your own language. Classes are your custom types, and functions are your custom operators.


>> 8) I've never been able to create usable C++ templates. Notice that the DMD front end (in C++) doesn't use a single template. I know how they work (in intimate detail) but I still can't use them.
> 
> Same here.

But I have been able to create usable D templates <g>.


> I also look at some D templates for ages, and still can't figure out just how they work. Don Clugston is known around here as the Template Ninja -- the very name itself shouts out "Here Dwell Demons!" :-D

The very fact that Don's called the Template Ninja is a problem - after all, there is no "function ninja", no "typedef ninja", no "+ ninja". It's a sign that templates are still not easy enough to use.

It's like Paul Mensonidas being recognized as the "World's Leading Expert on the C Preprocessor." Obviously, something is seriously wrong with the preprocessor if there's an ecological niche for a world's leading expert on it. (By the way, Paul is a very nice fellow and has been kind enough to help me iron out several subtle bugs in the DMC++ preprocessor. As long as we're saddled with that preprocessor spec, I'm glad there is a Paul to help!)


>> 12) Take a look at what Kirk McDonald is doing with Pyd. He needs all this stuff to make it slicker than oil on ground steel. He's on the bleeding edge of stuff D needs to *make* pedestrian.
> 
> Certainly :)
> 
> What Kirk has been doing (much awesometude there) is one of those things that fit into the "narrow focus" or "speciality" field that /can/ benefit in some manner. But it's a black-box. When it works, nobody will fuss with the insides unless they /really/ have to. That's not how most commerical software is done today, regardless of all the efforts to make it more like circuit-design.

Even if D fails to make metaprogramming easy for average joe coders to use, if it still can be used by experts to create useful black box code like Pyd, then it is worthwhile. After all, even though trig functions are very hard to write, they are easily used by joe coders as black box components without any problems.

(Nerd that I am, I've spent many contented hours twinking around with the internals of math functions to get them juuuust right <g>.)

k
>
> Agreed. But the issue is not about how badly they're flawed. Instead, it's the non-standard "language" problem. The MyDSL problem :)

What do u mean by MyDSL (I must have missed the discussion there)? I'll take a stab.  One issue DSL causes is that it can cause code to become un-standardized.  You come across a DSL you haven't seen before, and now you've got to scan though all the crazy template code to figure out what its doing.  Am I close?

At least in my line of work, developing new DSL is part of the job.  I probably use in 5 of them a day (XML sub-languages, file binaries, shader languages, makefiles, lua, scripting languages, network subscriptions, gui-loading, data-base communication ect...).  I probably create a new one about every month.  These are not compile into the language, but they might as well be so.  Design, art and even myself cannot let the complexity of C++ get in the way of these repetitive tasks.

Even C++ code is written in certain ways which I consider DSL's (albit much duplication and increased likelihood for errors).  Actually I'm considering making an intermediate language and a tool so design/art can write some of this themselves.  Then hand it to us for integration -> one less communication bottleneck.

If fact if you've ever created your own binary file or loaded an XML, that's a DSL.  Whether its in compiled-code or run-time code, the cost doesn't change as long as its just as easy to write either way.

Note: I'm not arguing that meta-programming should be higher priority then say reflection.  I'm just arguing that its just an extension to what programmers do on a day-to-day basis.

I also think it will be a while before we will realize the full potential of DSL.  Like anything else they should be used with care.

-Joel

Kevin Bealer wrote:
> My first thought on "how" is to build an interpreter in the compiler that can run the parse tree of a D function.  Which brings us fairly close to the LISP domain, the primary difference being that this interpreter would not necessarily be included in the compiled program (i.e. its not 'eval()', yet.)

Andrei and I have come up with some scribbling that looks like a start on being able to manipulate parse trees. At least we know what we want it to look like <g>, but it's a looong way from being a solid design and then an implementation.

I'm not sure where this will lead, it's like Fulton thinking "I've got this steam engine over here, and a boat over there, ..."

Walter Bright wrote:
> kris wrote:

>>> 5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.
>>
>> Yet, Lisp will always remain a niche language. You have to wonder why.
> 
> I'm pretty sure it's the syntax.

And the recursion.
People just don't naturally think recursively.

And the lack of mutable data structures.
OCaml tried to fix that, but OCaml's probably always going to be niche as well (see first point).

--bb

janderson wrote:
> k
>  >
>  > Agreed. But the issue is not about how badly they're flawed. Instead, it's the non-standard "language" problem. The MyDSL problem :)
> 
> What do u mean by MyDSL (I must have missed the discussion there)? I'll take a stab.  One issue DSL causes is that it can cause code to become un-standardized.  You come across a DSL you haven't seen before, and now you've got to scan though all the crazy template code to figure out what its doing.  Am I close?
> 
> At least in my line of work, developing new DSL is part of the job.  I probably use in 5 of them a day (XML sub-languages, file binaries, shader languages, makefiles, lua, scripting languages, network subscriptions, gui-loading, data-base communication ect...).
[Snip]

I forgot to mention that "Shader Languages" are actually compiled down to c code for fast loading.  We also have other DSL which are compiled into C++ code (which I can't say much more on) whenever we build.

-Joel

Walter Bright Wrote:

> kris wrote:
>  > Thus; shouting from the rooftops that D is all about meta-code, and DSL
>  > up-the-wazzoo, may well provoke a backlash from the very people who
>  > should be embracing the language. I'd imagine Andrei would vehemently
>  > disagree, but so what? The people who will ultimately be responsible for
>  > "allowing" D through the door don't care about fads or technical
>  > superiority; they care about costs. And the overwhelming cost in
>  > software development today, for the type of companies noted above, is
>  > maintenance. For them, software dev is already complex enough. In all
>  > the places I've worked or consulted, in mutiple countries, and since the
>  > time before Zortech C, pedestrian-code := maintainable-code := less
>  > overall cost.
> 
> Some comments:
> 
> 1) D has no marketing budget. It isn't backed by a major corporation. Therefore, it needs something else to catch peoples' attention. Mundane features aren't going to do it.
> 
> 2) I know Java is wildly successful. But Java ain't the language for me - because it takes too much code to do the simplest things. It isn't straightforward clarifying code, either, it looks like a lot of irrelevant bother. I'm getting older and I just don't want to spend the *time* to write all that stuff. My fingertips get sore <g>. I wouldn't use Java if it was twice as fast as any other language for that reason. I wouldn't use Java if it was twice as popular as it is now for that reason.
> 
> 3) Less code == more productivity, less bugs. I don't mean gratuitously less code, I mean less code in the sense that one can write directly what one means, rather than a lot of tedious bother. For example, if I want to visit each element in an array:
> 
> 	foreach(v; e)
> 	{...}
> 
> is more direct than:
> 
> 	for (size_t i = 0; i < sizeof(e)/sizeof(e[0]); i++)
> 	{ T v = e[i];
> 	 ... }
> 
> 
> 4) The more experience I have, the more it seems that the language that got a lot right is ... Lisp. But Lisp did one thing terribly, terribly wrong - the syntax. The Lisp experts who can get past that seem to be amazingly productive with Lisp. The rest of us will remain envious of what Lisp can do, but will never use it.
> 
> 5) Lisp gets things right, according to what I've read from heavy Lisp users, by being a language that can be modified on the fly to suit the task at hand, in other words, by having a customizable language one can achieve dramatic productivity gains.
> 
> 6) If I think about it a certain way, it looks like what C++ Boost is doing is a desperate attempt to add Lisp-like features. By desperate I mean that C++'s core feature set is just inadequate to the task. For example, look at all the effort Boost has gone through to do a barely functioning implementation of tuples. Put tuples into the language properly, and all that nasty stuff just falls away like a roofer peeling off the old shingles.
> 
> 7) A lot of companies have outlawed C++ templates, and for good reason. I believe that is not because templates are inherently bad. I think that C++ templates are a deeply flawed because they were ***never designed for the purpose to which they were put***.
> 
> 8) I've never been able to create usable C++ templates. Notice that the DMD front end (in C++) doesn't use a single template. I know how they work (in intimate detail) but I still can't use them.
> 
> 9) But I see what C++ templates can do. So to me, the problem is to design templates in such a way that they are as simple to write as ordinary functions. *Then*, what templates can do can be accessible and maintainable. It's like cars - they used to be very difficult to drive, but now anyone can hop in, turn the key, and go.
> 
> 10) Your points about pedestrian code are well taken. D needs to do pedestrian code very, very well. But that isn't enough because lots of languages do pedestrian code well enough.
> 
> 11) Today's way-out feature is tomorrow's pedestrian code. I'm old enough to remember when "structured code", i.e. while, for, switch instead of goto, was the way-out feature (70's). Then, OOP was all the rage (80's), now that's a major yawner. STL was then the way-out fad (90's), now that's pedestrian too. Now it's metaprogramming (00's), and I bet by 2015 that'll be ho-hum too, and it's D that's going to do it.
> 
> 12) Take a look at what Kirk McDonald is doing with Pyd. He needs all this stuff to make it slicker than oil on ground steel. He's on the bleeding edge of stuff D needs to *make* pedestrian.


for me reading this post it's amazingly encouraging and future-exciting,

thanx!