Jari-Matti Mäkelä wrote:
> bearophile wrote:
> 
>> As Sing# (http://en.wikipedia.org/wiki/Sing_sharp ) and Chapel, Scala is
>> one of the languages to be followed more, because they share some of
>> future purposes of D2/D3.
>>
>> A small presentation about the close future of Scala (it's not a general
>> introduction to Scala): "Scala -- The Next 5 Years" by Martin Odersky:
>> http://www.scala-lang.org/sites/default/files/odersky/scalaliftoff2009.pdf
> 
> Scala is an impressive language and overall well designed. There are certainly truckloads of features that could be taken from Scala to D. But I'm afraid that the desire to have familiarity and compatibility with the C/C++ family is more important in this community than cool new functional features. Here's a quick comparison of some factors:
[snip]

You are making very good points, thanks for an insightful post. A few of them are slightly inaccurate, but those were discussed by Walter already. I'd like to make a few high-level comments.

I find the lack of rigor in certain parts of D as damaging as you do. Fortunately, that has improved by leaps and bounds and is still improving. When I joined Walter with working on D back in 2006, D was quite a lot less theoretically-motivated - it was all greasy nuts and bolts and pegs hammered in place. There was a tacit agreement between Walter and me that D will improve the rigor of its ways, and indeed we have made strides in that respect. If things work out properly, we'll be able to define SafeD - a proper, checkable safe subset of the language.

It would be great if there were more PL heavyweights in the D community (as you mention there are in Scala), but D's community is in large part young and enthusiastic, and that's an asset. Possibly some of the young students using D will become PL experts themselves and help improve it. (Unlike you, I think that by and large there is no hatred or shunning of the ivory tower.) Among other things, it would be great if you yourself hung out more around here. It's a risky proposition, but I guess it's a simple fact that you have a better chance at making a vital contribution to D than to Scala.

About unifying FP and OOP, I hope you can educate me a bit. My understanding from a cursory search is that Scala does not have true immutability and purity, only Java-style by-convention immutability a la String class. Is that correct? If that's the case, I think Scala got the syntactic sugar part of FP right (pattern matching) but not the purity and immutability aspects, which I believe are much more important. These aspects are related to concurrency as well, where I think Scala has a distinctly underwhelming lineup. Even with what we know we can implement now in D concurrency-wise, we'll be in better shape than many (most? all?) other languages. Stay tuned.

I agree that D could improve its type deduction - it doesn't even do Hindley-Milner. (Incidentally, you chose a particularly poor example regarding the type of array literals - those are very easy to type properly (and in fact I do so with a library function), but Walter insisted that typing by the first member is simple enough for people to understand and remember.) I'm afraid D's type deduction abilities will stay at about this level at least for D2. The good news is that, due to templates' deferred typechecking, I haven't found that to be a huge problem in practice.


Andrei

Andrei Alexandrescu Wrote:

> Jari-Matti Mäkelä wrote:
> > bearophile wrote:
> > 
> >> As Sing# (http://en.wikipedia.org/wiki/Sing_sharp ) and Chapel, Scala is one of the languages to be followed more, because they share some of future purposes of D2/D3.
> >>
> >> A small presentation about the close future of Scala (it's not a general introduction to Scala): "Scala -- The Next 5 Years" by Martin Odersky: http://www.scala-lang.org/sites/default/files/odersky/scalaliftoff2009.pdf
> > 
> > Scala is an impressive language and overall well designed. There are certainly truckloads of features that could be taken from Scala to D. But I'm afraid that the desire to have familiarity and compatibility with the C/C++ family is more important in this community than cool new functional features. Here's a quick comparison of some factors:
> [snip]
> 
> You are making very good points, thanks for an insightful post. A few of them are slightly inaccurate, but those were discussed by Walter already. I'd like to make a few high-level comments.
> 
> I find the lack of rigor in certain parts of D as damaging as you do. Fortunately, that has improved by leaps and bounds and is still improving. When I joined Walter with working on D back in 2006, D was quite a lot less theoretically-motivated - it was all greasy nuts and bolts and pegs hammered in place. There was a tacit agreement between Walter and me that D will improve the rigor of its ways, and indeed we have made strides in that respect. If things work out properly, we'll be able to define SafeD - a proper, checkable safe subset of the language.

That's great to hear.

> It would be great if there were more PL heavyweights in the D community (as you mention there are in Scala), but D's community is in large part young and enthusiastic, and that's an asset. Possibly some of the young students using D will become PL experts themselves and help improve it.

That's more likely. I can see why the approach doesn't attract PL post-grad students, but that's not necessarily a huge problem.

> (Unlike you, I think that by and large there is no hatred or shunning of the ivory tower.)

I've experienced this kind of attitude in the D's irc channel and sometimes here on the newsgroup. It's quite understandable that since D consists of so many heterogenic features, different kinds of aspects motivate us. To me the coherent "user interface" of the language and its semantics are one of the appealing sides, but for some other people things like backend optimizations and compatibility with existing C/C++ libraries matters more and e.g. rigorous language semantics are just bikeshedding.

I have little problem reading or writing code written in a cryptic unorthogonal language - on the contrary I sometimes find it an exciting way of spending time. But when I sometimes suggest some feature or change in semantics, my goal is to make the language easier to use for all of us.

> Among other things, it would be great if you yourself hung out more around here. It's a risky proposition, but I guess it's a simple fact that you have a better chance at making a vital contribution to D than to Scala.

I do hang out here, have done so since 2003. And I have some hopes that D becomes useful for me again. I've liked the direction D2 is heading towards, and once the largest problems have been ironed out, I might actually start using it (or D3), and I'm sure there are others who will do the same.

> About unifying FP and OOP, I hope you can educate me a bit. My understanding from a cursory search is that Scala does not have true immutability and purity, only Java-style by-convention immutability a la String class.

It doesn't have a system similar to what D2 has, but the Java-style immutability is used to provide e.g. immutable collections. I'm expecting a change here later when the concurrency becomes a larger issue (it should be already, but you only have so much time!). I've done some research on concurrency issues lately and it seems to me that there are so many computational models to choose from that in some sense it might be too early to stick with one.

> Is that correct? If that's the case, I think Scala got the syntactic sugar part of FP right (pattern matching) but not the purity and immutability aspects, which I believe are much more important.

Agreed, it's not a pure functional language. In fact the functional feel is quite different from the old school functional languages.

> I agree that D could improve its type deduction - it doesn't even do Hindley-Milner. (Incidentally, you chose a particularly poor example regarding the type of array literals - those are very easy to type properly (and in fact I do so with a library function), but Walter insisted that typing by the first member is simple enough for people to understand and remember.) I'm afraid D's type deduction abilities will stay at about this level at least for D2. The good news is that, due to templates' deferred typechecking, I haven't found that to be a huge problem in practice.

Yea, I did some other tests and it really seems the type inference has gotten much better. It just felt dishonest to claim something without providing examples. I expected more problems, but was happy to see that many cases indeed worked now.

Jari-Matti Mäkelä wrote:
> Agreed, you /can/ do something similar. But in Scala it's the standard way of doing things. If you compare the grammars of both languages, you'll see that Scala is a bit lighter than D (see http://www.scala-
> lang.org/sites/default/files/linuxsoft_archives/docu/files/ScalaReference.pdf)

Yes, it's a bit different philosophy. I like the advantage of built in control structures, because that leads to commonality between code bases. I wished to avoid things like C++ string, where everyone invents their own string class that's incompatible with everyone elses'. Whether that latter effect occurs in Scala or not, I don't know.


> I exaggerated a bit. But there are some constructs that some think should not be there, e.g. foreach_reverse.

Perhaps, but you can just ignore it.


> I've submitted couple of reports years ago and luckily some of them have already been fixed. Maybe the search is broken.

Can you provide bug numbers?


>>> Scala is a genius at inferring
>>> types. D doesn't really have a clue.
>> Can you give an example?
> 
> http://d.puremagic.com/issues/show_bug.cgi?id=3042
> 
> auto foo = [ 1, 2L ]; // typeof == int[2u]
> auto foo = [ 2L, 1 ]; // typeof == long[2u]
> auto foo = [ "a", "abcdefgh" ]; // typeof == char[1u][2u] in D1
> auto foo = [ [], [1,2,3] ]; // doesn't even compile
> 
>>> Especially the array literal type inference is really naive.
>> How should it be done?
> 
> You shouldn't use the type of the first given element when constructing the type of the array. If you have [ e_1, ..., e_n ], the type of the literal is
> unify(type_of_e_1, ..., type_of_e_n) + "[]". For instance:
> 
> => typeof([ [], [1,2,3] ])
> => unify( typeof([]), typeof([1,2,3]) ) + "[]"
> => unify( "a[]", unify(typeof(1),typeof(2),typeof(3)) + "[]" ) + "[]"
> => unify( "a[]", unify("int","int","int") + "[]" ) + "[]"
> => unify( "a[]", "int" + "[]" ) + "[]"
> => unify( "a[]", "int[]" ) + "[]"   // a is a local type var, subst = { a -> int }
> => "int[]" + "[]"
> => "int[][]"

Ok.

bearophile wrote:
> Walter Bright:
> 
>> Actually, you can do them with "lazy" function arguments. There was
>> an example somewhere of doing control structures with it.<
> 
> There are some problems with this: - Are current (especially LDC)
> compilers able to inline those lazy delegates?

No, but it's not a technical issue, just more of no demand for it. But this would not be a reason to use Scala, as one doesn't use Scala for performance oriented programs.


> I'ver tried to see how the LDC compiles it and people
> there have shown me distaste for that code of mine, even just for a
> benchmark. So it seems the D community doesn't like to use lazy
> arguments to create control structures. - Andrei has shown so much
> distate for such things that the Phobos2 doesn't ususually even use
> normal delegates, and you have even added a "typeless" way to give a
> delegate to a template in D2. This shows there's little interest
> among D developers to go the way of Scala. Scala uses delegates for
> those purposes, and then inlines them.

D supports many styles of programming. Whether a particular style is popular or not is not really the issue - if someone wants to use that style, it is available. If someone builds a compelling case for the advantages of such a style, it can catch on.

We can see this in C++ over time, where different styles have gained ascendancy and then were replaced by new styles.


>> I've been hearing that (about Java, same problem) for as long as
>> Java has been around. It might get there yet, but that won't be in
>> the near future.<
> 
> Today Java is very fast, especially for very OOP-style code.
> Sometimes programs in C++ can be a little faster, but generally no
> more than 2 times. C# on dotnet too is fast, for example its GC and
> associative arrays are much faster.

A factor of 2 is a big deal.


>> Yes, but an interpreter or JIT is required to make that work. That
>> makes the language binary not lightweight.<
> D can be improved/debugged in several ways in this regard, even if
> keeps not using a VM.

Sure, but there's still a huge difference.


>> Do you mean knowing a class or virtual method has no descendants?
>> Sure, you need to know the whole program to do that, or just
>> declare it as final.<
> 
> I can see there's lot of confusion about such matters. Experience has
> shown that class-hierarchy-based optimization isn't much effective,
> because in most practical programs lot of virtual calls are bi- or
> multi- morphic. Other strategies like "Type feedback" work better. I
> have already discussed this topic a little, but it was on the D.learn
> newsgroup, so you have missed it.
> 
> A good old paper about this topic: "Eliminating Virtual Function
> Calls in C++ Programs" (1996), by Gerald Aigner, Urs Hölzle: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.7.7766

Thanks for the reference. DMC++ does a good job of eliminating many virtual calls.

> There are other papers written on this topic, and some of such papers
> are more modern/updated too (this is about 13 years old), but it's a
> good starting point. If yoy want more papers please ask.
> 
> Note that the LDC compiler has Link-Time Optimization too, and LTO
> can also be done when you "know the whole program". If the front-end
> gives some semantic annotations to LDC, it can do powerful things
> during LTO.

I didn't know that.

bearophile Wrote:

> Walter Bright:
> 
> >Actually, you can do them with "lazy" function arguments. There was an example somewhere of doing control structures with it.<
> 
> There are some problems with this:
> - Are current (especially LDC) compilers able to inline those lazy delegates? Scala compiler contains some parts to do that at compile-time (so it's not done at runtime by the JavaVM).
> - I have put inside my dlibs a select() (adapting code written by another person) that uses lazy arguments to implement an eager (it can't be lazy, unfortunately) array comphrension. I'ver tried to see how the LDC compiles it and people there have shown me distaste for that code of mine, even just for a benchmark. So it seems the D community doesn't like to use lazy arguments to create control structures.
> - Andrei has shown so much distate for such things that the Phobos2 doesn't ususually even use normal delegates, and you have even added a "typeless" way to give a delegate to a template in D2. This shows there's little interest among D developers to go the way of Scala. Scala uses delegates for those purposes, and then inlines them.

The call-by-name trailing block syntax is also more uniform with the built-ins in scala, e.g.:

scala> object Helpers { def run_twice[T](block: => T) = { block; block } }
defined module Helpers

scala> import Helpers._
import Helpers._

scala> var i = 1
i: Int = 1

scala> run_twice { i += 1; i }
res5: Int = 3

scala> run_twice { print("foo\n") }
foo
foo


> >I've been hearing that (about Java, same problem) for as long as Java has been around. It might get there yet, but that won't be in the near future.<
> 
> Today Java is very fast, especially for very OOP-style code. Sometimes programs in C++ can be a little faster, but generally no more than 2 times. C# on dotnet too is fast, for example its GC and associative arrays are much faster.

Java is /fast enough/ for many (if not most) purposes. People even use languages with slower implementations like php or javascript these days (ever heard of web 2.0?). Often the other aspects of the language have a larger impact on real world projects than raw execution speed.


> >[array literal type inference] How should it be done?<
> 
> Silently dropping information is bad. So cutting strings according to the length of the first one as in D1 is bad.

Since now I've mostly been experimenting with D1. But this seems to be fixed in D2.


> The type of an array literal has to be determined by the type specified by the programmer on the right. If such annotation is absent (because there's an auto, or because the array is inside an expression) the type has to be the most tight able to represent all the types contained in the array literal (or raise an error if no one can be found).
> By default array literals have to produce dynamic arrays, unless the programmers specifies that he/she/shi wants a fixed-size one.

The literals should also handle cases like [[],[],[1,2]], not only the string case. What makes this frustating is that one can assume that since the generic type inference engine isn't used consistently in all cases, there isn't one in dmd. This might result in more corner cases in other parts of the language.

Walter Bright wrote:
>>>> Especially the array literal type inference is really naive.
>>> How should it be done?
>>
>> You shouldn't use the type of the first given element when constructing the type of the array. If you have [ e_1, ..., e_n ], the type of the literal is
>> unify(type_of_e_1, ..., type_of_e_n) + "[]". For instance:
>>
>> => typeof([ [], [1,2,3] ])
>> => unify( typeof([]), typeof([1,2,3]) ) + "[]"
>> => unify( "a[]", unify(typeof(1),typeof(2),typeof(3)) + "[]" ) + "[]"
>> => unify( "a[]", unify("int","int","int") + "[]" ) + "[]"
>> => unify( "a[]", "int" + "[]" ) + "[]"
>> => unify( "a[]", "int[]" ) + "[]"   // a is a local type var, subst = { a -> int }
>> => "int[]" + "[]"
>> => "int[][]"
> 
> Ok.

Walter: I told you :o).

I'd already defined the unify meta-function, it's called CommonType, see

http://www.digitalmars.com/d/2.0/phobos/std_traits.html#CommonType

You pass any number of types and it will figure out the type that all types can be converted to. Using that, it is trivial to define functions that infer array types properly. At a point I had an array() function that took any number of arguments and returned a correctly-typed array. Then I dropped that and defined array() in std with a different meaning (transform a range into an array).


Andrei

On Tue, Aug 25, 2009 at 11:47 AM, Andrei Alexandrescu<SeeWebsiteForEmail@erdani.org> wrote:
> Walter Bright wrote:
>>>>>
>>>>> Especially the array literal type inference is really naive.
>>>>
>>>> How should it be done?
>>>
>>> You shouldn't use the type of the first given element when constructing
>>> the type of the array. If you have [ e_1, ..., e_n ], the type of the
>>> literal is
>>> unify(type_of_e_1, ..., type_of_e_n) + "[]". For instance:
>>>
>>> => typeof([ [], [1,2,3] ])
>>> => unify( typeof([]), typeof([1,2,3]) ) + "[]"
>>> => unify( "a[]", unify(typeof(1),typeof(2),typeof(3)) + "[]" ) + "[]"
>>> => unify( "a[]", unify("int","int","int") + "[]" ) + "[]"
>>> => unify( "a[]", "int" + "[]" ) + "[]"
>>> => unify( "a[]", "int[]" ) + "[]"   // a is a local type var, subst = { a
>>> -> int }
>>> => "int[]" + "[]"
>>> => "int[][]"
>>
>> Ok.
>
> Walter: I told you :o).
>
> I'd already defined the unify meta-function, it's called CommonType, see
>
> http://www.digitalmars.com/d/2.0/phobos/std_traits.html#CommonType
>
> You pass any number of types and it will figure out the type that all types can be converted to. Using that, it is trivial to define functions that infer array types properly. At a point I had an array() function that took any number of arguments and returned a correctly-typed array. Then I dropped that and defined array() in std with a different meaning (transform a range into an array).

That's also the rule used by NumPy.  I found D's array behavior very surprising.

--bb

Walter Bright:

>No, but it's not a technical issue, just more of no demand for it. But this would not be a reason to use Scala, as one doesn't use Scala for performance oriented programs.<

The usage of higher-order functions is pervasive in Scala, and they use delegates. So optimiizing those delegates was quite high on the priority list of optimization to perform.
In functional-style programming people use first-class functions all the time, so they have to be optimized. That's why D2 may need to do the same if it wants people to use functional-style programming :-)


>DMC++ does a good job of eliminating many virtual calls.<

I have not found much of such capability in DMD. If DMD too performs that, then can you show me 1 example when it's performed? :-)


>[LTO in LDC] I didn't know that.<

Note: LTO isn't done automatically yet by LDC, so to do it you have to input 3 different commands. Ask (to me, for example) if you want to know them.

Bye,
bearophile

Op Wed, 26 Aug 2009 02:04:18 +0200 schreef bearophile <bearophileHUGS@lycos.com>:

> Note: LTO isn't done automatically yet by LDC, so to do it you have to input 3 different commands.
> Ask (to me, for example) if you want to know them.

Well I do :-)

bearophile wrote:
> Walter Bright:
> 
>> No, but it's not a technical issue, just more of no demand for it.
>> But this would not be a reason to use Scala, as one doesn't use
>> Scala for performance oriented programs.<
> 
> The usage of higher-order functions is pervasive in Scala, and they
> use delegates. So optimiizing those delegates was quite high on the
> priority list of optimization to perform. In functional-style
> programming people use first-class functions all the time, so they
> have to be optimized. That's why D2 may need to do the same if it
> wants people to use functional-style programming :-)

There's quite a lot of D specific optimizations that are not done at the moment, simply because of other more pressing issues.


>> DMC++ does a good job of eliminating many virtual calls.<
> 
> I have not found much of such capability in DMD. If DMD too performs
> that, then can you show me 1 example when it's performed? :-)

DMC++, not DMD. I haven't done the effort yet to do it in DMD.