January 19, 2012
On 2012-01-19 11:11, Walter Bright wrote:
> On 1/19/2012 2:06 AM, Patrick Stewart wrote:
>> Long story short - I find new things added and premature optimizations
>> The
>> worst enemy of language at the moment. They might look like selling
>> point to
>> you, to me they look like distractions from fixing D's shaking legs and
>> solving some real problems underneath.
>
> Take a look at the D changelog.
>
> https://github.com/D-Programming-Language/d-programming-language.org/blob/master/changelog.dd
>
>
> I just don't see how it can be argued that we aren't doing exactly what
> you suggest we do.

Take a look at the latest commits, the language is not frozen. You have recently added support for SIMD and a new lambda syntax (which I really like BTW).

-- 
/Jacob Carlborg
January 19, 2012
On 1/19/12 3:25 AM, Manu wrote:
> On 16 January 2012 02:23, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org <mailto:SeeWebsiteForEmail@erdani.org>>
> wrote:
>
>     I do have ties with the gaming community; I taught a course at ENDI
>     and I am well acquainted with a few game developers. Also, at
>     conferences and events gaming programmers are represented. Finally,
>     game developers who are reading TDPL are likely to send me book
>     feedback and questions in proportion to their representation. From
>     where I stand, I can say there is more interest in D in other
>     communities than in gaming.
>
>
> I'd just like to add one more point, that gamedev is, in general,
> windows-centric. And D's support for windows is basically disgraceful.
[snip]
> You probably /won't/ hear from the gamedevs either until they are
> able to do anything more than dismiss it within 5 minutes.

This notion is akin to my "if we considered gamers a core market, we would have done a lot of things differently".

[snip]
> You lead me to an interesting though though...
> I have been realising one thing that is concerning me more and more
> though, and that is that for some reason, it always comes back to
> Walter. This seems absurd.
> Why should any feature be contingent on Walters time/excitement for the
> feature? He should be allowed to work on the SIMD if he wants, and it
> shouldn't significantly affect anyone. Is he being paid? (I don't
> actually know)

Walter is working full time on D. I and most others are working on it part-time. (I'm not counting people who work _with_ D.)

> I was initially under the impression that D was an open source language,
> and there were a significant number of contributors... if Walter were
> hit by a bus, what would happen to D?

That hasn't been an issue for a while now. There are several other people who know and understand the language, dmd compiler, and the standard library very well. But as of now Walter is the gatekeeper of fixes and features for the simple reason that he wants to still understand what's going on. This is a common approach. He's also the foremost decision maker for what major features go into the language.

> This point is highlighted by your anger/frustration that Walter worked
> on the SIMD stuff out of step. If you're the second(?) most authoritive
> voice around here, what does that say about the state of the language
> development and the team responsible?

To me it says we need to improve our organization.


Andrei
January 19, 2012
On Thursday, January 19, 2012 14:59:54 Jacob Carlborg wrote:
> On 2012-01-19 11:11, Walter Bright wrote:
> > On 1/19/2012 2:06 AM, Patrick Stewart wrote:
> >> Long story short - I find new things added and premature optimizations
> >> The
> >> worst enemy of language at the moment. They might look like selling
> >> point to
> >> you, to me they look like distractions from fixing D's shaking legs
> >> and
> >> solving some real problems underneath.
> > 
> > Take a look at the D changelog.
> > 
> > https://github.com/D-Programming-Language/d-programming-language.org/blo b/master/changelog.dd
> > 
> > 
> > I just don't see how it can be argued that we aren't doing exactly what you suggest we do.
> 
> Take a look at the latest commits, the language is not frozen. You have recently added support for SIMD and a new lambda syntax (which I really like BTW).

The language isn't frozen, but it's close. Almost all of the language changes at this point are backwards compatible. And I don't think that it necessarily ever makes sense to say that we won't make backwards compatible changes anymore. The issue is that we need to try and resolve any further issues that require non-backwards compatible changes sooner rather than later and that we need to get the compiler stabilized to the point that it can be reasonably used for fulltime development without fear of compiler bugs making that infeasible.

We're getting there, and aside from tangents such as SIMD, the only thing that I can think of that could be done better (aside from somehow fixing bugs faster) is to focus more on some of the more critical bugs - such as TDPL bugs and major bugs which prevent major features working properly (such as const- related issues). But that's been improving as well of late, if perhaps not always as quickly as would be nice.

- Jonathan M Davis
January 19, 2012

> On 1/18/2012 2:15 PM, Adam Wilson wrote:
> > [...] There are 2719 open issues in the
> > bugtracker; that number alone will scare off many potential users.

Walter Bright Wrote:
> Take a look at the changelog. I just don't see how anyone could conclude that is not exactly what we are doing. Here's the current list for the upcoming version of D2:
> 
> 
> 314 - [module] Static, renamed, and selective imports are always public
> [...]
> 7309 - [2.058] Regression caused by new inlining code



Hello!

I agree the changelog is impressive and a solid proof the D community is hard working.

However, I still subscribe to Adam's view that the D bug situation _is_ a problem, even a scary one: http://d.puremagic.com/issues/reports.cgi?product=D&datasets=NEW

That's over an ~8 year time frame. I think it's perfectly clear: D has a chronic monthly bug surplus.

Of course, all software has bugs. The difference is healthy software has a "rhythm", and as far as I can tell, D doesn't. Compare with a text-book example (not exactly comparable, but close enough):
http://bugs.debian.org/release-critical/

The key here, I believe, is the psychological effects of bundling high-quality stable software with buggy components. Although the curve above doesn't mean all of D is "unstable", as features of course do get stable in time, it does mean the perception of D is bound to be mostly negative. Why? If you buy a car and it has a solid engine, good wheels, wonderful suspension, but the seats are uncomfortable and the radio switches stations randomly, you're not going to think "Ah, this is an overall nice car, because the awesome engine makes up for the bad seats and radio", no, you're going to get annoyed and think "this is a crap car that makes me irritated and hurts my back" and you'll wish you bought another one, even one without a radio. With D2, because it mixes new features and bug fixes into the same branch, there's always something that's the annoying radio.

This is a perfectly good reason to add a lib namespace like "ext.*", "lab.*" or even "beta.*" and add all new libs there with the warning that it's experimental and not for production.

Then again, that's not really a solution. The latest SIMD lib is a perfect example: It seems contained and like it could only do good for D's overall quality. Yet it doesn't take long before someone realizes SIMD could perhaps speedup regexp, then someone wants all float[4] to be optimized automatically, of course std.math must be picked apart and reassembled with SIMD in place sooner or later, then hashing, then crc, utf-8 tricks, who knows? The next tier is that 3rd party libs will start using the half-baked SIMD lib too, because they follow D's sloppy lead. And voilá, we have a bug soufflé spreading to all D code, and the end-user doesn't have a choice. There's no such thing as an isolated feature. Every new feature comes attached with a systemic cost. An alternative way of thinking about SIMD is that, sure, it will speed up some things, it will also add around 300 bugs over the next year to arbitrary parts of D, as a measure of the effort needed to sustain D's current quality level, with SIMD in place. And that would be ok, _if_ we could handle it, but not if it means an overall bug surplus, and especially not if there already is a surplus that it just adds to. Then D's quality will deteriorate from it.

I think D really needs to start thinking economically about new features and bugs. New features must be picked carefully and strategically with a maintenance budget in mind. This economy needs to be balanced, with the means available here and now, not with some pie-in-the-sky idea that if we only get this feature we'll get 50 new users and then we'll fix all bugs in a month. That behavior will burn already existing users once the prediction fails, as it must, from time to time. New features are costly, over a long period of time, and need to be treated as such. New users typically do not evaluate software on the basis on how good it could get if they fixed all the bugs first.

Then, I get the impression D is acting somewhat out of desperation, understandably. At some level I see a talented and skillful community, with an innovative and beautiful product, waiting for a lucky break. This can be a frustrating situation. There seems to be a belief, subconsciously perhaps, that if D would only have enough "wow" people would finally get impressed enough and start to use it. I think this is a fundamentally flawed belief that leads to a slow death sided by wasted efforts. I think when we all think about it, we chose software, when we can chose, that works well over software that's cool but annoying and frustrating. I mean look at other successful low-budget software projects. Most got their break by doing a particular thing really well, not by being impressive on paper or to an audience at a conference. D needs to stop "wowing" and start focusing on becoming a focused high-quality product, and given the psychology of humans and the things I've mentioned, I see no other way of doing that except getting that bug curve back on the ground, and then make an actual stable release. And getting a bug curve down can only mean one thing: Freeze all new features.

Of course, halting development alltogether isn't good. Software projects needs to be actively developed, creativity needs to be kept warm, or it dries up. But this is a universal problem, not something unique to D, and it has already been solved, many times over. The conventional industry-standard way of releasing stable high-quality software, while still being able to work on the next cool version, is to use stable/testing branches. This is no mystery. The question is: Why doesn't D have this? If the answer is that D1 is the stable branch, then the problem is a different one, but equally troublesome: The stable release cycle of D, that is, the time the end-user have to wait for the next batch of stable features, is what, 6 to 9 years? No one will wait around for that long. D needs a new stable release every 1–2 years, especially in the beginning when new features are available in abundance. No one cares about how great D will be when all envisioned features are finally in place. People care about what's here and now.

Summary
D can not liftoff with this monthly bug surplus. The surplus is in one way good, because it shows there's creativity happening, but it needs to be confined in a separate branch, out of the sight of the end-users. The version of D that the end-users see needs to have a bug deficit, asap. Further, D needs to start to think economically about features and bugs, and there probably should be a policy in place that no new stable version is allowed to be released before the current one has fixed 90% of its bugs, or something like that. I think D is long due on putting the two activities of creative development and maintenance on two separate planets, or moons, in this case, I suppose.

Just to be clear, I don't think SIMD should never have happened. Creativity must be allowed and creativity comes when it comes, but it shouldn't be triumphantly dumped in the laps of the end-user every month like a dead mouse the cat brought in. If D was an airline, it would be like you've just boarded the airplane and you're waiting for takeoff, but then the pilot comes on: "Dear passengers, we're just going to make a routine break here. Our engineers have just come up with a new better wing flap and we'll just wait 20 minutes while they install it". Huh? Is this a joke? you'd be thinking. Then you look out the window, and there they are, the engineering team, disassembling part of the wing. I mean it's madness to think the end-user will enjoy using a product while it's in active development, especially when it's such a low-level product as a programming language. It's also madness to think the end-user will wait 7 years for the next stable update. There needs to be a strong separation between development and maintenance, and there needs to be regular timely stable releases with near zero known critical bugs.

Anyway, I think D has great potential and I use it regularly for hobby stuff and like it very much, especially as I'm allergic to C++. (No really.) There's a lot of really good stuff in there.


Kind regards
/HF


(PS. ...oh, I forgot to bring up the documentation, oh my, next time perhaps...)

January 19, 2012
I agree 100% with this, must say, beautifully written essay.
January 19, 2012
On 01/19/2012 06:13 PM, Heywood Floyd wrote:
> However, I still subscribe to Adam's view that the D bug situation _is_ a problem, even a scary one:
> http://d.puremagic.com/issues/reports.cgi?product=D&datasets=NEW
>

More than one fourth of those are enhancement requests, therefore it is somewhat less bad than it looks.
January 19, 2012
Am 19.01.2012, 19:02 Uhr, schrieb Timon Gehr <timon.gehr@gmx.ch>:

> On 01/19/2012 06:13 PM, Heywood Floyd wrote:
>> However, I still subscribe to Adam's view that the D bug situation _is_ a problem, even a scary one:
>> http://d.puremagic.com/issues/reports.cgi?product=D&datasets=NEW
>>
>
> More than one fourth of those are enhancement requests, therefore it is somewhat less bad than it looks.

Well written, Heywood. I liked the cat allegory. I could go on and on thinking about the philosophical aspects of using this "pet". But I also agree with Timon. The perceived situation on my end isn't bad any longer. Those who use D for a longer while have noticed that the annoying bug rate went down to an acceptable level, where you don't want to jump off and use C++. I just opened a bug report for a missing optimization opportunity. I would not have done that a year ago, thinking that there are still enough real bugs to be fixed first.
January 19, 2012
On 1/19/2012 1:25 AM, Manu wrote:
> You lead me to an interesting though though...
> I have been realizing one thing that is concerning me more and more though, and that is that for some reason, it always
> comes back to Walter. This seems absurd.
> Why should any feature be contingent on Walters time/excitement for the feature? He should be allowed to work on the
> SIMD if he wants, and it shouldn't significantly affect anyone.

Focusing on dmd for this email, phobos and druntime are essentially not dependent on Walter in any way:

Very little is truly contingent on Walter (though some parts are, such as optlink).  Currently, for dmd, all the patches flow through him (there's a couple others that do have commit privs but we don't use them).  ANYONE is free to develop changes and submit pull requests.  The problem is that there's few who currently do (around a dozen, and only 3 or 4 that are particularly active).  For changes that don't come from the community, Walter fills in the gaps.  Over the last couple years, the number of contributors to dmd has grown nicely, if slowly.  If someone wrote the code for dmd to emit coff format .obj files and it was demonstrated to work well with the microsoft compiler tool chain, I'll bet my next paycheck that it'd be integrated seriously fast.

I pulled the stats on the number of pull requests we've seen since moving to github yesterday (see the phobos list for the full details), but I'll repeat one from memory.  There's been 615 pulls to dmd of which only 58 are still open. That's 91% handled and 9% left.  A decent number of requests are delt with on a daily basis.

Another interesting tool is github's impact grph:
  https://github.com/D-Programming-Language/dmd/graphs/impact

Anyway, hope that colors things a bit for you.

Later,
Brad
January 19, 2012
On 1/19/12 11:13 AM, Heywood Floyd wrote:
[snip]

Thanks for a very insightful post.

Andrei
January 19, 2012
On 1/19/12 1:30 PM, Brad Roberts wrote:
> Another interesting tool is github's impact grph:
>    https://github.com/D-Programming-Language/dmd/graphs/impact

I was never able to interpret that saliently.

Andrei