On Friday, October 18, 2013 08:57:00 Andrei Alexandrescu wrote:
> > Speaking of which, insisting on using .zip files is another beef I have with Walter. The whole "everyone on Windows is stuck in the 90s" mentality is plain wrong, especially for programmers. 7zip (or Peazip or whatever) should be part of every modern programmer's toolbox.
> 
> I don't think that makes a large difference. Probably the better thing to do is trimming the contents of the archive.

The #1 reason why zip has got to go is symlinks. The shared libraries for Linux in the zip are messed up, because you can't put symlinks in the zip file. We really need to have OS-specific archives with the Linux one being something like .tar.gz or .tar.bz2.

- Jonathan M Davis

On 10/18/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> I don't think that makes a large difference. Probably the better thing to do is trimming the contents of the archive.

Right, but this is more general. He also dislikes non-zip archives in bugzilla attachments.

On 18 Oct 2013 19:45, "Jonathan M Davis" <jmdavisProg@gmx.com> wrote:
>
> On Friday, October 18, 2013 08:57:00 Andrei Alexandrescu wrote:
> > > Speaking of which, insisting on using .zip files is another beef I have with Walter. The whole "everyone on Windows is stuck in the 90s" mentality is plain wrong, especially for programmers. 7zip (or Peazip or whatever) should be part of every modern programmer's toolbox.
> >
> > I don't think that makes a large difference. Probably the better thing to do is trimming the contents of the archive.
>
> The #1 reason why zip has got to go is symlinks. The shared libraries for Linux in the zip are messed up, because you can't put symlinks in the zip
file.
> We really need to have OS-specific archives with the Linux one being
something
> like .tar.gz or .tar.bz2.
>
> - Jonathan M Davis

Quite simply zip is not the correct format. Zip is normally only used to distribute windows specific versions of software. Does it even support executable permissions?

This is a good list of things that we could and should improve. Getting all minded to perfection would be difficult, but definitely worth living into.

I appreciate your enthusiasm for and involvement in D. At the top level a few simple realities need to be understood.

First, there is no OSS community that doesn't have its annoyances. I've been a direct participant to one other and am lurking on the forums of three more. Some are led by juntas of mini-tyrants; some are unnecessarily snooty; some are consumed by intestine fighting, politics, and horse-trading; and so on.

Second, some of your points revolve around the fact that Walter and myself are not as good as we should at certain tasks and roles, such as build master, PR person, manager, and operational officer. The general argument pattern revolves around "I/we told Walter/Andrei several times they need to do X, and they forget/ignore/do it badly." Clearly Walter is a self-confessed mediocre build czar at best. But he's doing it because, although there is no shortage of suggestions on how he should be doing it, nobody has the time and inclination to actually _be_ the build czar (which is entirely understandable).

Within a traditional organization where people are paid to work on a certain project, the solution would be simple and obvious - Walter would be relieved of the roles he doesn't excel in, and left to focus on those he's really good at. Other people would fill in duties and responsibilities such as preparing betas, sending them out for testing, announcing them, etc.

Within our current organizational landscape, where nobody is paid to work on D (not _with_ D) except for myself, addressing issues such as "we should have a better build master" is much more difficult to address.

Third, for what it's worth, here's what I believe are the top issues with D today:

1. Nobody is being paid to work on D (aside from me since recently, and on work that would benefit my employer).

2. D is not backed up solidly by a large engineering organization.

3. We are unable to review and accept contributions at the rate they are arriving.

I tend to ask myself how various proposals for improving our process address these three key points. I believe your list effects mostly (3), albeit not directly.

More answers inline.

> - We do not have any vision or major plans ahead for the language.
> Currently we're stuck in a bug-driven development environment, where
> bugs are arbitrarily picked off of bugzilla and fixed. But there's no
> major plans ahead, e.g. "let's plan to fix these X major bugs for some
> upcoming release". We can't force people to work on X or Y, but if
> they're in a visual place and marked important and scheduled to be
> fixed, this will give an incentive for contributors to work on these
> problems.

Walter and I frequently think of ways to gently steer people toward working on specific items. Votes, categorization, discussions are of limited impact. On occasion we've emailed major contributors directly and asked politely but point blank if they could look into an issue (sometimes something they had an expertise in, or even an issue they had caused). The rate of success has been very low. People still love working on things, just not on the things they're told to.

We've added the "preapproved" tag - take a look: http://d.puremagic.com/issues/buglist.cgi?quicksearch=preapproved. A couple have opened pull requests, which have not yet received reviews yet (which is not my or Walter's fault as much as a larger community issue that we need to fix, see (3) above). Most don't. You yourself didn't find the time to update a task you volunteered on.

That said, it's entirely possible a formula for success does exist. Looking forward to proposals, like improving the visuals of "bugs to be fixed". What I think is obvious is that solution involving more work for Walter and myself are unlikely to work well, for the simple reason we are both impatiently waiting for the sun to rise every day to do more work on D.

> - We do not have any defined release timeline. When is it time to
> start prepping for a release? It's up to Walter's arbitrary decision
> for when this happens, he doesn't even talk to the community or
> contributors on whether it's a good time for a beta phase (maybe
> there's a huge or disruptive new pull request that's planning to be
> merged and a beta should be delayed).

I understand how that can be a bother. Walter figures the time is ripe for a new release when we have enough compelling features and fixes. I'll try to make the appropriate announcements in the future prior to deciding on starting a beta.

> - We do not have a defined timeline for the beta testing period. How
> long before we decide that the beta has been tested for long enough
> before a release is made? Again, it's up to Walter's decision.

We are generally aiming for two weeks, but it sometimes gets longer because of the impending regressions. One good phenomenon has been that we've got an increasing number of people who are testing the beta.

> Having a defined release cycle and a beta testing period will
> ultimately be beneficial for everyone. People who are busy can
> rearrange their schedule to make free time and ensure they have enough
> time to test a beta compiler on their projects, and contributors to D
> can prepare for a cycle of days or weeks where they can rapidly work
> on reducing regressions and polish everything up for the release (e.g.
> writing an elaborate changelog).

I also believe a better cycle would be beneficial, but I don't think it would address our core issues.

> - We do not have a proper release system. Nick Sabalausky has been
> working hard on one[1], but Walter seems to have completely ignored
> it. It would have been a terrific opportunity to try and make it work
> for this release. What better way to test it than to test it on a
> release?

We'll look into it, but this harkens back to the simple dynamics mentioned above. That is essentially a request for Walter to change the way he does releases, and people tend to get set in their ways and have difficulty finding the time to stop and change things when there are so many other things vying for their attention. The best solution here is if Nick (or someone else) would _be_ the build manager using those great tools that Nick wrote.

Anyhow, I'll look into that.

> - The betas are still not visible. The homepage makes no notes on a
> new beta being available, the download page does not list the betas
> either, and AFAICT there's no RSS feed either. The social media groups
> are not notified either (neither Andrei's nor Walter's twitter feed
> make a mention of a new beta being out, the same applies to
> https://twitter.com/D_Programming or the #dlang hash tags). To top it
> all off, you cannot post to the dmd-beta newsgroups from the D Forums,
> you have to separately register to this mailing list.

As a simple start, did you consider announcing the beta yourself? Anyone can tweet #dlang @D_programming, and in all likelihood we and many others would retweet. Also, did you consider submitting a pull request for placing the beta announcement on the homepage?

> If we want user feedback on betas we absolutely must make the betas
> visible and give an opportunity for people to post feedback.

I agree. "We" is the right word.

> - Walter is still not tagging the beta releases by the file name (it's
> always dmd2beta.zip). I've complained about this several times and
> IIRC someone else did as well at dconf (maybe I'm remembering wrong
> though). They should at least be named as "dmd2_064_beta1.zip",
> "dmd2_064_beta2.zip". And all of them should always be available for
> download (including visibility on the download page), so people who do
> not use Git or build manually from master can quicky check whether a
> regression was introduced in a specific beta version.

Probably that's a good thing to do, and easy enough. I don't think it would push the needle significantly.

> - I still sigh when I hear about Walter and Andrei having private
> phone conversations, or any kind of decision is made behind-the-scenes
> rather than publicly where the community has a say in it. Walter's
> implementation of UDAs directly in master which lead to having a
> deprecated syntax even though nobody used this syntax is what comes to
> mind.

I understand this is well intended but it's the kind of remark that bends me out of shape. First, there's no substitute for real communication than direct conversation. We can't have a phone conversation with the community. Then, your first three points complain about a lack of leadership, and in the same breath you complain about there being too much of it.

We believe we've done well in making this community a meritocracy where competent contributors can make a difference and make their word heard. To the extent we're doing it insufficiently, it's because too few people assume the responsibility to review and accept contributions.

Walter scrambled to implement UDAs in a rush and breaking protocol in order to win a corporate D user, Remedy Games. It was a major, exceptional event. Would you have preferred the protocol to have been followed at the cost of Remedy?

> - Both Walter and Andrei not following the rules and making novice
> mistakes w.r.t Git and Github. Walter still seems to struggle with
> basic usage of git, where people continually have to re-explain what's
> wrong and how to fix an issue. I'm sorry, but if someone bought the
> Git book years ago and is still struggling with *concepts* then no
> amount of hand-guiding is going to help.

I agree that's a problem. Probably what would help would be more quality reviews and pulls by our members with commit rights, of whom you are one.

> And Andrei doing things like
> merging a dozen pull requests at once, with complete disregard to the
> fact that merging to master means other pulls could easily break (and
> so master can be broken). You cannot make so many merges unless you're
> absolutely sure each pull request does not interfere with another.

If a pull request is sensible and meaningful it shall be pulled. That's the way we do things at Facebook, too, and it scales. There may be pulls that are so closely related they break the build when put in conjunction, but I believe that case is rare.

I also want to convey a bit of perspective here. There are 221 open pull requests right now. These are 221 instances of good work put in by talented people, who associated hopes and wishes to improve things with that work. This bothers me. I lose sleep at night because of it. And the fact that those contributions don't get looked at is our entire community's fault as it is mine.

So when I have a few free minutes, I try to take a look at the extant pull requests - really, any would do. I try to pull in those that are meaningful and uncontroversial. I agree I could probably spend more time on carefully analyzing interactions between pulls, but that's time I can't afford.

> - Back to Walter, a few weeks ago he merged a pull request over night,
> without regard to the pull request not being fully tested by the test
> machines. The result? Master was broken **for the entire next day**.
> Nobody knew which commit broke master, so nothing was done until
> Walter came back to Github the next day and started reverting his
> pulls. In the meantime the entire day was wasted because nobody's pull
> request could get green. Luckily it was a sunday, so there weren't too
> many complaints. But I could have easily merged a few pulls that day
> (as it happens I like to do things on a sunday), and as a result we
> would have a smaller pull queue.

I guess such accidents are bound to happen to the best of us. A build czar with the appropriate skills would have swiftly undone the damage. But there is no build czar.

> There's just many things that are going plain wrong here, and a lot of
> promises are always made but ultimately never delivered (whether it's
> about breaking changes or an improved development process -- again
> think about scheduling bug fixes for the future, we could help people
> prepare for the breaking changes).

Whatever we don't get delivered, it's not for the lack of trying. Whatever we do, it doesn't come easily.

> Personally I find D to be a huge part of me right now (probably not
> the other way around, I'm a small part of this compared to the huge
> contributors), and I feel really bummed out when both Andrei and
> Walter take a casual stance when things are broken (whether it's the
> system itself or something specific like master being broken). I
> really think we have a great thing going here with the language, but
> everything else has to improve, and particularly the development and
> release process.
>
> So that's what I'm protesting about.

The individual points are salient, but I fail to grab the most significant bit. The logic doesn't follow.

We're in this together; to wit, for many of the things you're asking why they don't get done, you could be asked the same thing.

You did a great job on formatting the changelog. It has been an unqualified success, it was amazing marketing for D, and it has been thoroughly appreciated by everyone. It is great work that we need more of. Now you're not doing that work in protest against... not enough work getting done.


Andrei

On 10/18/2013 12:33 AM, deadalnix wrote:
> I highly doubt that this fit into cases 1 to 4 as you mention. I'll however
> double check with that in mind.

I want to know about any other cases, so please investigate.


> That also doesn't explain why I get a closure bug (frame pointer or frame
> content is corrupted) when compiling with unittest. The code that fail isn't
> even unittested !

Right, that sounds like some thing else.

On 10/18/2013 11:17 AM, Rory McGuire wrote:
> Does it even support executable permissions?

Yes, it does.

On 18 Oct 2013 21:00, "Walter Bright" <newshound2@digitalmars.com> wrote:
>
> On 10/18/2013 11:17 AM, Rory McGuire wrote:
>>
>> Does it even support executable permissions?
>
>
> Yes, it does.
>
Nice. It's there any particular reason you prefer zip?
I guess it's irrelevant what the current format is if we start using Nick's
builder.

> I get bitten by it locally too: if there's a test with an inaccurate sql query with time formatted as string, the sql server doesn't always compute it, because the default "human-readable" time format is taken from the current locale, and to parse it one first has to guess the format itself, which is quite difficult in a heterogeneous system.

DATE '2013-10-12' is the date October 12th to most SQL parsers.  Locales and NLS in SQL have bitten me many times until I learned this.

On 10/18/2013 12:14 PM, Rory McGuire wrote:
> Nice. It's there any particular reason you prefer zip?

It's easy and works on all platforms.

I also point out that, for all platforms supported, when we do a release we also build a custom download package for each platform in that platform's preferred format. If these are inadequate, then bug reports need to be filed and pull requests issued for the package building scripts.

I expect that managing all this should be the responsibility of the Build Czar, which is an open position.


> I guess it's irrelevant what the current format is if we start using Nick's
> builder.

What I prefer is that all the packages are built automatically and daily by Brad's autotester, and that this process is controlled by scripts checked into github and that anyone who wishes to improve it can issue pull requests against it, and the Build Czar makes sure the process is working.

On 10/18/13, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> nobody has the time and inclination to actually _be_ the build czar (which is entirely understandable).

Building something should be a command on the command-line. The whole issue is about currently having a person in place of a tool.

> You yourself didn't find the time to update a task you volunteered on.

You mean https://github.com/D-Programming-Language/dmd/pull/2235 ? I'll get back to it soon.

> We'll look into it, but this harkens back to the simple dynamics mentioned above. That is essentially a request for Walter to change the way he does releases, and people tend to get set in their ways and have difficulty finding the time to stop and change things when there are so many other things vying for their attention. The best solution here is if Nick (or someone else) would _be_ the build manager using those great tools that Nick wrote.

Like I said before, it's a command, you don't need a manager to do it, you need a reliable tool. "$ make_build" is all it takes. This "manager" and "czar" nonsense is really a corporate way of solving issues. I can see how you might have gotten used to that when you have people on a payroll that you can easily asign to a task. Even if ultimately it's beneficial for having a tool in place of a person, a company might decide against it if it can "hammer the nail right away and make the problem go away" (until the problem comes walking back). We're programmers, we should rely on tools for automated tasks.

And I want this not because I'm an automation freak (I'm not), but precisely because Walter has screwed up the zip package many times before. Hence, if Walter is the one who is currently in control, why doesn't he interact with Nick to ease into using a build tool rather than use some X number of scripts that he keeps on his PC only?

> As a simple start, did you consider announcing the beta yourself?

I didn't, because I disliked the current model and tied my hands when I saw the new beta was out due to the sum of frustrations which I've listed. This will change if the situation improves, of course.

> Anyone can tweet #dlang @D_programming, and in all likelihood we and many others would retweet.

Good. I'll certainly start to build a better online presence, including tweeting, and I should start blogging too (there's plenty to blog about w.r.t. D). I can't be the pot calling the kettle black.

> Also, did you consider submitting a pull request
> for placing the beta announcement on the homepage?

Good idea, and it's something I can work on.

But I hope you understand this isn't really about you or Walter not doing these things yourself, but rather the fact that you didn't seem to recognize this as being a problem. I've mentioned the build/release issue many times, and now we finally have a pull request that could make this problem go away. Just as I thought the problem was being resolved, Walter announced the new beta. So clearly he still doesn't see the pull request as being important.

> Probably that's a good thing to do, and easy enough. I don't think it would push the needle significantly.

But that's just the thing. The things I've listed are what pushed the needled "too much to the left" for me. Small improvements like this are great, and they add up. Otherwise people (like me) will keep complaining about small issues, which collect and add up to the frustration.

> We can't have a phone conversation with the community.

Why do you even need these phone conversations related to D or decision making? The community has an insane number of intelligent people who can contribute to resolving issues.

> Then, your first three points complain
> about a lack of leadership, and in the same breath you complain about
> there being too much of it.

Lack of leading the community and the development process, not lack of decision making.

> Walter scrambled to implement UDAs in a rush and breaking protocol in order to win a corporate D user, Remedy Games. It was a major, exceptional event. Would you have preferred the protocol to have been followed at the cost of Remedy?

This is what doesn't inspire me at all. So in the future when company X decides it wants some feature in the language you're saying we should not follow the protocol like we do for all the other user-requested features? Because the news of company X using D will create headlines?

> I agree that's a problem. Probably what would help would be more quality reviews and pulls by our members with commit rights, of whom you are one.

I don't see what this has to do with not understanding Git basics. If I merge more pull requests it isn't going to improve Walter's knowledge of Git. Also, I tend to review pull requests which I actually understand (doing otherwise would be counter-productive). Sometimes I fall behind track because I also want to work on some of my own D projects. I know we're all very busy, but I didn't say Walter didn't want to review something, I said in particular that Walter is still struggling with Git fundamentals.

> I agree I could probably spend more time
> on carefully analyzing interactions between pulls, but that's time I
> can't afford.

Well that's a damn shame. You see, most of us who have commit rights wait for the pull request questions to be answered, for commits to be properly reviewed, and for any final bookkeping and polishing to be done, and then we wait for the lights to go green before we pull.

> I guess such accidents are bound to happen to the best of us. A build czar with the appropriate skills would have swiftly undone the damage. But there is no build czar.

Here we go again with this corporate speak. Now you want to add another person to the mix, when we already have the tooling in place. Don't merge a pull request unless it's green, how can that be so hard to understand?

> We're in this together; to wit, for many of the things you're asking why
> they don't get done, you could be asked the same thing.
> Now you're not doing that work in protest against... not enough work
> getting done.

I see that most of your arguments ultimately revolve around throwing the ball to my side, and asking why I didn't do all the things I listed. That's completely fair game (it is), but I still can't control when we do a beta/release cycle, and I can't control zipping up a release package (and I don't want to, I just want this to become a small task of issuing a command on the command line). Now we have a shot at the second problem with automatic tooling to make the release package. Is Walter going to be ok with this?

Btw, if you think that I'm just going to put my tail between my legs and run, you're out of your mind. I'll deliver on the promises I made (which includes the part about twitter/blogging/etc), and I'll make the 2.064 changelog, and probably future ones as well. I can bite the bullet and do hard work for D.

But, if you're going to keep saying things like:

- Damn the protocol, don't you see feature X for Company Y is
important? We'll implement it regardless of community.
- Damn the protocol, the pull request list is long and I'll just do
whatever I want to reduce it, even if it means having a broken master.

Then you cannot expect me to be silent about it. This little protest of mine was an attention grabber to the issues that we face, and especially to some issues that I'm personally frustated about. You can consider the protest over.

As for me, I'll do my part to improve whatever I can with regards to D. I love D and the community too much to be stopped from contributing more just out of a few frustrations.