December 20, 2012
On Wednesday, 19 December 2012 at 23:05:59 UTC, deadalnix wrote:

> master : used as a base for development. New feature are merged here.
> staging : used to provide a view of what the next version will look like. Regular snapshot of that branch are made so public can use the last features.
> version : used to contain a version that will have a support for an extended period of time.

I do not see how development in master is moved to staging based on this description. I'll try and be more specific.

I realize you don't like features, but we need to talk about Phobos additions, @property mechanics, and other highly disruptive bugs. I'll call it the major-change.

The major-change branch is generally developed to completion and may just be a pull requests from developer342's master.

From the sound of it this request is pulled into master. We continue to pull many of these changes in. How do we decide they should be placed into staging, when we pull them into master?. If we wait for some 'magic time' how do we pull it into staging, does that mean we now cherry pick commits of master?

Another issue is it sounds like master becomes a "phantom" branch. At no point in time would master resemble what is released. I see this as a problem because it is the branch people are developing off of, it means nothing to test in master as staging has the actual state that will be released.
December 20, 2012
On Thursday, 20 December 2012 at 04:11:00 UTC, Jesse Phillips wrote:
> On Wednesday, 19 December 2012 at 23:05:59 UTC, deadalnix wrote:
>
>> master : used as a base for development. New feature are merged here.
>> staging : used to provide a view of what the next version will look like. Regular snapshot of that branch are made so public can use the last features.
>> version : used to contain a version that will have a support for an extended period of time.
>
> I do not see how development in master is moved to staging based on this description. I'll try and be more specific.
>
> I realize you don't like features, but we need to talk about Phobos additions, @property mechanics, and other highly disruptive bugs. I'll call it the major-change.
>
> The major-change branch is generally developed to completion and may just be a pull requests from developer342's master.
>
> From the sound of it this request is pulled into master. We continue to pull many of these changes in. How do we decide they should be placed into staging, when we pull them into master?. If we wait for some 'magic time' how do we pull it into staging, does that mean we now cherry pick commits of master?
>
> Another issue is it sounds like master becomes a "phantom" branch. At no point in time would master resemble what is released. I see this as a problem because it is the branch people are developing off of, it means nothing to test in master as staging has the actual state that will be released.

That is very true. With the given modification master and stagging become redundant with one another. Why not get rid of stagging ?
December 20, 2012
On Thursday, 20 December 2012 at 04:11:00 UTC, Jesse Phillips wrote:
>
> From the sound of it this request is pulled into master. We continue to pull many of these changes in. How do we decide they should be placed into staging, when we pull them into master?. If we wait for some 'magic time' how do we pull it into staging, does that mean we now cherry pick commits of master?
>
> Another issue is it sounds like master becomes a "phantom" branch. At no point in time would master resemble what is released. I see this as a problem because it is the branch people are developing off of, it means nothing to test in master as staging has the actual state that will be released.

Most of the D users are not D devs, and they don't want to be testing highly unstable features, however some of them will want to use reasonably stable new features, even though there may be some bugs and there may be some changes. The reason for using the testing branch at all, is that they need something in it that's not available in the stable branch, so they may take a smallish risk and try out the testing or staging branch (need to decide on the terms here).

I placed a note in the wiki about the issue of making sure there are incentives for people to do the things you wish them to do for each major step in the process. If there is no incentives for people to test what comes out of development, few people  will test it, but you need plenty of testers. The carrot is that a staging tester gets a snapshot of what is most current that is also reasonably stable, ie., there has to be enough confidence that the code is stable enough to use for real, else the risk will be too high to bother with it.

I know this because I'm one of those people. I pick and choose stable vs
"reasonably" stable based on my needs and the risk assessment. You need to be quick about supporting the testing branch and careful about not dumping crap into it, it's a delicate balance yes, but you *need* testers, and there's only one good way to get them, you have to dangle a carrot out and hope they take the bait, and then you have to be prepared to help them out when things go wrong.

I expect very few people will take a chance on the dev branch other than the devs themselves and a few of the experienced people who know their way around the compiler, that's just the way things will be because it'll be too risky for the regular folks to bother with.

The user count will always be lowest for the dev branch, more for staging but you want to encourage highest use possible which means quick support, and the highest use will be for stable, but because it's stable means it's not really being tested because it should be mostly bug free, i.e., it's being "used" as opposed to being "tested".

I don't think you can afford to drop the testing branch unless you think there's no need for very many Average Joe testers, but those are the guys who will uncover a ton of unexpected bugs missed by the dev testers.

--rt
December 20, 2012
On Thu, Dec 20, 2012 at 05:48:13AM +0100, deadalnix wrote:
> On Thursday, 20 December 2012 at 04:11:00 UTC, Jesse Phillips wrote:
> >On Wednesday, 19 December 2012 at 23:05:59 UTC, deadalnix wrote:
> >
> >>master : used as a base for development. New feature are merged
> >>here.
> >>staging : used to provide a view of what the next version will
> >>look like. Regular snapshot of that branch are made so public
> >>can use the last features.
> >>version : used to contain a version that will have a support for
> >>an extended period of time.
[...]
> >From the sound of it this request is pulled into master. We continue to pull many of these changes in. How do we decide they should be placed into staging, when we pull them into master?. If we wait for some 'magic time' how do we pull it into staging, does that mean we now cherry pick commits of master?

In my mind, after a release, the contents of staging are updated to be exactly the same as master. This can be done either via a merge, or simply deleting the current staging and making a new one by branching from master.


> >Another issue is it sounds like master becomes a "phantom" branch. At no point in time would master resemble what is released. I see this as a problem because it is the branch people are developing off of, it means nothing to test in master as staging has the actual state that will be released.
> 
> That is very true. With the given modification master and stagging become redundant with one another. Why not get rid of stagging ?

No, staging does not receive new features except immediately after release. But it will get bugfixes that do not introduce new things.

Example timeline:
- time=0: we just made a release, so staging := master.

- time=1: a crash bug is discovered in master. Walter checks in a fix.
  Fix is propagated to staging because it doesn't introduce new things,
  only fixes existing problems.

- time=2: Walter decides to merge UDA implementation into master (for
  example). Staging does NOT get UDA merged in, because UDA is new.

- time=3: more bugs are found in master. Fixes are checked in. Fixes get
  propagated to staging.

- time=4: a UDA bug is fixed. Fix is NOT propagated to staging, because
  staging doesn't have UDA (yet).

- time=5: we decide that it's time to release. So:
   - we tag a release on current staging
   - we merge master into staging, so now staging gets UDA

- time=6: Walter adds HalfFloat to master. HalfFloat is NOT propagated
  to staging, because it's new.

- time=7: users running staging find a nasty design flaw in UDA. Walter
  makes a (breaking) fix in master, and the fix is propagated to
  staging, because UDA is already in staging.

- time=8: Walter finds bugs in HalfFloat. Fixes are checked into master,
  but NOT staging.

- time=9: Walter adds more new features to master. These are NOT
  propagated to staging, because they're new.

- time=10: it is decided that UDA design is stable now, and we're ready
  for release. So:
   - we tag a new release on current staging
   - we merge master into staging, now staging gets HalfFloat.

- etc.

Hopefully this makes it clearer what we're trying to achieve.


T

-- 
 Arise, you prisoners of Windows
 Arise, you slaves of Redmond, Wash,
 The day and hour soon are coming
 When all the IT folks say "Gosh!"
 It isn't from a clever lawsuit
 That Windowsland will finally fall,
 But thousands writing open source code
 Like mice who nibble through a wall.
-- The Linux-nationale by Greg Baker
December 20, 2012
On Wednesday, 19 December 2012 at 21:58:12 UTC, foobar wrote:
>
> Personally, I think the whole pre-development stage needs to get looks at -
> Specifically having some sort of at least high-level *binding* planning - road map, mile stones, todo lists. This should give more weight to DIPs.
> DIPs at the moment have no weight whatsoever. The attributes showed all the major flows in the decision making process:
>
> 1. Feature idea comes up in discussion.
> 2. Feature was discussed heavily by the community reaching some design consensus.
> 3. Plan is abandoned/forgotten due to Walter's objections - not to the design but the idea itself.
> 4. Feature comes up again in discussion, thus returning to point 1 above in infinite loop.
>

A big fix is in order there too, it's actually what should be fixed first but from what I've seen going on in here if we can get something even basic implemented for the development and releases that will be a *huge* step forward. Once we have some kind of formalized process in place and prove to everyone how much better things are because of it, it'll be a first by the looks of things, and from that experience the other big holes will become easier to pick out and deal with.

I think we have to keep it simple for now, focus on a dev, staging, release process, implement something, work out the bugs, get people used to having it, and prove the value, then we can get on with tackling the other major issues in a similar way.

There's lots of room for improvement ahead, but we need to make at least one significant step forward in a successful way before we can hope to move on to the next one.

--rt
December 20, 2012
On Thursday, 20 December 2012 at 05:32:30 UTC, H. S. Teoh wrote:
> On Thu, Dec 20, 2012 at 05:48:13AM +0100, deadalnix wrote:
>> On Thursday, 20 December 2012 at 04:11:00 UTC, Jesse Phillips
> In my mind, after a release, the contents of staging are updated to be
> exactly the same as master. This can be done either via a merge, or
> simply deleting the current staging and making a new one by branching
> from master.

Yes, exactly!

>
> No, staging does not receive new features except immediately after
> release. But it will get bugfixes that do not introduce new things.
>
> Example timeline:
> - time=0: we just made a release, so staging := master.
>
> - time=1: a crash bug is discovered in master. Walter checks in a fix.
>   Fix is propagated to staging because it doesn't introduce new things,
>   only fixes existing problems.
>
> - time=2: Walter decides to merge UDA implementation into master (for
>   example). Staging does NOT get UDA merged in, because UDA is new.
>
> - time=3: more bugs are found in master. Fixes are checked in. Fixes get
>   propagated to staging.
>
> - time=4: a UDA bug is fixed. Fix is NOT propagated to staging, because
>   staging doesn't have UDA (yet).
>
> - time=5: we decide that it's time to release. So:
>    - we tag a release on current staging
>    - we merge master into staging, so now staging gets UDA
>
> - time=6: Walter adds HalfFloat to master. HalfFloat is NOT propagated
>   to staging, because it's new.
>
> - time=7: users running staging find a nasty design flaw in UDA. Walter
>   makes a (breaking) fix in master, and the fix is propagated to
>   staging, because UDA is already in staging.
>
> - time=8: Walter finds bugs in HalfFloat. Fixes are checked into master,
>   but NOT staging.
>
> - time=9: Walter adds more new features to master. These are NOT
>   propagated to staging, because they're new.
>
> - time=10: it is decided that UDA design is stable now, and we're ready
>   for release. So:
>    - we tag a new release on current staging
>    - we merge master into staging, now staging gets HalfFloat.
>
> - etc.
>
> Hopefully this makes it clearer what we're trying to achieve.

Yes, fully agreeable with me!

>
>
> T

Thanks for writing that out so well. Excellent!

--rt
December 20, 2012
On Thursday, 20 December 2012 at 05:33:27 UTC, Rob T wrote:
> On Wednesday, 19 December 2012 at 21:58:12 UTC, foobar wrote:
>>
>> Personally, I think the whole pre-development stage needs to get looks at -
>> Specifically having some sort of at least high-level *binding* planning - road map, mile stones, todo lists. This should give more weight to DIPs.
>> DIPs at the moment have no weight whatsoever. The attributes showed all the major flows in the decision making process:
>>
>> 1. Feature idea comes up in discussion.
>> 2. Feature was discussed heavily by the community reaching some design consensus.
>> 3. Plan is abandoned/forgotten due to Walter's objections - not to the design but the idea itself.
>> 4. Feature comes up again in discussion, thus returning to point 1 above in infinite loop.
>>
>
> A big fix is in order there too, it's actually what should be fixed first but from what I've seen going on in here if we can get something even basic implemented for the development and releases that will be a *huge* step forward. Once we have some kind of formalized process in place and prove to everyone how much better things are because of it, it'll be a first by the looks of things, and from that experience the other big holes will become easier to pick out and deal with.
>
> I think we have to keep it simple for now, focus on a dev, staging, release process, implement something, work out the bugs, get people used to having it, and prove the value, then we can get on with tackling the other major issues in a similar way.
>
> There's lots of room for improvement ahead, but we need to make at least one significant step forward in a successful way before we can hope to move on to the next one.
>
> --rt

I see both as going hand-in-hand, otherwise we have chicken-egg problem.
We need a better process to allow more developers to contribute code more easily *and* we need better planning to provide incentive for new developer to contribute code.
Implementing only the the planning provides incentive but no means. Implementing only the process provides the means but not the incentive.

While improving git mastery levels is beneficial of itself, it brings no benefit to the _users_ if only Walter is still the main developer. We end up wasting Walter's time on adjusting to a new system for a hypothetical non existent system of many developers, when he already has a working system for the actual existing situation of being the sole main developer.

We need an *initial high-level* design for both parts. We can define and refine the details later and we probably should defer it to later and adjust it as we go along based on experience.
Arguing now on the exact time span of a release just wastes mind cycles and detracts from the important bits - reaching a consensus regarding the goals of the project and defining a general workflow that achieves those goals with minimal amount of overhead.
December 20, 2012
On Thursday, 20 December 2012 at 05:32:30 UTC, H. S. Teoh wrote:
>> >On Wednesday, 19 December 2012 at 23:05:59 UTC, deadalnix wrote:
>> >
>> >>master : used as a base for development. New feature are merged
>> >>here.
>> >>staging : used to provide a view of what the next version will
>> >>look like. Regular snapshot of that branch are made so public
>> >>can use the last features.
>> >>version : used to contain a version that will have a support for
>> >>an extended period of time.
>
> In my mind, after a release, the contents of staging are updated to be
> exactly the same as master. This can be done either via a merge, or
> simply deleting the current staging and making a new one by branching
> from master.

I can get behind this.

deadalnix, can we please go back to calling "version" as "release" that is what it is and what everyone else uses I don't see a need make a new name because you see "release" as something different.
December 20, 2012
On Thursday, 20 December 2012 at 08:27:22 UTC, foobar wrote:
> I see both as going hand-in-hand, otherwise we have chicken-egg problem.
> We need a better process to allow more developers to contribute code more easily *and* we need better planning to provide incentive for new developer to contribute code.
> Implementing only the the planning provides incentive but no means. Implementing only the process provides the means but not the incentive.
>
> While improving git mastery levels is beneficial of itself, it brings no benefit to the _users_ if only Walter is still the main developer. We end up wasting Walter's time on adjusting to a new system for a hypothetical non existent system of many developers, when he already has a working system for the actual existing situation of being the sole main developer.
>
> We need an *initial high-level* design for both parts. We can define and refine the details later and we probably should defer it to later and adjust it as we go along based on experience.
> Arguing now on the exact time span of a release just wastes mind cycles and detracts from the important bits - reaching a consensus regarding the goals of the project and defining a general workflow that achieves those goals with minimal amount of overhead.

I think what was proposed - to fast track this one, and not over engineer - is a good idea. That way we can at least get something that's far better than before, and be able to more quickly move on to the next big problem area which I would agree is what you are mentioning. There's lack of planning and how decisions are made is very inefficient and arbitrary, and I would also say we have a big problem with the specification being poorly managed, eg, it's not even available for download, has no process for revisions and releases, nothing at all - it's worse than the existing development process which we all agree is in need of a big overhaul.

Please, let's all agree to deal with that monster next, but it's a big problem to solve, and if we try to bundle it in with this one, we may not get any accomplished at all. I'd prefer the divide and conquer approach.

--rt
December 20, 2012
On Wednesday, 19 December 2012 at 21:30:44 UTC, Andrei Alexandrescu wrote:
>
> I agree with one "stable" branch.
>
> Andrei

Just some food for thought.

In the section about the "Branching model", the wiki currently has a staging branch in addition to the master branch. From what I understand, the idea seems to be to vet a release on staging until it is considered production level and then marked as the release.

Another idea could be to keep the quality of the master branch at a high level so as to be able to branch into a release at any time, directly from master. Before feature branches are merged back into master, their quality is vetted so the quality of master is maintained.

This idea seems similar to what is used for the vibe.d project (http://vibed.org/temp/branch-model-small.png). My apologies if I misunderstood their process.

It looks like Xamarin has been using this process for a while and it seems to be working for them. http://tirania.org/blog/archive/2011/Oct-14.html

Joseph