On Thursday, 10 January 2013 at 15:51:13 UTC, Andrei Alexandrescu wrote:
> My understanding was that staging is worked on only during the (short) time span from initiating a new release and finalizing that release.
>
> Andrei

I know, that is my general understanding of how staging would be used. This is a release branch in most described models.

However I think that extending that to a longer period isn't bad. But it does mean maintaining 3 branches (released, staging, dev). If we only want the usual short prep period, staging should die and we just branch into a Version branch (release branch) and tag when it is released. This would not need a transition period.

On Thursday, 10 January 2013 at 15:51:13 UTC, Andrei Alexandrescu wrote:
>
> My understanding was that staging is worked on only during the (short) time span from initiating a new release and finalizing that release.
>
> Andrei

There are significant side effects of that approach:

1) There's no beta made available for release. While you can try really hard to claim that there is a beta, in practical terms there isn't one, because very few people will even know it is there, and even less will have the ability to test it. Even if someone managed to try it out, not enough time is available to do anything significant with it and report back the results for refinement.

2) You'll never gain a high degree of stability between releases. The reason why is because the poorly tested development copy, is the released version, and following point 1) above, it will never be sufficiently tested and refined prior to release.

The results we got with the 2.061 release is what the process will give you, which is an unstable compiler that no one will want to use in a production environment unless they don't mind getting fired, in other words what was released is in fact a beta release of the unstable development branch.

If our intention is to produce a stable release that is maintained, then the process needs to be designed in a way that produces one.

The only way stability can happen, is if all development stops on a release branch for a sufficient period of time, ie., it enters into a maintenance mode where only critical bug fixes are introduced.

We don't want to stop the clock however, there's always newer exciting things to be done, so the development must proceed, and new major releases must follow. The tricky part is to do it in a non-disruptive way, that smooths out the instability between releases, that's the purpose the beta branch servers, a middle ground between stability and instability, but it's far more than that, it can also provide the development team with crucial feed back from real-world users.

We have to realize that the release process is not intended for developers, you have to consider what the users of the compiler want to see.

1) Users want to have access to a well maintained release of the compiler, where incremental upgrades contain only bug fixes up until the next major release, which will contain new features and other major improvements.

2) Some users want to have access to a reasonably stable beta release. The only reason why a user will want to try a beta version, is when the stable released version does not yet have something that they want to try out that is available in the beta, such as a new feature like UDA's. The beta should be in packaged form and available on the download page because you have to make it very easy for people to try it.

How we achieve the results may not matter so much, but from what I can see, to achieve point 1), the release must come from the beta, and there must be a release branch that can receive bug fixes. To achieve point 2) which is providing a reasonably stable beta for the users to try out, there must be a beta branch. The beta branch receives only new features and bug fixes that have been fully agreed to and well thought out.

Meanwhile all of the half baked new concepts are contained in the development branch (we've pegged it into master), and all the experimental stuff resides in the various scattered feature branches that only developers and high adventures are willing to try out.

--rt

Short comment about cherry pick - it is only bad in sense that it
creates separate untrackable commit with same content, which may
easily result in merging issues. If there is only one-way
direction for commit transitioning ( stuff goes from master to
staging and LTS and never goes back ) there is nothing inherently
bad about cherry pick.

On topic: well, it was somewhat discussed in the very first
thread. One pure technical issue is that github choses master as
default target for pull requests and people gonna do it wrong by
a simple mistake rather often. Much more important point is that
it complicates development process for newcomers without any
reasonable benefit. And if we still gonna push for success as an
open-source project, being easy understandable by an outsider is
as important as a clear release process.

Come on, I am reading all new posts in D and D.announce almost
every day (for 2+ years), check most of github pull requests for
druntime and phobos on weekly basis and still have a hard day
figuring out how some stuff is supposed to be done regarding
process and intentions. Imagine some random guy who just fixed
some stuff for his project and now he is full of altruistic
intentions to provide this fix/feature to an upstream. We have
current scenario A - fork repo on github, do stuff, push "pull
request" button. And now scenario B is proposed - go to wiki,
read release process guidelines, chose branch to fork from, do
not forget to chose the same branch when doing pull request
shortly after. Minor details regarding people comfort matter a
lot when they are supposed to work for free.

On Thursday, 10 January 2013 at 15:47:30 UTC, Andrei Alexandrescu
wrote:
> On 1/10/13 1:28 AM, mist wrote:
>> Sounds like different people have different understanding of staging
>> concept here and each one is doing minor tweaks to own direction. I ran
>> through updated wiki and some stuff like "pull requests against staging"
>> just scares me a lot.
>
> Why is that scary?
>
> Thanks,
>
> Andrei

My understanding is that your understanding is somewhat different from initial proposal but that is where discussion has flow to since then and that makes me sad :)

They very reason to have staging is to have better replacement to beta process which simply does not work good enough currently. Is good to have a single branch all the time, which some obscure project maintainer can check from time to time to make sure his stuff still compiles and fire regression bug reports if needed. He may even have add this test to own continuous integration suite (I'd do this if I had a serious D project) to be notified when stuff goes wrong without paying any constant attention.

Attention is a key here. How many D projects are out there? How many of their maintainers pay close attention to newsgroup and read beta mail list? Compare this to being able to check your stuff on very next release at any moment you have time and want to.

I stand at the point that for open source projects release and development processes should care most about end users and developers and least - about maintainers. Maintainers should have perfect git and process knowledge anyway, or at some scales thing are doomed to be crewed (2.061 anyone?).

Thus I vote for a persistent staging branch.

>
> My understanding was that staging is worked on only during the (short) time span from initiating a new release and finalizing that release.
>
> Andrei

On Friday, 11 January 2013 at 18:03:33 UTC, mist wrote:
> My understanding is that your understanding is somewhat different from initial proposal but that is where discussion has flow to since then and that makes me sad :)
>
> They very reason to have staging is to have better replacement to beta process which simply does not work good enough currently. Is good to have a single branch all the time, which some obscure project maintainer can check from time to time to make sure his stuff still compiles and fire regression bug reports if needed. He may even have add this test to own continuous integration suite (I'd do this if I had a serious D project) to be notified when stuff goes wrong without paying any constant attention.
>
> Attention is a key here. How many D projects are out there? How many of their maintainers pay close attention to newsgroup and read beta mail list? Compare this to being able to check your stuff on very next release at any moment you have time and want to.
>
> I stand at the point that for open source projects release and development processes should care most about end users and developers and least - about maintainers. Maintainers should have perfect git and process knowledge anyway, or at some scales thing are doomed to be crewed (2.061 anyone?).
>
> Thus I vote for a persistent staging branch.
>
>>
>> My understanding was that staging is worked on only during the (short) time span from initiating a new release and finalizing that release.
>>
>> Andrei

I don't understand the problem you seem to see in the process.

I've been away from the PC so couldn't respond earlier but I did manage to read most of the wiki page and I think Johnathan did a very good job explaining the various concepts. The only thing missing IMO is CI:
It's already agreed that master is the development channel in the proposed process. The next logical step would be to generate nightly builds off of that branch. The staging branch is the "beta" channel - and also makes sense to have a periodical beta build. This depends on the release time span but something like fortnight or monthly beta builds makes sense to me.
the additional manual builds described on the wiki on staging or even before an official release on the version branch will be something like RCs.

Regarding pull requests targeting master, the current model *is* geared around that. Most contributions _should_ go to master (aka devel) and go through the full process. pull-requests for staging are meant for fixing regressions and critical bugs only, where there is _higher urgency_ for the fix that justifies the short-cut. Regular "bug fixes" should simply go through the regular process and will be released in the _next_ release.

Am Thu, 10 Jan 2013 20:36:40 +0100
schrieb "Jesse Phillips" <Jessekphillips+D@gmail.com>:

>[...]
> 
> However I think that extending that to a longer period isn't bad. But it does mean maintaining 3 branches (released, staging, dev). If we only want the usual short prep period, staging should die and we just branch into a Version branch (release branch) and tag when it is released. This would not need a transition period.

Staging has also 2 other small benefits: It is a central 'beta branch', you can check out 'staging' any time and be sure this is going to be the next release. You'll always have all the features that are in the next release and the only updates till the next release will be bug fixes. This is an interesting point imho.

And then there is the benefit that you'll have a central target for bug
fix pull requests:
Without staging, bug fixes usually go into master. But when a release
branch is created, bug fixes suddenly have to go into that release
branch. Then after the release, bug fixes have to go to master again
and only regression fixes can go into the release branch.

This is kinda annoying as it does not only mean the target branch for pull requests on github has to be changed a lot, you might also have to rebase the bugfix on the other branch.

With staging you always target staging for bug fixes. They can be merged at any point in time.

Am Sat, 12 Jan 2013 09:22:27 +0100
schrieb "foobar" <foo@bar.com>:

> [...]
> 
> Regarding pull requests targeting master, the current model *is* geared around that. Most contributions _should_ go to master (aka devel) and go through the full process. pull-requests for staging are meant for fixing regressions and critical bugs only, where there is _higher urgency_ for the fix that justifies the short-cut. Regular "bug fixes" should simply go through the regular process and will be released in the _next_ release.

I also think targeting staging for some pull requests is not that bad. In the end it's not that bad if a pull request was accidentally merged into master instead of staging. It just means that it'll take more time for the fix to appear in a release (It'll be delayed by one release), but it won't mess anything up.

Regarding where "most" requests go: This also depends on the project. I guess for phobos most requests are enhancements/new features and would go to master. For druntime it's mostly bugfixes, so probably more requests target staging. And for the dmd almost everything is a bug fix and could target staging.

The wiki currently says all bug fixes should go to staging. This is a concession to D's rapid development. But again, it's not that important where the pull requests go. We should try to make breaking changes in master and have only "harmless" changes in staging, but the exact decision is always up to the contributor & maintainers.

But those 'minor' tweaks such as defining what exactly is merged to master and what to staging can always be made later on.

Am Fri, 11 Jan 2013 18:54:12 +0100
schrieb "mist" <none@none.none>:

> Short comment about cherry pick - it is only bad in sense that it creates separate untrackable commit with same content, which may easily result in merging issues. If there is only one-way direction for commit transitioning ( stuff goes from master to staging and LTS and never goes back ) there is nothing inherently bad about cherry pick.

Having multiple commit ids for the same commit is also annoying. For example if someone bisected a regression and you want to revert a commit you have to figure out the commit id of the cherry picked commits.

> 
> On topic: well, it was somewhat discussed in the very first thread. One pure technical issue is that github choses master as default target for pull requests and people gonna do it wrong by a simple mistake rather often.
It doesn't harm though if a requests goes to master instead of staging - the only difference with the approach described on the wiki page is that the bug fix will be delayed by one release if it's merge into master instead of staging.

> Much more important point is that
> it complicates development process for newcomers without any
> reasonable benefit. And if we still gonna push for success as an
> open-source project, being easy understandable by an outsider is
> as important as a clear release process.
The rule is quite simple and easy to understand though:
regression fix -> version branch
bug fix -> staging
enhancement -> master

The benefits have already been stated in other posts.
> 
> Come on, I am reading all new posts in D and D.announce almost every day (for 2+ years), check most of github pull requests for druntime and phobos on weekly basis and still have a hard day figuring out how some stuff is supposed to be done regarding process and intentions. Imagine some random guy who just fixed some stuff for his project and now he is full of altruistic intentions to provide this fix/feature to an upstream. We have current scenario A - fork repo on github, do stuff, push "pull request" button. And now scenario B is proposed - go to wiki, read release process guidelines, chose branch to fork from, do not forget to chose the same branch when doing pull request shortly after. Minor details regarding people comfort matter a lot when they are supposed to work for free.

It's always possible for a maintainer to get the changes from the pull request into their personal repo, rebase on the correct branch, reopen. And it's not that bad if a request goes to master accidentally.

Regression fixes have to go to version branches anyway, you can't avoid this without cherry-picking. But newbies don't do regression fixes anyway.

Separating between staging and master then has the nice benefit of a more stable "beta" branch and a fast moving development branch.

On Wednesday, 9 January 2013 at 17:12:15 UTC, Jesse Phillips wrote:
> Before starting this I did come across Walter's complaint of issues, but feel this needs a new thread. But didn't read all replies.
>
> The new process introduces a new branch called staging. This is _not_ used as one would first think.
>
> staging is created/updated _after_ release. This means a delay in changes in master (breaking changes/new additions) and release (a good thing). It also means that we have to do some awkward delaying to get some useful changes into staging so that it is different from this last release.

Since we don't have much opposition here for the staging process I will make my suggestion for this transition.

Since std.uni is up for review, lets plan that upon inclusion we move master into staging. This will be the next release as defined in our process. When another group of changes is considered ready we can make the release and update staging.

We could choose other changes, but I think this will be a good arbitrary point to make the transition (and not end up with an "empty" release).

On Saturday, 12 January 2013 at 12:30:05 UTC, Johannes Pfau wrote:
> Am Fri, 11 Jan 2013 18:54:12 +0100
> schrieb "mist" <none@none.none>:
>
>> Short comment about cherry pick - it is only bad in sense that it
>> creates separate untrackable commit with same content, which may
>> easily result in merging issues. If there is only one-way
>> direction for commit transitioning ( stuff goes from master to
>> staging and LTS and never goes back ) there is nothing inherently
>> bad about cherry pick.
>
> Having multiple commit ids for the same commit is also annoying. For
> example if someone bisected a regression and you want to revert a commit
> you have to figure out the commit id of the cherry picked commits.
>

AFAIK reverts are content-based, so knowing any one commit id is enough. It is annoying mostly when comparison happens between two branches with those commits.

>> Much more important point is that
>> it complicates development process for newcomers without any
>> reasonable benefit. And if we still gonna push for success as an
>> open-source project, being easy understandable by an outsider is
>> as important as a clear release process.
> The rule is quite simple and easy to understand though:
> regression fix -> version branch
> bug fix -> staging
> enhancement -> master

Ye, it is easy.. except you need to actually check out wiki page to be aware about it. At the very least we need relevant excerpts for outer devs separated to one small wiki page and set links to this page in github marked with big "NB:".

I don't have a reason to argue about the rest.