On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis wrote:
> On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
>> What I am most concerned about are the timespans discussed:
>> "I propose to go for a yearly release of the
>> stable branches with one year support (In the beginning)."
>> 
>> The wiki discussion page even mentions "I don't think 4 months are a
>> ridiculously long time for staging if the release is made every 3
>> years."
>
> That makes it sound like they want the current stuff to be marked as staging
> and then have some other release that sits around for a very long time being
> treated as somehow more stable than the staging branch. In general, there's
> nothing about bug fixing which is stable, and separating out bug fixes between a
> more stable branch and a less stable one doesn't make all that much sense.
> Separating out new features and not putting them in the "stable" branch makes
> sense, but that really doesn't make sense for bugs.
>
> Also, the name "staging" implies that it's purely for preparing a release, in
> which case keeping it around makes _no_ sense. Not to mention, as already
> mentioned, it would make more sense to simply create a new branch for each
> release to begin with than have a staging branch if that's what it's for. And
> if that's not what it's for, then it's a horrible name.
>
> - Jonathan M Davis

As I explained several times before, this is a HORRIBLE idea. If each release has its own branch than a bug fix needs to be applied to _all_ such branches _manually_ and it is easy to get a situation where a fix on say version 3.1 was accidentally not merged to version 4.0 (forgotten) and suddenly the bug was "unfixed".
Whatever the name of this branch is, the semantics/purpose of it is to provide integration.

On 1/3/13 6:40 PM, foobar wrote:
> I suggest to read about how git works and how best to utilize its
> features, and NOT apply common wisdom from CVS/SVN/RCS which are based
> on a _completely different_ conceptual model.

Better yet, since you are already so conversant with git, you could help us improve the process directly. Care to make a pass through the wiki page?

Andrei

On Friday, January 04, 2013 00:47:08 foobar wrote:
> On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis
> 
> wrote:
> > On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
> >> What I am most concerned about are the timespans discussed:
> >> "I propose to go for a yearly release of the
> >> stable branches with one year support (In the beginning)."
> >> 
> >> The wiki discussion page even mentions "I don't think 4 months
> >> are a
> >> ridiculously long time for staging if the release is made
> >> every 3
> >> years."
> > 
> > That makes it sound like they want the current stuff to be
> > marked as staging
> > and then have some other release that sits around for a very
> > long time being
> > treated as somehow more stable than the staging branch. In
> > general, there's
> > nothing about bug fixing which is stable, and separating out
> > bug fixes between a
> > more stable branch and a less stable one doesn't make all that
> > much sense.
> > Separating out new features and not putting them in the
> > "stable" branch makes
> > sense, but that really doesn't make sense for bugs.
> > 
> > Also, the name "staging" implies that it's purely for preparing
> > a release, in
> > which case keeping it around makes _no_ sense. Not to mention,
> > as already
> > mentioned, it would make more sense to simply create a new
> > branch for each
> > release to begin with than have a staging branch if that's what
> > it's for. And
> > if that's not what it's for, then it's a horrible name.
> > 
> > - Jonathan M Davis
> 
> As I explained several times before, this is a HORRIBLE idea. If
> each release has its own branch than a bug fix needs to be
> applied to _all_ such branches _manually_ and it is easy to get a
> situation where a fix on say version 3.1 was accidentally not
> merged to version 4.0 (forgotten) and suddenly the bug was
> "unfixed".
> Whatever the name of this branch is, the semantics/purpose of it
> is to provide integration.

??? Bug fixes go to master. New release branches are branched from master. So, you're not going to have future branches not getting previous bug fixes. The only way that you're going to get problems with bug fixes not making it into future releases is if you commit them to the branch first rather than directly to master (which is a bad idea IMHO) and then you forget to merge that back into master.

- Jonathan M Davis

On 1/3/2013 11:19 AM, Andrei Alexandrescu wrote:
> I think this is a pertinent point - the process proposed at github is incomplete
> and scantily motivated. Can the experts make one more pass through it?


One feature I really liked about github was when you created a new repository, it would give you an exact list of corresponding git commands to set up your local version.

On 1/3/2013 12:17 PM, Rob T wrote:
> I'm rather concerned when I see comments that suggest that the purpose of the
> staging branch was not understood in the slightest. There's a lot of discussion
> on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1@digitalmars.com
>
> The people who are attempting to use the new process have to at least take some
> time to read through the process discussion threads, certainly that is not
> ideal, and the wiki should explain everything much better.

I agree that the pertinent rationales must be in the wiki.

On Thursday, 3 January 2013 at 22:11:59 UTC, H. S. Teoh wrote:
> We could either:
>
> (1) Have an actual branch called 'staging', where changes made to master
> since the last merge are merged in immediately after release, and when
> things have sufficiently stabilized, we tag/branch that particular
> commit as release 2.X.Y;
>
> (2) Or, we can follow Walter's suggestion of branching master into 2.X.Y
> after a release and just tag it as a release when it's sufficiently
> stable. In this case, the latest 2.X.Y branch acts as 'staging'.
>
> Part of the draw of (1) is that betatesters and adventurous people have
> a fixed place to go to, to get the latest and greatest features, but
> without the instability of master.

(1) is what I was advocating precisely so that beta testers could use it shortly after a release and up to the final next release. There are a few benefits to this that may be under appreciated. There's plenty of benefits to having a lot of beta testers, so we want to draw them in and encourage them to use the latest beta version as soon as possible. Real world programmers do not want to make use of something that is too unstable, so they will tend to avoid the master development branch and will prefer the latest stable release instead. The only reason they will try out the latest beta version, is to gain access to a bug fix or new feature that they really need for their real world programming tasks, but they will only try it out if they think that the beta is stable enough and is not going to get horribly broken by anything radical being merged in without proper oversight and planning (that's what master and the related feature branches are for).

If we immediately move master into the next staging after a new release, and freeze the introduction of new features, only to concentrate on stabilizing it for the next release, then we're more likely to attract beta testers from the real world, these are the guys who will do testing under the most realistically conditions possible, something that in invaluable in gaining feed back and routing our difficult to discover bugs.

> I think (2) is more commonly what is actually done by software companies
> and projects. In this case, 'staging' isn't so much as an actual
> physical branch, as the general concept of having a branch (2.X.Y, which
> changes per release) to stabilize things for a release, while
> development on master continues unhindered.
>
> The Linux kernel also uses a similar kind of process: X.Y.Z is a
> development release (can be thought of as 'master') when Y is odd, and a
> stable release (can be thought of as 'staging' or Walter's release
> branches) when Y is even.
>
> Basically, the idea is that master should be able to receive the latest
> and greatest features, completely unhindered, while stabilizing the code
> for a release is done in a separate branch which doesn't receive new
> features, but only bugfixes, prior to an actual release.
>

I'm not certain what the major difference is between (1) and (2), because they both sound very nearly identical. In both cases, they seem to provide the same goals of stabilizing the next release, while allowing development to proceed unhindered, so if (2) can be made to achieve the goal I described for attracting real world beta testers, it may not matter which is used, however I would suggest that we always focus on serving the needs of the end user as a top priority as much as is possible. D requires plenty of significant end users working on significant projects.

You did mention that (1) is more suitable for beta testers, so I ask how is (2) not as suitable?

--rt

Am Fri, 04 Jan 2013 00:40:03 +0100
schrieb "foobar" <foo@bar.com>:
> 
> Let's clarify a bunch of things here:
> 1. There is no overhead with maintaining many feature brunches.
I didn't mean the overhead of the branches itself. I meant the overhead of merging those first into master. Then later determining if master is stable (how?) and then merging into staging. The wiki says this is useful to allow fixing integration bugs. I think with mostly bugfixes there won't be many integration bugs. And merging into staging always requires someone to do it, which is imho overhead (regarding "human resources", not technical).

> 2. Feature branches can be safely removed after their contents have been merged. This is useful for avoiding clutter.
sure
> 3. There is no need to re-review the _same_ code on each branch it is merged to. That's just plain stupid.
Well, the wiki says things are merged into staging if they are considered sufficiently stable. Who determines what is stable and how?

> 4. Chery picking is a HORRIBLE idea for managing releases and should only be used on development. Chery-picking COPIES commits and thus loses the connections in the history graph.

Sure, this is why I said mixing cherry picking and merging is not a good idea. As long as you only cherry pick into a branch and never merge into that branch or merge it back there is no issue.

I missed the detail that you suppose basing bugfix branches on the oldest applicable release branch. That should be made very clear in the wiki.

I just think that this process is counter-productive for dmd as most changes are bugfixes and regressions are caused by bug fixes as well. Applying _every_ bugfix in older releases has a high risk of reintroducing regressions (and / or breaking user code. If the compiler didn't enforce const in some situations, fixing it is a bug fix. But fixing it will break user code - so you have a 2.061 that works for users and a 2.061.1 which breaks code).

It makes more sense to me to only fix regressions that were introduced with the base release. So 2.061.1 would only fix regressions introduced in 2.061. (This can still be done by basing that fix on the latest applicable version and merging back into master / staging).

Also the lowest applicable version for most dmd bugs is probably 2.01. So if we start this process now, we'll still be fixing bugs in 2.061 in 10 years, then merging into 2.062, 2.063.... Here "lowest _supported_ applicable version" makes more sense imho, where supported needs to be defined (e.g. 2 releases). This is not a big problem if we only talk about regressions, but I'd still prefer "lowest _supported_ applicable version"

> 5. There is no problem developing bug_fixes on separate topic branches.
As we already do.
> 6. There is no problem merging back to master.
As long as there are no cherry picked commits into that branch. AFAIK we have already cherry picked commits from master into staging. Then merging staging back into master is not good - It will probably work but as you said cherry picking duplicates commits and we'll therefore merge identical changes with different ids.

We should probably delete staging and start again as soon as the workflow is clear to everyone.

> 7. Let me say this again. Merges are cheap and easy in git. So are re-merges. Avoiding merges is an artifact of client-server designs such as SVN and is NOT compatible with Git's conceptual model.
> 
> I suggest to read about how git works and how best to utilize its features, and NOT apply common wisdom from CVS/SVN/RCS which are based on a _completely different_ conceptual model.

I don't have common wisdom from cvs/svn ;-)
Merges are not that easy if you want to merge only a single commit or
skip certain commits. Although you can tell merge to skip commits it's
in no way easy. http://stackoverflow.com/a/729723/471401

It's  probably best to not get into this situation. The workflow described on the wiki can indeed work well without cherry picking.

But what was done till 2.061 was basing bug fixes on the master branch. If you want to get these into a release branch without features, etc cherry-picking is way easier than merging.

Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob@ucora.com>:

> 
> You did mention that (1) is more suitable for beta testers, so I ask how is (2) not as suitable?

With (1) you can always use staging, it'll always be the latest beta.
With (2) you'll have to figure out the most recent branch (2.061,
2.062, 2.063).

But I think we can start with (2) and change to (1) later if we think
it's useful. Staging complicates the concept(on a cognitive level) and
it needs additional "human resources" to do the merges into it.


I think I understand the whole concept now. I seems to be a good solution but it requires some discipline (basing changes on the correct branches) so we should document it as clearly as possible. I'd strongly recommend to only base regression fixes on release branches and usual bug fixes on master (see my other message).

On Fri, 2013-01-04 at 00:40 +0100, foobar wrote:
[…]
> Let's clarify a bunch of things here:
> 1. There is no overhead with maintaining many feature brunches.

Which is why Git is superior to Mercurial for the sort of process going on here.

> 2. Feature branches can be safely removed after their contents have been merged. This is useful for avoiding clutter.

See above :-)

> 3. There is no need to re-review the _same_ code on each branch it is merged to. That's just plain stupid.

I am not so sure on this. A changeset in a different context is a different changeset, which is why Mercurial is the way it is and why it has some benefits overf Git.  The effect of a code change in one branch is not necessarily the same as the effect of the same change in a different branch. However review is not the right tool here, CI is. Each and every branch in the mainline must have a CI listener that runs all the code on every commit.

Branches in the mainline are CVCS thinking, not wrong, but must be treated as an integral part of the project, hence the CI requirement. Branches held in forks of the mainline have no such requirement, but people should run all the tests after each commit as a matter of professionalism. Others can of course pull these feature branches into their own fork repository for working on. This is the DVCS way. Eventually of course commit to the mainline is required otherwise a project fork has been made.

> 4. Chery picking is a HORRIBLE idea for managing releases and should only be used on development. Chery-picking COPIES commits and thus loses the connections in the history graph.

Agreed.

> 5. There is no problem developing bug_fixes on separate topic branches.

See 1 above. Git makes this trivially easy.

> 6. There is no problem merging back to master.

See 1 above. Git makes this trivially easy.

> 7. Let me say this again. Merges are cheap and easy in git. So are re-merges. Avoiding merges is an artifact of client-server designs such as SVN and is NOT compatible with Git's conceptual model.

It is important to note that Mercurial has different properties again, so leads to different workflow. Given the choice has been Git then Git workflows apply.

Every developer should be keeping master as a mirror of the mainline master and working on feature branches. When a changeset is accepted after review, the committer commits to the mainline and then everyone can pull and garbage collect, and in the process delete the feature branch they have.

I tend to fork the mainline repository locally and also on GitHub and then manually link my two forks. This means "git pull" locally pulls all branches from the mainline repository. I work on feature branches pushed to my GitHub fork, people can then pull them as they wish. You have to remember to fetch from all the remote branches you are tracking. I can then make a pull request on my GitHub fork and still continue to track the mainline locally.

> I suggest to read about how git works and how best to utilize its features, and NOT apply common wisdom from CVS/SVN/RCS which are based on a _completely different_ conceptual model.

Indeed.

Star model has a role in DVCS, the mainline repository is the mainline from which all else stems. But not all traffic has to be routed through it with Git. People can share feature branches between themselves to collaborate to create a pull request without ever having to use the mainline. That is the whole point.

It means that the mainline committer role is not to do all the development work, but is to coordinate and mentor activity which they then review and when happy merge into mainline.

No-one should ever be doing development on master.

No-one should ever push to mainline without someone at least knowing they are about to do it and have previously checked the work.

No mainline repository should ever be without a full CI.

http://d.puremagic.com/test-results/

Looks like part of a solution.  It should also provides artefacts for download and install as nightly releases.  It assumes there is only master in the repository. If there are any other branches in the mainline repository then where are the CIs for those?

In my clone of DMD I see master, staging, dmd-1.x branches. Given DMD is a single branch development, I don't see a need for staging. I shall have to go through all the email to find the rationale, unless someone has a short paragraph rationale.

What I find really bizarre is that the tags v2.061 and v1.076 are on branches other than master. To me the whole Git history looks a mess, essentially broken. I suggest some serious rebasing and merging to make the history consistent with the version numbering.

The questions to ask:

If I want to create a feature branch, which branch do I use as a base?

Is staging just the 2.x maintenance branch?

When do changes on master make it to staging?

Should staging and dmd-1.x branch HEADS be moved forwards on the branch?

Sorry if this is all answered somewhere, just point me at the URL where the answers are explained.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

Weird. While I was reading "Next focus: PROCESS" topic and http://wiki.dlang.org/Proposed_new_D_development_process entries, I was completely convinced that switch to faster release model is proposed, so that master replaces staging the very moment one version is released (and it is the same moment as testing of next release begins). Plus once a year or so release is marked LTS and cherry-pick bug-fixes until next LTS is out. Was my understanding wrong?

Anyway, now I am reading all those comments and it feels absurdly like no one has discussed this stuff few weeks ago and different topic and argument has started from beginning. This is madness.