View mode: basic / threaded / horizontal-split · Log in · Help
January 03, 2013
Re: github release procedure
On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis 
wrote:
> On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
>> What I am most concerned about are the timespans discussed:
>> "I propose to go for a yearly release of the
>> stable branches with one year support (In the beginning)."
>> 
>> The wiki discussion page even mentions "I don't think 4 months 
>> are a
>> ridiculously long time for staging if the release is made 
>> every 3
>> years."
>
> That makes it sound like they want the current stuff to be 
> marked as staging
> and then have some other release that sits around for a very 
> long time being
> treated as somehow more stable than the staging branch. In 
> general, there's
> nothing about bug fixing which is stable, and separating out 
> bug fixes between a
> more stable branch and a less stable one doesn't make all that 
> much sense.
> Separating out new features and not putting them in the 
> "stable" branch makes
> sense, but that really doesn't make sense for bugs.
>
> Also, the name "staging" implies that it's purely for preparing 
> a release, in
> which case keeping it around makes _no_ sense. Not to mention, 
> as already
> mentioned, it would make more sense to simply create a new 
> branch for each
> release to begin with than have a staging branch if that's what 
> it's for. And
> if that's not what it's for, then it's a horrible name.
>
> - Jonathan M Davis

As I explained several times before, this is a HORRIBLE idea. If 
each release has its own branch than a bug fix needs to be 
applied to _all_ such branches _manually_ and it is easy to get a 
situation where a fix on say version 3.1 was accidentally not 
merged to version 4.0 (forgotten) and suddenly the bug was 
"unfixed".
Whatever the name of this branch is, the semantics/purpose of it 
is to provide integration.
January 04, 2013
Re: github release procedure
On 1/3/13 6:40 PM, foobar wrote:
> I suggest to read about how git works and how best to utilize its
> features, and NOT apply common wisdom from CVS/SVN/RCS which are based
> on a _completely different_ conceptual model.

Better yet, since you are already so conversant with git, you could help 
us improve the process directly. Care to make a pass through the wiki page?

Andrei
January 04, 2013
Re: github release procedure
On Friday, January 04, 2013 00:47:08 foobar wrote:
> On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis
> 
> wrote:
> > On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
> >> What I am most concerned about are the timespans discussed:
> >> "I propose to go for a yearly release of the
> >> stable branches with one year support (In the beginning)."
> >> 
> >> The wiki discussion page even mentions "I don't think 4 months
> >> are a
> >> ridiculously long time for staging if the release is made
> >> every 3
> >> years."
> > 
> > That makes it sound like they want the current stuff to be
> > marked as staging
> > and then have some other release that sits around for a very
> > long time being
> > treated as somehow more stable than the staging branch. In
> > general, there's
> > nothing about bug fixing which is stable, and separating out
> > bug fixes between a
> > more stable branch and a less stable one doesn't make all that
> > much sense.
> > Separating out new features and not putting them in the
> > "stable" branch makes
> > sense, but that really doesn't make sense for bugs.
> > 
> > Also, the name "staging" implies that it's purely for preparing
> > a release, in
> > which case keeping it around makes _no_ sense. Not to mention,
> > as already
> > mentioned, it would make more sense to simply create a new
> > branch for each
> > release to begin with than have a staging branch if that's what
> > it's for. And
> > if that's not what it's for, then it's a horrible name.
> > 
> > - Jonathan M Davis
> 
> As I explained several times before, this is a HORRIBLE idea. If
> each release has its own branch than a bug fix needs to be
> applied to _all_ such branches _manually_ and it is easy to get a
> situation where a fix on say version 3.1 was accidentally not
> merged to version 4.0 (forgotten) and suddenly the bug was
> "unfixed".
> Whatever the name of this branch is, the semantics/purpose of it
> is to provide integration.

??? Bug fixes go to master. New release branches are branched from master. So, 
you're not going to have future branches not getting previous bug fixes. The 
only way that you're going to get problems with bug fixes not making it into 
future releases is if you commit them to the branch first rather than directly 
to master (which is a bad idea IMHO) and then you forget to merge that back 
into master.

- Jonathan M Davis
January 04, 2013
Re: github release procedure
On 1/3/2013 11:19 AM, Andrei Alexandrescu wrote:
> I think this is a pertinent point - the process proposed at github is incomplete
> and scantily motivated. Can the experts make one more pass through it?


One feature I really liked about github was when you created a new repository, 
it would give you an exact list of corresponding git commands to set up your 
local version.
January 04, 2013
Re: github release procedure
On 1/3/2013 12:17 PM, Rob T wrote:
> I'm rather concerned when I see comments that suggest that the purpose of the
> staging branch was not understood in the slightest. There's a lot of discussion
> on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1@digitalmars.com
>
> The people who are attempting to use the new process have to at least take some
> time to read through the process discussion threads, certainly that is not
> ideal, and the wiki should explain everything much better.

I agree that the pertinent rationales must be in the wiki.
January 04, 2013
Re: github release procedure
On Thursday, 3 January 2013 at 22:11:59 UTC, H. S. Teoh wrote:
> We could either:
>
> (1) Have an actual branch called 'staging', where changes made 
> to master
> since the last merge are merged in immediately after release, 
> and when
> things have sufficiently stabilized, we tag/branch that 
> particular
> commit as release 2.X.Y;
>
> (2) Or, we can follow Walter's suggestion of branching master 
> into 2.X.Y
> after a release and just tag it as a release when it's 
> sufficiently
> stable. In this case, the latest 2.X.Y branch acts as 'staging'.
>
> Part of the draw of (1) is that betatesters and adventurous 
> people have
> a fixed place to go to, to get the latest and greatest 
> features, but
> without the instability of master.

(1) is what I was advocating precisely so that beta testers could 
use it shortly after a release and up to the final next release. 
There are a few benefits to this that may be under appreciated. 
There's plenty of benefits to having a lot of beta testers, so we 
want to draw them in and encourage them to use the latest beta 
version as soon as possible. Real world programmers do not want 
to make use of something that is too unstable, so they will tend 
to avoid the master development branch and will prefer the latest 
stable release instead. The only reason they will try out the 
latest beta version, is to gain access to a bug fix or new 
feature that they really need for their real world programming 
tasks, but they will only try it out if they think that the beta 
is stable enough and is not going to get horribly broken by 
anything radical being merged in without proper oversight and 
planning (that's what master and the related feature branches are 
for).

If we immediately move master into the next staging after a new 
release, and freeze the introduction of new features, only to 
concentrate on stabilizing it for the next release, then we're 
more likely to attract beta testers from the real world, these 
are the guys who will do testing under the most realistically 
conditions possible, something that in invaluable in gaining feed 
back and routing our difficult to discover bugs.

> I think (2) is more commonly what is actually done by software 
> companies
> and projects. In this case, 'staging' isn't so much as an actual
> physical branch, as the general concept of having a branch 
> (2.X.Y, which
> changes per release) to stabilize things for a release, while
> development on master continues unhindered.
>
> The Linux kernel also uses a similar kind of process: X.Y.Z is a
> development release (can be thought of as 'master') when Y is 
> odd, and a
> stable release (can be thought of as 'staging' or Walter's 
> release
> branches) when Y is even.
>
> Basically, the idea is that master should be able to receive 
> the latest
> and greatest features, completely unhindered, while stabilizing 
> the code
> for a release is done in a separate branch which doesn't 
> receive new
> features, but only bugfixes, prior to an actual release.
>

I'm not certain what the major difference is between (1) and (2), 
because they both sound very nearly identical. In both cases, 
they seem to provide the same goals of stabilizing the next 
release, while allowing development to proceed unhindered, so if 
(2) can be made to achieve the goal I described for attracting 
real world beta testers, it may not matter which is used, however 
I would suggest that we always focus on serving the needs of the 
end user as a top priority as much as is possible. D requires 
plenty of significant end users working on significant projects.

You did mention that (1) is more suitable for beta testers, so I 
ask how is (2) not as suitable?

--rt
January 04, 2013
Re: github release procedure
Am Fri, 04 Jan 2013 00:40:03 +0100
schrieb "foobar" <foo@bar.com>:
> 
> Let's clarify a bunch of things here:
> 1. There is no overhead with maintaining many feature brunches.
I didn't mean the overhead of the branches itself. I meant the overhead
of merging those first into master. Then later determining if master is
stable (how?) and then merging into staging. The wiki says this is
useful to allow fixing integration bugs. I think with mostly bugfixes
there won't be many integration bugs. And merging into staging always
requires someone to do it, which is imho overhead (regarding "human
resources", not technical).

> 2. Feature branches can be safely removed after their contents 
> have been merged. This is useful for avoiding clutter.
sure
> 3. There is no need to re-review the _same_ code on each branch 
> it is merged to. That's just plain stupid.
Well, the wiki says things are merged into staging if they are
considered sufficiently stable. Who determines what is stable and how?

> 4. Chery picking is a HORRIBLE idea for managing releases and 
> should only be used on development. Chery-picking COPIES commits 
> and thus loses the connections in the history graph.

Sure, this is why I said mixing cherry picking and merging is not a
good idea. As long as you only cherry pick into a branch and never
merge into that branch or merge it back there is no issue.

I missed the detail that you suppose basing bugfix branches on the
oldest applicable release branch. That should be made very clear in the
wiki.

I just think that this process is counter-productive for dmd as most
changes are bugfixes and regressions are caused by bug fixes as well.
Applying _every_ bugfix in older releases has a high risk of
reintroducing regressions (and / or breaking user code. If the compiler
didn't enforce const in some situations, fixing it is a bug fix. But
fixing it will break user code - so you have a 2.061 that works for
users and a 2.061.1 which breaks code).

It makes more sense to me to only fix regressions that were introduced
with the base release. So 2.061.1 would only fix regressions introduced
in 2.061. (This can still be done by basing that fix on the latest
applicable version and merging back into master / staging).

Also the lowest applicable version for most dmd bugs is probably 2.01.
So if we start this process now, we'll still be fixing bugs in 2.061 in
10 years, then merging into 2.062, 2.063.... Here "lowest _supported_
applicable version" makes more sense imho, where supported needs to be
defined (e.g. 2 releases). This is not a big problem if we only talk
about regressions, but I'd still prefer "lowest _supported_ applicable
version"

> 5. There is no problem developing bug_fixes on separate topic 
> branches.
As we already do.
> 6. There is no problem merging back to master.
As long as there are no cherry picked commits into that branch. AFAIK
we have already cherry picked commits from master into staging. Then
merging staging back into master is not good - It will probably work
but as you said cherry picking duplicates commits and we'll therefore
merge identical changes with different ids.

We should probably delete staging and start again as soon as the
workflow is clear to everyone.

> 7. Let me say this again. Merges are cheap and easy in git. So 
> are re-merges. Avoiding merges is an artifact of client-server 
> designs such as SVN and is NOT compatible with Git's conceptual 
> model.
> 
> I suggest to read about how git works and how best to utilize its 
> features, and NOT apply common wisdom from CVS/SVN/RCS which are 
> based on a _completely different_ conceptual model.

I don't have common wisdom from cvs/svn ;-)
Merges are not that easy if you want to merge only a single commit or
skip certain commits. Although you can tell merge to skip commits it's
in no way easy. http://stackoverflow.com/a/729723/471401

It's  probably best to not get into this situation. The workflow
described on the wiki can indeed work well without cherry picking.

But what was done till 2.061 was basing bug fixes on the master branch.
If you want to get these into a release branch without features, etc
cherry-picking is way easier than merging.
January 04, 2013
Re: github release procedure
Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob@ucora.com>:

> 
> You did mention that (1) is more suitable for beta testers, so I 
> ask how is (2) not as suitable?

With (1) you can always use staging, it'll always be the latest beta.
With (2) you'll have to figure out the most recent branch (2.061,
2.062, 2.063).

But I think we can start with (2) and change to (1) later if we think
it's useful. Staging complicates the concept(on a cognitive level) and
it needs additional "human resources" to do the merges into it.


I think I understand the whole concept now. I seems to be a good
solution but it requires some discipline (basing changes on the correct
branches) so we should document it as clearly as possible. I'd strongly
recommend to only base regression fixes on release branches and usual
bug fixes on master (see my other message).
January 04, 2013
Re: github release procedure
On Fri, 2013-01-04 at 00:40 +0100, foobar wrote:
[…]
> Let's clarify a bunch of things here:
> 1. There is no overhead with maintaining many feature brunches.

Which is why Git is superior to Mercurial for the sort of process going
on here.

> 2. Feature branches can be safely removed after their contents 
> have been merged. This is useful for avoiding clutter.

See above :-)

> 3. There is no need to re-review the _same_ code on each branch 
> it is merged to. That's just plain stupid.

I am not so sure on this. A changeset in a different context is a
different changeset, which is why Mercurial is the way it is and why it
has some benefits overf Git.  The effect of a code change in one branch
is not necessarily the same as the effect of the same change in a
different branch. However review is not the right tool here, CI is. Each
and every branch in the mainline must have a CI listener that runs all
the code on every commit.

Branches in the mainline are CVCS thinking, not wrong, but must be
treated as an integral part of the project, hence the CI requirement.
Branches held in forks of the mainline have no such requirement, but
people should run all the tests after each commit as a matter of
professionalism. Others can of course pull these feature branches into
their own fork repository for working on. This is the DVCS way.
Eventually of course commit to the mainline is required otherwise a
project fork has been made.

> 4. Chery picking is a HORRIBLE idea for managing releases and 
> should only be used on development. Chery-picking COPIES commits 
> and thus loses the connections in the history graph.

Agreed.

> 5. There is no problem developing bug_fixes on separate topic 
> branches.

See 1 above. Git makes this trivially easy.

> 6. There is no problem merging back to master.

See 1 above. Git makes this trivially easy.

> 7. Let me say this again. Merges are cheap and easy in git. So 
> are re-merges. Avoiding merges is an artifact of client-server 
> designs such as SVN and is NOT compatible with Git's conceptual 
> model.

It is important to note that Mercurial has different properties again,
so leads to different workflow. Given the choice has been Git then Git
workflows apply.

Every developer should be keeping master as a mirror of the mainline
master and working on feature branches. When a changeset is accepted
after review, the committer commits to the mainline and then everyone
can pull and garbage collect, and in the process delete the feature
branch they have.

I tend to fork the mainline repository locally and also on GitHub and
then manually link my two forks. This means "git pull" locally pulls all
branches from the mainline repository. I work on feature branches pushed
to my GitHub fork, people can then pull them as they wish. You have to
remember to fetch from all the remote branches you are tracking. I can
then make a pull request on my GitHub fork and still continue to track
the mainline locally.

> I suggest to read about how git works and how best to utilize its 
> features, and NOT apply common wisdom from CVS/SVN/RCS which are 
> based on a _completely different_ conceptual model.

Indeed.

Star model has a role in DVCS, the mainline repository is the mainline
from which all else stems. But not all traffic has to be routed through
it with Git. People can share feature branches between themselves to
collaborate to create a pull request without ever having to use the
mainline. That is the whole point.

It means that the mainline committer role is not to do all the
development work, but is to coordinate and mentor activity which they
then review and when happy merge into mainline.

No-one should ever be doing development on master.

No-one should ever push to mainline without someone at least knowing
they are about to do it and have previously checked the work.

No mainline repository should ever be without a full CI.

http://d.puremagic.com/test-results/

Looks like part of a solution.  It should also provides artefacts for
download and install as nightly releases.  It assumes there is only
master in the repository. If there are any other branches in the
mainline repository then where are the CIs for those?

In my clone of DMD I see master, staging, dmd-1.x branches. Given DMD is
a single branch development, I don't see a need for staging. I shall
have to go through all the email to find the rationale, unless someone
has a short paragraph rationale.

What I find really bizarre is that the tags v2.061 and v1.076 are on
branches other than master. To me the whole Git history looks a mess,
essentially broken. I suggest some serious rebasing and merging to make
the history consistent with the version numbering.

The questions to ask:

If I want to create a feature branch, which branch do I use as a base?

Is staging just the 2.x maintenance branch?

When do changes on master make it to staging?

Should staging and dmd-1.x branch HEADS be moved forwards on the branch?

Sorry if this is all answered somewhere, just point me at the URL where
the answers are explained.  

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
January 04, 2013
Re: github release procedure
Weird. While I was reading "Next focus: PROCESS" topic and 
http://wiki.dlang.org/Proposed_new_D_development_process entries, 
I was completely convinced that switch to faster release model is 
proposed, so that master replaces staging the very moment one 
version is released (and it is the same moment as testing of next 
release begins). Plus once a year or so release is marked LTS and 
cherry-pick bug-fixes until next LTS is out. Was my understanding 
wrong?

Anyway, now I am reading all those comments and it feels absurdly 
like no one has discussed this stuff few weeks ago and different 
topic and argument has started from beginning. This is madness.
1 2 3 4 5 6 7
Top | Discussion index | About this forum | D home