On 2013-12-04 05:29, Andrew Edwards wrote:

> Tried it but came across the following errors:
>
>
> andrews-mini:osx-release ace$ ~/create_dmd_release v2.065-b1
> --extras=$HOME/localextras-osx --archive-zip --archive-7z
>
> Cloning: git@github.com:D-Programming-Language/dmd.git
>
> The authenticity of host 'github.com (192.30.252.128)' can't be
> established.
>
> RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
>
> Are you sure you want to continue connecting (yes/no)? yes
>
> Warning: Permanently added 'github.com,192.30.252.128' (RSA) to the list
> of known hosts.
>
> Permission denied (publickey).
>
> fatal: Could not read from remote repository.
>
>
>
> Please make sure you have the correct access rights
>
> and the repository exists.
>
> Couldn't do git protocol, falling back to 'https://'...

You need to have your SSH key uploaded to github. See:

https://help.github.com/articles/generating-ssh-keys

-- 
/Jacob Carlborg

On 2013-12-04 06:25, Andrew Edwards wrote:

> My next question is whether or not this can be used on my platform (OSX)
> to build binaries for the other platforms (Linux, Windows, FreeBSD) or
> do I have to be on those platforms to do so? If that is the case, then I
> will need assistance getting for Windows binaries created and it will
> require more time than anticipated to get the the FreeBSD and Linux
> binaries complete because I will be teaching at 8AM and have not yet
> look at the material.

Everything needs to be built on each platform. We don't support true cross-platform builds. The autotester should handle this.

-- 
/Jacob Carlborg

On 12/4/13, 3:09 AM, Jacob Carlborg wrote:
> On 2013-12-03 15:25, Andrew Edwards wrote:
>
>> I am working on a MacMini running OS X v10.9. I have Ubuntu 13.10 Server
>> loaded in VirtualBox and will be using Jordi Sayol's script to build
>> packages for linux/Windows and Jacob Carlborg script for OSX.
>>
>> Both of these scripts require an preexisting release zip and as of this
>> moment, I am unaware of the steps to create that file. I will need some
>> instructions on how to access and run the auto tester if that's what
>> generates the zip or, if it's already automatically created,
>> instructions on how to retrieve and stage it for the build.
>
> Unfortunately there is no publicly available script (or similar) that
> creates the zip. It's Walter how always creates it.
>
> As others have said, the closest you get it probably the script Nick is
> working on: https://github.com/D-Programming-Language/installer/pull/24
>
>> I will be setting up a tag on github today for the first beta release of
>> 2.065 (2.065-b1).
>>
>> Request your corporation/support in ensuring a smooth process.
>
> Ideally there should be a "one-click" tool/script that manages the
> complete release of a new version, including beta and release
> candidates. With one command it should:
>
> * Create a git tag for all involved repositories

Working on that.

> * Instruct the autotester to build the compiler, libraries, installers
> and all tools for each platform based on the newly created tag

Will need access from Brad Roberts. Have yet to receive any kind of response to date.

> * Assemble the cross-platform zip on one of the autotester machines

ditto

> * Upload zip and all installers to the Digital Mars FTP and/or Amazon or
> wherever these things are currently hosted

ditto

> * Generate the changelog from bugzilla. The overview/general information
> should already be present in the changelog at this time

Will need help on this one.

> * Updates the website with the new changelog and links to the new release

ditto

> * Posts a message to the announce newsgroup, twitter, reddit? and any
> social networks we would like to post to

This should be separate. Previous experience (not my own) shows that some days are better posting than others.

> Preferably there should be options to disable any of these steps make it
> possible to just assembly the compiler and tools but not make a proper
> release (update website post to social networks)
>

Makes sense. Will be great when this is complete. In the interim though, we need to be able to perform manual semi-manual production so that the release cycle is not delayed.

On 12/4/13, 3:21 AM, Jacob Carlborg wrote:
> On 2013-12-04 05:29, Andrew Edwards wrote:
>
>>
>> Couldn't do git protocol, falling back to 'https://'...
>
> You need to have your SSH key uploaded to github. See:
>
> https://help.github.com/articles/generating-ssh-keys
>

I did this but it is not working for me. I probably did something wrong.

On 2013-12-04 09:23, Andrew Edwards wrote:

> Working on that.
>
>> * Instruct the autotester to build the compiler, libraries, installers
>> and all tools for each platform based on the newly created tag
>
> Will need access from Brad Roberts. Have yet to receive any kind of
> response to date.
>
>> * Assemble the cross-platform zip on one of the autotester machines
>
> ditto
>
>> * Upload zip and all installers to the Digital Mars FTP and/or Amazon or
>> wherever these things are currently hosted
>
> ditto

I'm hoping it would be possible to create a tool/script without access to the all this.

> Will need help on this one.

I'm pretty sure Walter has a script for this.

> This should be separate. Previous experience (not my own) shows that
> some days are better posting than others.

Ok.

> Makes sense. Will be great when this is complete. In the interim though,
> we need to be able to perform manual semi-manual production so that the
> release cycle is not delayed.

We really need to push for the automate release process. Either you're doing it yourself or delegating to others.

-- 
/Jacob Carlborg

04-Dec-2013 12:23, Andrew Edwards пишет:
> On 12/4/13, 3:09 AM, Jacob Carlborg wrote:
>> On 2013-12-03 15:25, Andrew Edwards wrote:
[snip]
>> * Generate the changelog from bugzilla. The overview/general information
>> should already be present in the changelog at this time
>
> Will need help on this one.

There is a tool for that. Nothing stellar but it does the job:
https://github.com/D-Programming-Language/tools/blob/master/changed.d

Also be sure to contact Andrej Mitrovic he is a mastermind behind nice changelogs we had for the last couple of releases.

-- 
Dmitry Olshansky

On 12/4/13 12:18 AM, Jacob Carlborg wrote:
> On 2013-12-03 20:08, Brad Anderson wrote:
>
>> Historically, Walter has created the release zips (just using the
>> makefile, I believe). As far as I know, work hasn't begun on getting the
>> autotester to roll releases.
>
> I doubt he has a completely automated process of doing it. It has failed too many times.
>
> Having the autotester building and handling everything should be our primary goal.
>

I'm not sure this is really a good goal any more.  The overlap between testing and releasing isn't as big as I originally thought.  Add to that that the very definition of what a release is is in question.  The only overlap that really exists is that there's a set of boxes that exist and that they build in somewhat similar ways.

But there's also a hell of a lot of differences.  Ignoring pulls for the most part, there's little coordination between the platforms in terms of what to build.  It's always "whatever is current" which changes over time.  There's no current automation for how to package a release that's part of the packages being tested (there's some fledgling support inside the packages, there's some tooling that lives off in other places).  There's parts of the releases that have no source and aren't part of what's being tested.  If we stick to having a zip that contains multiple platforms, there's no code to rendezvous multiple platforms together.  The auto-tester covers a smaller set of platforms than release builds are currently done for (example, there's several linux distros that have distro specific packages, not all of which are represented by test boxes).

All in all, while I was of the opinion that the test fleet could serve the role of release builders and pushed that with Walter and Andrei, I'm much less convinced that's the right short term answer right now.  It might become the right answer once a number of the open questions are answered and once the automation for building the releases are improved, but not yet.  I guess the 'have the auto-tester do it' mentality really hand waves away that most of the work still has to be done to make it something that might one day be run automatically.

The auto-tester isn't nearly as magic as I think a lot of people think it is.  It's very little more than a set of clients that:

0) ask a server for what package/branches to checkout and optionally what package/branch to merge
1) checkout a set of git repos
2) for a pull test: merge a branch in a particular repo
3) run make in each
4) run make unittest in each
5) for each of the actions in 1-4, ship the success or failure status and the log file to the results server
6) Repeat forever.

And a server that collects the results and answers the question asked in #0 above.  There's some interesting logic in deciding how to answer that question and a little bit of fun logic to sync between state between github and the server.  But it's not particularly fancy or complicated at the component level.

My current 2 cents,
Brad

On 2013-12-04 20:52, Brad Roberts wrote:

> I'm not sure this is really a good goal any more.  The overlap between
> testing and releasing isn't as big as I originally thought.  Add to that
> that the very definition of what a release is is in question.  The only
> overlap that really exists is that there's a set of boxes that exist and
> that they build in somewhat similar ways.

I have only ever thought that the autotester provides multiple different platforms which we could build the tools and create the releases[1] on.

What I see as the advantage is:

* The autotester is always available

* Not a single person need to have all different platforms to create a release

* We don't need to rely on someone else to build packages for a given platform. He/she may be away for a few days and suddenly we can't get a release for a given platform

* It's reproducible. We're creating the release on the same machines every time

Just look at the last release. It doesn't include pre-compiled binaries for FreeBSD, because Walter's machine apparently can't build it anymore without running out of memory. That must never happen.

> But there's also a hell of a lot of differences.[SNIP]

I'm not saying that the autotester will just magically be able to build releases. I'm saying it provides the platforms we need (ok, seems to be missing one or two Linux distributions) to be able to build releases.

As you say, we don't yet have an automated process of creating releases. But when we're creating that (writing a script/tool) it need to support building on several different machines, then be able to collect the result (cross-platform zip). We don't want to create a script that automates the release, then change all that when we won't it to run on the autotester machines.

BTW, I'm not sure we even can create an automate process without something like the autotester. We need to be able to build on all supported platforms. Sure, someone else might have a complete fleet of machines. It's not the autotester itself that is important, it's all the machines it runs on.

[1] When I say "release", in this case I'm referring to building the compiler, all the tools and libraries, packaging it in a platform specific installer and cross-platform zip.

-- 
/Jacob Carlborg

On 2013-12-04 20:28, Dmitry Olshansky wrote:

> There is a tool for that. Nothing stellar but it does the job:
> https://github.com/D-Programming-Language/tools/blob/master/changed.d

Great.

> Also be sure to contact Andrej Mitrovic he is a mastermind behind nice
> changelogs we had for the last couple of releases.

Yeah, that still needs to be manually written.

-- 
/Jacob Carlborg

On Tuesday, 3 December 2013 at 22:24:45 UTC, Jacob Carlborg wrote:
> Why can't that be automatable? We do have the servers running the tests. The idea is that they should build them.

Conflating binaries for multiple platforms is the problem here.