Hello,


I was talking yesterday with Walter and Brad about enacting a build farm and the build master who would tend to it.

The basic idea is to amass a number of remote machines (real and/or virtual) running various OSs and use them for building and testing recent commits of dmd and phobos. The website will have a "cutting-edge" downloadable package that contains the most recent successful build. At the end of the day I want us to get to the point where I type:

% make world

and everything is built from source all the way to uploading the installers for all OSs to the website and updating html files accordingly.

Walter and I agreed that wine should be adequate for building and testing on Windows, which is helpful because we can use the Unix command line tools throughout.

The build master will act as a coordinator of check ins, reviews, builds, test harnesses, and probably more. Walter is currently handling these roles, and he would be relieved to pass them on to someone reliable.

Any ideas, proposals, contributions and candidates are welcome!


Andrei

Hello Andrei,

> Hello,
> 
> I was talking yesterday with Walter and Brad about enacting a build
> farm and the build master who would tend to it.
>

I'd be interested in partaking in such an endeavor. I was actually thinking of doing something related (automatic regression testing). I don't think I'd be a good candidate for maintaining the whole things but I'd be willing to help.

> The basic idea is to amass a number of remote machines (real and/or
> virtual) running various OSs and use them for building and testing
> recent commits of dmd and phobos. The website will have a
> "cutting-edge" downloadable package that contains the most recent
> successful build. At the end of the day I want us to get to the point
> where I type:
> 
> % make world
> 
> and everything is built from source all the way to uploading the
> installers for all OSs to the website and updating html files
> accordingly.
> 
> Walter and I agreed that wine should be adequate for building and
> testing on Windows, which is helpful because we can use the Unix
> command line tools throughout.
> 
> The build master will act as a coordinator of check ins, reviews,
> builds, test harnesses, and probably more. Walter is currently
> handling these roles, and he would be relieved to pass them on to
> someone reliable.
> 
> Any ideas, proposals, contributions and candidates are welcome!
> 
> Andrei
> 
-- 
... <IXOYE><

On 08/06/2010 12:00 AM, BCS wrote:
> Hello Andrei,
>
>> Hello,
>>
>> I was talking yesterday with Walter and Brad about enacting a build
>> farm and the build master who would tend to it.
>>
>
> I'd be interested in partaking in such an endeavor. I was actually
> thinking of doing something related (automatic regression testing). I
> don't think I'd be a good candidate for maintaining the whole things but
> I'd be willing to help.

That's great, thanks for volunteering. Automatic regression testing sounds like a good starting point, feel free to start on it as you find fit and post your progress. You may want to get familiar with the build and test processes of dmd and phobos if you aren't.

In terms of machines, if anyone could volunteer ssh accounts to stable machines (e.g. preferably not laptops and not connected via dynamic IP, though the latter is satisfactorily fixed by dyndns.org), please chime in!

Andrei

On 2010-08-06 03:59, Andrei Alexandrescu wrote:
> Hello,
>
>
> I was talking yesterday with Walter and Brad about enacting a build farm
> and the build master who would tend to it.
>
> The basic idea is to amass a number of remote machines (real and/or
> virtual) running various OSs and use them for building and testing
> recent commits of dmd and phobos. The website will have a "cutting-edge"
> downloadable package that contains the most recent successful build. At
> the end of the day I want us to get to the point where I type:
>
> % make world
>
> and everything is built from source all the way to uploading the
> installers for all OSs to the website and updating html files accordingly.
>
> Walter and I agreed that wine should be adequate for building and
> testing on Windows, which is helpful because we can use the Unix command
> line tools throughout.
>
> The build master will act as a coordinator of check ins, reviews,
> builds, test harnesses, and probably more. Walter is currently handling
> these roles, and he would be relieved to pass them on to someone reliable.
>
> Any ideas, proposals, contributions and candidates are welcome!
>
>
> Andrei

Sounds like a good idea.

-- 
/Jacob Carlborg

Hello Andrei,

> On 08/06/2010 12:00 AM, BCS wrote:
> 
>> Hello Andrei,
>> 
>>> Hello,
>>> 
>>> I was talking yesterday with Walter and Brad about enacting a build
>>> farm and the build master who would tend to it.
>>> 
>> I'd be interested in partaking in such an endeavor. I was actually
>> thinking of doing something related (automatic regression testing). I
>> don't think I'd be a good candidate for maintaining the whole things
>> but I'd be willing to help.
>> 
> That's great, thanks for volunteering. Automatic regression testing
> sounds like a good starting point, feel free to start on it as you
> find fit and post your progress. You may want to get familiar with the
> build and test processes of dmd and phobos if you aren't.
> 
> In terms of machines, if anyone could volunteer ssh accounts to stable
> machines (e.g. preferably not laptops and not connected via dynamic
> IP, though the latter is satisfactorily fixed by dyndns.org), please
> chime in!

Where will the build system reside? in the same repo and DMD or Phobos or in it's own?

-- 
... <IXOYE><

Andrei Alexandrescu wrote:

> Hello,
> 
> 
> I was talking yesterday with Walter and Brad about enacting a build farm and the build master who would tend to it.
> 
> The basic idea is to amass a number of remote machines (real and/or virtual) running various OSs and use them for building and testing recent commits of dmd and phobos. The website will have a "cutting-edge" downloadable package that contains the most recent successful build. At the end of the day I want us to get to the point where I type:
> 
> % make world
> 
> and everything is built from source all the way to uploading the installers for all OSs to the website and updating html files accordingly.
> 
> Walter and I agreed that wine should be adequate for building and testing on Windows, which is helpful because we can use the Unix command line tools throughout.
> 
> The build master will act as a coordinator of check ins, reviews,
> builds, test harnesses, and probably more. Walter is currently handling
> these roles, and he would be relieved to pass them on to someone reliable.
> 
> Any ideas, proposals, contributions and candidates are welcome!
> 
> 
> Andrei

Have you considered going with an existing continuous integration package, such as Hudson? I am in the process of setting one up (but using CruiseControl.NET), it does offer some extra value at the expense of an intial setup cost. The benefit of such system are better reporting, automated builds and notifications. Hudson should be able to integrate information from svn and bugzilla for example.

http://hudson-ci.org/

Andrei Alexandrescu napisał:

> Hello,
> 
> 
> I was talking yesterday with Walter and Brad about enacting a build farm and the build master who would tend to it.
> 
> The basic idea is to amass a number of remote machines (real and/or virtual) running various OSs and use them for building and testing recent commits of dmd and phobos. The website will have a "cutting-edge" downloadable package that contains the most recent successful build. At the end of the day I want us to get to the point where I type:
> 
> % make world
> 
> and everything is built from source all the way to uploading the installers for all OSs to the website and updating html files accordingly.
> 
> Walter and I agreed that wine should be adequate for building and testing on Windows, which is helpful because we can use the Unix command line tools throughout.
> 
> The build master will act as a coordinator of check ins, reviews,
> builds, test harnesses, and probably more. Walter is currently handling
> these roles, and he would be relieved to pass them on to someone reliable.
> 
> Any ideas, proposals, contributions and candidates are welcome!

An idea: could the assembled farm be reused to build other D projects? Being able to easily run thorough unittests on a variety of machines would be a massive boost in quality of D's software ecosystem.


Tomek

BCS wrote:
> Hello Andrei,
> 
>> On 08/06/2010 12:00 AM, BCS wrote:
>>
>>> Hello Andrei,
>>>
>>>> Hello,
>>>>
>>>> I was talking yesterday with Walter and Brad about enacting a build
>>>> farm and the build master who would tend to it.
>>>>
>>> I'd be interested in partaking in such an endeavor. I was actually
>>> thinking of doing something related (automatic regression testing). I
>>> don't think I'd be a good candidate for maintaining the whole things
>>> but I'd be willing to help.
>>>
>> That's great, thanks for volunteering. Automatic regression testing
>> sounds like a good starting point, feel free to start on it as you
>> find fit and post your progress. You may want to get familiar with the
>> build and test processes of dmd and phobos if you aren't.
>>
>> In terms of machines, if anyone could volunteer ssh accounts to stable
>> machines (e.g. preferably not laptops and not connected via dynamic
>> IP, though the latter is satisfactorily fixed by dyndns.org), please
>> chime in!
> 
> Where will the build system reside? in the same repo and DMD or Phobos or in it's own?

I suggest you start experimenting in a separate project.

Andrei

Tomek Sowiński wrote:
> Andrei Alexandrescu napisał:
> 
>> Hello,
>>
>>
>> I was talking yesterday with Walter and Brad about enacting a build farm
>> and the build master who would tend to it.
>>
>> The basic idea is to amass a number of remote machines (real and/or
>> virtual) running various OSs and use them for building and testing
>> recent commits of dmd and phobos. The website will have a "cutting-edge"
>> downloadable package that contains the most recent successful build. At
>> the end of the day I want us to get to the point where I type:
>>
>> % make world
>>
>> and everything is built from source all the way to uploading the
>> installers for all OSs to the website and updating html files accordingly.
>>
>> Walter and I agreed that wine should be adequate for building and
>> testing on Windows, which is helpful because we can use the Unix command
>> line tools throughout.
>>
>> The build master will act as a coordinator of check ins, reviews,
>> builds, test harnesses, and probably more. Walter is currently handling
>> these roles, and he would be relieved to pass them on to someone reliable.
>>
>> Any ideas, proposals, contributions and candidates are welcome!
> 
> An idea: could the assembled farm be reused to build other D projects? Being able to easily run thorough unittests on a variety of machines would be a massive boost in quality of D's software ecosystem.

I think that's a great idea. We only need the appropriate people power and computer power. In an ideal world dsource.org would grow to provide that, but my perception is that Brad A. can only put this much time into it (which is a lot btw).

Andrei

Hello Andrei,

On Thu, Aug 5, 2010 at 6:59 PM, Andrei Alexandrescu wrote:
> and everything is built from source all the way to uploading the installers for all OSs to the website and updating html files accordingly.
>

How about CDash?  http://www.cdash.org/cdash/project/about.html CMake+CTest+CDash provide integrated build, test and reporting.  For example, here the dashboard for CDash itself: http://www.cdash.org/CDash/index.php?project=CDash

That said, I have not tried CDash myself, though I'm a user and fan of the first two pieces: CMake and CTest.  Speaking of which, I'm working on CMake for D2, which seems to have lost its maintainer.

Also, I highly recommend using virtual machines rather than bare metal
for a build farm -- easier and much more economical.
Regards,
-steve