D stability testing framework

The framework would pull in a specified version of dmd, druntime and Phobos, and build them; and then using those, attempt to pull in and build a selection of 3rd-party libraries and to run test code those libraries would provide. It would then report the number of build failures and runtime/unittest failures and try to classify them according to whether they are due to a change in the _language_, a change in the runtime, or a change in Phobos.

The idea would be to get reliable statistics on what breaking changes are causing what degree of pain (and why) for D users. Currently we have arguments over how stable the language is and what kind of breakages are or aren't acceptable. With a good enough selection of 3rd-party code, it might be possible to quantify the prospective impact of a change.

Ideally the test code could be provided simply by providing a list of git repositories and branches to test. A practical stumbling block might be different build systems etc., so it might be necessary to have some kind of standardized build system the testing framework could expect (e.g. a testbuild.d script, just as some libraries currently provide a build.d script).

The goal would be to provide mutual benefit for the D language and providers of public test code -- patches to D, druntime or Phobos could be tested to see the extent of breakage they cause, providers of test code could get automated early warning that a change to the D frontend, druntime or Phobos is going to impact their project (and a framework to test patches designed to cope with those breaking changes). It should also be possible to run the testing framework on one's personal machine (so for example, downstream users could test their code against latest-off-GitHub versions of dmd, druntime and Phobos without having to make their code publicly available).

I don't know if this idea is practical in reality -- it might be difficult to distinguish between breakages caused by changes to D and breakages caused by other problems such as incorrect testbuild scripts, etc. It's also cheeky as I definitely don't have the knowhow or time to do this myself. But I thought I'd throw it out there to see if it's an idea worth pursuing by someone.

Best wishes,

-- Joe

On 05/23/2013 05:11 PM, Dicebot wrote: > Why not make one step further and make it part of current CI suite? And add as a separate stat table to automatic pull request tester. Simple and useful. Obviously it could be incorporated into that -- my concern was that it might involve too big a test run to operate on a per-patch basis. It should still be possible to use as a standalone framework, though, because it would be useful not just for dmd/druntime/phobos devs to test their work but also for authors of 3rd-party libraries to test their code against latest development version of D, without publishing their code or sending it to some centralized server.

On 5/23/13 10:30 AM, Joseph Rushton Wakeling wrote: > Hello all, > > Having listened to Andrei and Walter's Q&A and read some of the discussion > surrounding it, an idea occurred to me. > > How about leveraging the selection of 3rd-party D code out there to provide a > testing framework for D's stability as a language? I think that's a great idea. What would be a seed of 1-3 projects? I wonder how difficult it would be for Brad to set up some beta testing scripts on the build machines. Andrei

On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote: > I think that's a great idea. What would be a seed of 1-3 projects? I wonder how difficult it would be for Brad to set up some beta testing scripts on the build machines. Will happily offer my own code, though I currently only have 1 public D project and it's probably not extensive enough to be useful. Time to get that graph library off the ground ... :-) But as suggestions, how about vibe.d, GtkD and SciD? I'd also add QtD except that as far as I can tell, it's not being maintained right now. :-(

On 5/23/13 1:54 PM, Joseph Rushton Wakeling wrote: > On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote: >> I think that's a great idea. What would be a seed of 1-3 projects? I wonder how >> difficult it would be for Brad to set up some beta testing scripts on the build >> machines. > > Will happily offer my own code, though I currently only have 1 public D project > and it's probably not extensive enough to be useful. Time to get that graph > library off the ground ... :-) > > But as suggestions, how about vibe.d, GtkD and SciD? I'd also add QtD except > that as far as I can tell, it's not being maintained right now. :-( Sounds good. I'll discuss with Walter and Brad. Andrei

On 5/23/13 11:35 AM, Andrei Alexandrescu wrote: > On 5/23/13 1:54 PM, Joseph Rushton Wakeling wrote: >> On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote: >>> I think that's a great idea. What would be a seed of 1-3 projects? I >>> wonder how >>> difficult it would be for Brad to set up some beta testing scripts on >>> the build >>> machines. >> >> Will happily offer my own code, though I currently only have 1 public >> D project >> and it's probably not extensive enough to be useful. Time to get that >> graph >> library off the ground ... :-) >> >> But as suggestions, how about vibe.d, GtkD and SciD? I'd also add QtD >> except >> that as far as I can tell, it's not being maintained right now. :-( > > Sounds good. I'll discuss with Walter and Brad. > > Andrei I think this ought to be prototyped independent of the current systems. It's got _some_ commonality. It's also likely to be a lot of work, which I don't have time for.

On 2013-05-23 19:54, Joseph Rushton Wakeling wrote: > Will happily offer my own code, though I currently only have 1 public D project > and it's probably not extensive enough to be useful. Time to get that graph > library off the ground ... :-) > > But as suggestions, how about vibe.d, GtkD and SciD? I'd also add QtD except > that as far as I can tell, it's not being maintained right now. :-( DWT and Tango are pretty large projects. -- /Jacob Carlborg

On 05/23/2013 09:06 PM, Jacob Carlborg wrote: > DWT and Tango are pretty large projects. I'm not familiar with them and didn't know their maintenance status, otherwise I'd have suggested both. I was also wondering about your dvm and/or orbit ... ?

D:YAML might be of use if needed. There's been no release in recent past (featureset has not changed, which I'm still planning for a release), but I've been maintaining its compatibility with current DMD for a while and I have no intention to stop in forseeable future as I use it in pretty much every project I start. It's somewhere around 10-20kLOC. https://github.com/kiith-sa/D-YAML Also, Derelict3 might be a good idea. Derelict has been actively maintained for pretty much most of D's history.

Forums