On Fri, Aug 23, 2013 at 11:07 AM, Walter Bright <newshound2@digitalmars.com> wrote:

That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them?

The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse.

AFAIR both Clang and GCC have entire third-party projects in their test suite. I know that at least SQLite is part of both, and that's a pretty big project. If I recall correctly, GCC releases are blocked on successfully compiling the Linux kernel, all of Firefox and I possibly Qt. The third-party project tests need to finish without failures as well.

My recollection is a bit vague here though.

Now, do they compile and run all the tests for these projects on every commit or make sure nothing has broken just before making a new release, I don't know. But I do know that it's the latter at the very least.