On Tuesday, 2 June 2020 at 08:22:03 UTC, Seb wrote:
> On Tuesday, 2 June 2020 at 06:17:50 UTC, mw wrote:
>> E.g. https://code.dlang.org/ has total 1808 packages. Given some packages are no longer maintained, let's add the top 500 (or 1000) most recently updated packages to the compiler test suite: make sure at least these top 500 can be compiled (weak guarantee), and their `dub test` all pass (strong guarantee), before making new compiler releases.
>
> We're already doing the strong guarantee (dub test) continuously ;-)
> Check the buildkite.com/dlang or a buildkite link on any of the main D repos.
> At the moment it's about 50 projects, but that's already a huge pain to maintain as some of these projects are hard to build, relied on network to test, had spurious failures due to randomness in the testsuite, rely on very specific OS features (or versions) and every now and then a new version is released which either introduces some of the above or just has an outright problem with the testsuite.

Package owners can help out by setting up weekly or nightly cron jobs that run the package test suite on different platform and compiler combinations, including nightly and beta builds. For some packages this can generate a broader set of tests than are run in the project tester.

On Tuesday, 2 June 2020 at 08:44:56 UTC, Jon Degenhardt wrote:
> On Tuesday, 2 June 2020 at 08:22:03 UTC, Seb wrote:
>> On Tuesday, 2 June 2020 at 06:17:50 UTC, mw wrote:
>>> E.g. https://code.dlang.org/ has total 1808 packages. Given some packages are no longer maintained, let's add the top 500 (or 1000) most recently updated packages to the compiler test suite: make sure at least these top 500 can be compiled (weak guarantee), and their `dub test` all pass (strong guarantee), before making new compiler releases.
>>
>> We're already doing the strong guarantee (dub test) continuously ;-)
>> Check the buildkite.com/dlang or a buildkite link on any of the main D repos.
>> At the moment it's about 50 projects, but that's already a huge pain to maintain as some of these projects are hard to build, relied on network to test, had spurious failures due to randomness in the testsuite, rely on very specific OS features (or versions) and every now and then a new version is released which either introduces some of the above or just has an outright problem with the testsuite.
>
> Package owners can help out by setting up weekly or nightly cron jobs that run the package test suite on different platform and compiler combinations, including nightly and beta builds. For some packages this can generate a broader set of tests than are run in the project tester.

The main problem is replicating Travis' infrastructure. Anything that doesn't rely on the CI of the project is bound to get outdated. And for the moment we rely on the parsing the testsuite from the `.travis.yml`. But the runners have their own environment (provisioned) and there's no way for a project to install its dependencies / specific tools (e.g. the lack of meson have led to countless issues). You also have to pin package versions to a single one.

Personally I'd like to have support for Github CI and explore having the test-suite run cross-platform when possible (for the moment we only test Linux).

On 01.06.20 21:38, Johan wrote:
> Hi all,
>    Please be more careful when changing the language behavior.
> 
> The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect. We were lucky to detect one case. There is no warning/deprecation whatsoever to warn me about any other potential breakages in a heavily templated 400kloc codebase with 212 instances of alias this.
> 
> The breakage introduced was very obvious, and warning about it is not difficult either. We need to raise the bar for accepting changes like this. The code on buildkite project tester is (fortunately) not the only D code in the world.
> 
> Again, please be more careful when changing language behavior.
> 
> Thanks,
>    Johan
> 

Another example: https://github.com/dlang/dmd/commit/0f2889c3aa9fba5534e754dade0cae574b636d55#r36465193

On Monday, 1 June 2020 at 19:38:30 UTC, Johan wrote:
> Hi all,
>   Please be more careful when changing the language behavior.
>
> The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect. We were lucky to detect one case. There is no warning/deprecation whatsoever to warn me about any other potential breakages in a heavily templated 400kloc codebase with 212 instances of alias this.
>
> The breakage introduced was very obvious, and warning about it is not difficult either. We need to raise the bar for accepting changes like this. The code on buildkite project tester is (fortunately) not the only D code in the world.
>
> Again, please be more careful when changing language behavior.
>
> Thanks,
>   Johan

is it that impossible to have a guy that works on regression ?

On Tuesday, 2 June 2020 at 09:56:41 UTC, Mathias LANG wrote:
> On Tuesday, 2 June 2020 at 08:44:56 UTC, Jon Degenhardt wrote:
>> Package owners can help out by setting up weekly or nightly cron jobs that run the package test suite on different platform and compiler combinations, including nightly and beta builds. For some packages this can generate a broader set of tests than are run in the project tester.
>
> The main problem is replicating Travis' infrastructure. Anything that doesn't rely on the CI of the project is bound to get outdated. And for the moment we rely on the parsing the testsuite from the `.travis.yml`. But the runners have their own environment (provisioned) and there's no way for a project to install its dependencies / specific tools (e.g. the lack of meson have led to countless issues). You also have to pin package versions to a single one.

BuildKite certainly improved the overall situation, but it's obviously not scalable and can already be a problem if compiler changes require a new version of one of the tested projects. E.g., I've been waiting for months for a new Ocean release to be finally published with a tiny adaptation, and now that Sociomantic isn't anymore, I'm not sure I'll see that release happen this year at all.

Anyway, projects contributing themselves to regression testing via regularly triggered builds using the latest beta would be very much appreciated. But that doesn't work for DMD, where the final release usually happens 1-2 weeks after the 1st beta - that won't be enough for people to adapt their projects to language/library changes and investigate any apparent regressions.

On Tuesday, 2 June 2020 at 13:40:08 UTC, kinke wrote:
>
> Anyway, projects contributing themselves to regression testing via regularly triggered builds using the latest beta would be very much appreciated. But that doesn't work for DMD, where the final release usually happens 1-2 weeks after the 1st beta - that won't be enough for people to adapt their projects to language/library changes and investigate any apparent regressions.

Instead of betas, which I think are more useful for manual testing before a release, I'd recommend to set up a dimension / a cron with nightly builds.
Github CI and the install script support them (hence so does travis-ci), by using `dmd-master` / `ldc-master` and `ldc-latest-ci` / `dmd-nightly`, respectively.

On Tuesday, 2 June 2020 at 14:13:20 UTC, Mathias LANG wrote:
> Instead of betas, which I think are more useful for manual testing before a release, I'd recommend to set up a dimension / a cron with nightly builds.
> Github CI and the install script support them (hence so does travis-ci), by using `dmd-master` / `ldc-master` and `ldc-latest-ci` / `dmd-nightly`, respectively.

Master is waaay too unstable IMHO. I definitely wouldn't use it for projects of mine, regardless how small.

On Tuesday, 2 June 2020 at 14:50:41 UTC, kinke wrote:
> On Tuesday, 2 June 2020 at 14:13:20 UTC, Mathias LANG wrote:
>> Instead of betas, which I think are more useful for manual testing before a release, I'd recommend to set up a dimension / a cron with nightly builds.
>> Github CI and the install script support them (hence so does travis-ci), by using `dmd-master` / `ldc-master` and `ldc-latest-ci` / `dmd-nightly`, respectively.
>
> Master is waaay too unstable IMHO. I definitely wouldn't use it for projects of mine, regardless how small.

If the project is already on BuildKite then nightly is actually a pretty solid option, since upstream developers are forced to either not break BuildKite projects, or adapt them, before introducing a breaking change (which can even be a just a new warning).
Even for a few of the projects that I maintain that are not on BuildKite I have used nightly builds without much issues. I think it mainly depends on how much one is willing to be on the bleeding edge. Of course, projects of the scale of Weka, are a different matter all together.

On 6/1/2020 11:53 PM, mw wrote:
> If you seriously want to fix the multiple inheritance problem: freeze D2 prod, and make D3 dev, I also mean it seriously.


Yeah, well, even more breakage :-/

On Wednesday, 3 June 2020 at 02:25:23 UTC, Walter Bright wrote:
> On 6/1/2020 11:53 PM, mw wrote:
>> If you seriously want to fix the multiple inheritance problem: freeze D2 prod, and make D3 dev, I also mean it seriously.
>
>
> Yeah, well, even more breakage :-/

The difference is: for a new *major* version release, breakage are more acceptable and tolerable by the users. Python 3  doesn’t have backward compatibility with Python 2, and there’s even a tool called ‘2to3’ coming with Python 3 to help people migrate. It’s a hard decision, but in the long run, it helps Python to be a more consistent language.

With minor version release, it’s difficult to get things fixed, both for you and for the language users. Just as both parties have expressed in this very thread.

In my view, D2 has already been feature rich enough for people to do their work. What they want most is compiler stability, not any more new language features. By freezing D2, they will have a more stable platform to get their work done without worrying about breakage. And for you, it will buy you more time, e.g. a few years, to design carefully the next major features / fixes you want introduce in D3.


Currently,
https://dlang.org/changelog/release-schedule.html

— New release are published every two months, on the first day of every uneven month.
— Two weeks before a new release master is merged into stable and a first beta is released.

I think this release schedule is too frequent for a mature v2 language, it’s not application software, but language compiler!


No wonder kinke said:

> Master is waaay too unstable IMHO. I definitely wouldn't use it for projects of mine, regardless how small.


I hope you can think about this D2 / D3 suggestion.