Hi all,
  Please be more careful when changing the language behavior.

The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect. We were lucky to detect one case. There is no warning/deprecation whatsoever to warn me about any other potential breakages in a heavily templated 400kloc codebase with 212 instances of alias this.

The breakage introduced was very obvious, and warning about it is not difficult either. We need to raise the bar for accepting changes like this. The code on buildkite project tester is (fortunately) not the only D code in the world.

Again, please be more careful when changing language behavior.

Thanks,
  Johan

On Monday, 1 June 2020 at 19:38:30 UTC, Johan wrote:
> Hi all,
>   Please be more careful when changing the language behavior.
>
> The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect. We were lucky to detect one case. There is no warning/deprecation whatsoever to warn me about any other potential breakages in a heavily templated 400kloc codebase with 212 instances of alias this.
>
> The breakage introduced was very obvious, and warning about it is not difficult either. We need to raise the bar for accepting changes like this. The code on buildkite project tester is (fortunately) not the only D code in the world.
>
> Again, please be more careful when changing language behavior.
>
> Thanks,
>   Johan

Wholeheartedly agree!

On Monday, 1 June 2020 at 19:38:30 UTC, Johan wrote:
> Hi all,
>   Please be more careful when changing the language behavior.
>
> The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect.

An example in our standard lib:
https://issues.dlang.org/show_bug.cgi?id=20892

-Johan

On Monday, 1 June 2020 at 19:38:30 UTC, Johan wrote:
> Hi all,
>   Please be more careful when changing the language behavior.
>
> The change made by https://github.com/dlang/dmd/pull/9289 silently breaks code in ways that can be very hard to detect. We were lucky to detect one case. There is no warning/deprecation whatsoever to warn me about any other potential breakages in a heavily templated 400kloc codebase with 212 instances of alias this.
>
> The breakage introduced was very obvious, and warning about it is not difficult either. We need to raise the bar for accepting changes like this. The code on buildkite project tester is (fortunately) not the only D code in the world.
>
> Again, please be more careful when changing language behavior.
>
> Thanks,
>   Johan

It appears they tried to come to some sort of plan as to what should be done to notify the user. Not very inspiring that Walter was pinged and he didn't leave any feedback other than a ✔.

I don't think a "simple" warning would have worked. That's a warning that would then have to be manually disabled all the time. It could have been put under the deprecation flag, but still then you'd be flooded with warnings that can't be "fixed" if someone treats deprecations as errors.

This is where it would be nice to have a different versioning scheme than what we have now. You could put breaking changes like these where it's difficult to nicely notify the user. These change could have been deferred to a major version, where potentially breaking changes could be listed so that they can be checked. Then include a flag that does check if a code base uses that pattern anywhere so it can be fixed. A flag like that wouldn't work now as people expect their code to just work. There's only one version of D. They would most likely miss this breaking change, especially since it doesn't appear to have been added to any of the subsequent change logs. Which would help a user, if they did update up a few versions and don't have the time to read everything. There is a lot that could have been done better for sure.

All I can say is you're right, and we handled this badly.

alias this has a lot of problems, mainly because it tries to implement multiple inheritance without thinking it through. I've looked at fixing them, and essentially it is unfixable. We'll just have to live with the way it works, and shouldn't try to modify its behavior again.

Changes to the compiler rely very heavily on the test suite. If cases aren't in the test suite, we don't know when break things.

On Tuesday, 2 June 2020 at 05:32:08 UTC, Walter Bright wrote:
> All I can say is you're right, and we handled this badly.
>
> alias this has a lot of problems, mainly because it tries to implement multiple inheritance without thinking it through. I've looked at fixing them, and essentially it is unfixable. We'll just have to live with the way it works, and shouldn't try to modify its behavior again.
>
> Changes to the compiler rely very heavily on the test suite. If cases aren't in the test suite, we don't know when break things.

In this case, the test case *was* in the test suite, or more precisely in druntime (https://github.com/dlang/druntime/pull/2476). So druntime was changed and is was merged without a user-visible message.

That's why I've been advocating to make *any* change that change user code's behavior, even if it disallow something that shouldn't have been allowed in the first place, to be deprecation first. Because of D unparalleled compile-time introspection capabilities, there is not a single change you can make to the language or any library that *cannot* result in code breakage. For example, adding any public function to a library will break code relying on the symbol not existing (e.g. `!is(typeof(SomeSymbol))`). Now it doesn't mean we should stop adding new things or changing / fixing things, but it shouldn't be done without consideration for the potential breakage. One thing that should be at the very top of the list of "absolutely not going to happen" is silently changing which user code is executed without notification, and that's what happened here.

Having worked at companies that use D, we *had* to have "sync points" where we updated the compiler. From what I saw, Weka is doing the exact same thing. This is because experience have proved countless times that upstream cannot be trusted not to break stuff. I'm glad to say that things have gotten drastically better over the year, and I think Buildkite was a major factor in this. Nevertheless, it's still at a point where an update is rarely a no-op.

On Tuesday, 2 June 2020 at 05:32:08 UTC, Walter Bright wrote:
> All I can say is you're right, and we handled this badly.
>
> alias this has a lot of problems, mainly because it tries to implement multiple inheritance without thinking it through. I've looked at fixing them, and essentially it is unfixable. We'll just have to live with the way it works, and shouldn't try to modify its behavior again.
>
> Changes to the compiler rely very heavily on the test suite. If cases aren't in the test suite, we don't know when break things.

I've read industry users complaining about unstable compiler breaking their existing code base in the past forum threads. I think we need to take this seriously, otherwise D's community will shrink over time.

For the two issues you mentioned:

1) If we do not want to separate two compiler versions (production 2.x vs development 3.x), then we better add some client code base to the current compiler test suite:

E.g. https://code.dlang.org/ has total 1808 packages. Given some packages are no longer maintained, let's add the top 500 (or 1000) most recently updated packages to the compiler test suite: make sure at least these top 500 can be compiled (weak guarantee), and their `dub test` all pass (strong guarantee), before making new compiler releases.


2) for the multiple inheritance problem you try to solve, it actually is solvable: as far as I know, only Eiffel language solved the diamond inheritance problem gracefully, with keywords `rename`, `export`, `undefine`, `redefine` and `select`. Simply put: by default the Eiffel compiler will join the features and only keep one copy, but allow the user *explicitly* using these keywords to define the correct behavior that the user want. (I mailed you a copy of Eiffel language manual many years ago at pre-D1-age, esp. about multiple inheritance. Maybe you still remember? :-)

Here are some online material on Eiffel's multiple inheritance solution:

https://en.wikipedia.org/wiki/Multiple_inheritance

"""
In Eiffel, the ancestors' features are chosen explicitly with select and rename directives. This allows the features of the base class to be shared between its descendants or to give each of them a separate copy of the base class. Eiffel allows explicit joining or separation of features inherited from ancestor classes. Eiffel will automatically join features together, if they have the same name and implementation. The class writer has the option to rename the inherited features to separate them. Multiple inheritance is a frequent occurrence in Eiffel development; most of the effective classes in the widely used EiffelBase library of data structures and algorithms, for example, have two or more parents.[7]
"""

more detail here:
https://www.eiffel.org/doc/eiffel/ET-_Inheritance

On 6/1/2020 11:17 PM, mw wrote:
> 2) for the multiple inheritance problem you try to solve, it actually is solvable: as far as I know, only Eiffel language solved the diamond inheritance problem gracefully, with keywords `rename`, `export`, `undefine`, `redefine` and `select`. Simply put: by default the Eiffel compiler will join the features and only keep one copy, but allow the user *explicitly* using these keywords to define the correct behavior that the user want. (I mailed you a copy of Eiffel language manual many years ago at pre-D1-age, esp. about multiple inheritance. Maybe you still remember? :-)

I meant it is unsolvable without breaking everything.

The trouble was, it was inserted without realizing it was multiple inheritance, meaning its behaviors are ad-hoc and don't make a whole lot of sense when examined carefully.

I know I have that Eiffel manual around here somewhere :-)

I also have Meyers' tome "Object Oriented Software Construction".

On Tuesday, 2 June 2020 at 06:43:40 UTC, Walter Bright wrote:
> I meant it is unsolvable without breaking everything.
>
> The trouble was, it was inserted without realizing it was multiple inheritance, meaning its behaviors are ad-hoc and don't make a whole lot of sense when examined carefully.

Right, if it's done in pre-D1-age ...


> I know I have that Eiffel manual around here somewhere :-)
>
> I also have Meyers' tome "Object Oriented Software Construction".

I'm so glad.

If you seriously want to fix the multiple inheritance problem: freeze D2 prod, and make D3 dev, I also mean it seriously.

On Tuesday, 2 June 2020 at 06:17:50 UTC, mw wrote:
> On Tuesday, 2 June 2020 at 05:32:08 UTC, Walter Bright wrote:
>> All I can say is you're right, and we handled this badly.
>>
>> alias this has a lot of problems, mainly because it tries to implement multiple inheritance without thinking it through. I've looked at fixing them, and essentially it is unfixable. We'll just have to live with the way it works, and shouldn't try to modify its behavior again.
>>
>> Changes to the compiler rely very heavily on the test suite. If cases aren't in the test suite, we don't know when break things.
>
> I've read industry users complaining about unstable compiler breaking their existing code base in the past forum threads. I think we need to take this seriously, otherwise D's community will shrink over time.
>
> For the two issues you mentioned:
>
> 1) If we do not want to separate two compiler versions (production 2.x vs development 3.x), then we better add some client code base to the current compiler test suite:
>
> E.g. https://code.dlang.org/ has total 1808 packages. Given some packages are no longer maintained, let's add the top 500 (or 1000) most recently updated packages to the compiler test suite: make sure at least these top 500 can be compiled (weak guarantee), and their `dub test` all pass (strong guarantee), before making new compiler releases.


We're already doing the strong guarantee (dub test) continuously ;-)
Check the buildkite.com/dlang or a buildkite link on any of the main D repos.
At the moment it's about 50 projects, but that's already a huge pain to maintain as some of these projects are hard to build, relied on network to test, had spurious failures due to randomness in the testsuite, rely on very specific OS features (or versions) and every now and then a new version is released which either introduces some of the above or just has an outright problem with the testsuite.