On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
> the compiler must warn D programmers that their declarations need to be reviewed for compatibility with @safe-by-default. Whether they were reviewed before that or not makes no difference.

I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.)

On Monday, 25 May 2020 at 17:09:50 UTC, Clarice wrote:
> On Monday, 25 May 2020 at 16:47:50 UTC, Paul Backus wrote:
>> the compiler must warn D programmers that their declarations need to be reviewed for compatibility with @safe-by-default. Whether they were reviewed before that or not makes no difference.
>
> I'm not an engineer; I'm not learned in CS. So some clarification on this would be nice: if you're pulling in C code, or any code written in an unsafe language, wouldn't the developers doing such a thing already be aware they need to do their due diligence? Foreign code isn't just going to magically appear in your codebase, right? Maybe through dependencies, but one should check those too, eh? (And if the dependency tree is needlessly large, then the language has another, maybe bigger, problem à la the NPM ecosystem.)

Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default.

In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime:

https://github.com/dlang/druntime/pull/3117

Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling.

[1] https://github.com/dlang/dmd/pull/11176

On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
> Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default.
>
> In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime:
>
> https://github.com/dlang/druntime/pull/3117
>
> Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling.
>
> [1] https://github.com/dlang/dmd/pull/11176


Can we declare the new D compiler version 3.0 for @safe-as-default? and keep version 2.x compiler for the old @system-as-default?

In industries, it is a big-NO-NO to break exiting *working* code base.

I've read some past threads talking about the instability of compilers (due to new feature introduced), e.g. in this on_leaving_d thread:

https://forum.dlang.org/thread/ncbawciyybdksecurmsc@forum.dlang.org?page=1

https://gitlab.com/mihails.strasuns/blog/blob/master/articles/on_leaving_d.md

Let me quote what D's *industry users* (whose trademark is on the front page of dlang.org) have said 2 years ago:

"""
But when it comes to be boring and predictable, D fails terribly. You can't
assume that released features will become 100% usable (no bugs, sufficient docs
and tests, well-defined integration with other language features) even few
years after initial release. You can't assume that next compiler upgrade won't
suddenly break your project or any of its transitive dependencies. You can't
assume that any compiler version will be maintained for more than few months.
You can't assume there is any control over how declared vision
documents get executed in practice. You
can't trust any promises from language authors because they don't keep any track
of those. It is anarchy driven development in all its glory.
"""


I understand the desire of Walter to continue innovation and I appreciate it, and I also think this change from @system to @safe-as default is an important step for D's future direction, as I commented in my thread yesterday:

https://forum.dlang.org/post/brvexgjuqkbebizdmfjj@forum.dlang.org

"""
-- the recent change @safe as default opt-in is good thing: without a @nogc phobos, nobody is going to write @system level software using D. I think this @safe change means the shift of D's focus from system software to application software. And I'm glad to see the the template project that dub generate starts with "app.d", which is the right direction for D.
"""

But on the other hand, we also need to take care of our existing D users, esp industry users, are they going to be happy about breaking their working code base with a new compiler version?

For example, we can define a metric: say, how many percent of the packages on https://code.dlang.org/ will continue work as-it-is *without* to make any change with the new @safe-as-default compiler. If > 95% of these packages (the 68–95–99.7 three-sigma rule of thumb https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) require not any code change, then go ahead, release the new @safe-as-default compiler as 2.093.

Otherwise, I'd suggest to keep 2 versions of compiler:

-- version 2.x as stable compiler for (industry) users to use in production, and will only provide bug fixes, but no new language feature (changes) to the compiler. (And I believe the 2.x version D language has already be *feature rich enough* to beat all those Java, C#, and Rust for people to use in production software).

-- version 3.x for Walter to experiment and explore new ideas, and warn all the users that version 3 is unstable compiler feature-wise, use-it-at-your-own-risk, your code base may need major change in the future.

Thoughts?

On Monday, 25 May 2020 at 18:39:16 UTC, mw wrote:
>
> Can we declare the new D compiler version 3.0 for @safe-as-default? and keep version 2.x compiler for the old @system-as-default?

@safe-by-default is currently hidden behind a -preview switch. I think it's safe to assume there will be a long deprecation period before it's made the default, and even then, there will be a -revert switch to turn it off.

On Monday, 25 May 2020 at 17:31:29 UTC, Paul Backus wrote:
> Even if you did review your code *and* all of your dependencies, if you did it before DIP 1028 was accepted (i.e., any time in the last 10 years or so), you may have deliberately left external @system functions un-annotated, because @system was the default.
>
> In fact, I *know* that people have done this, because I've been running my patched version of the compiler [1] against the dmd, druntime, and phobos codebases, and I have found an enormous number of such un-annotated declarations. Here are just a few of them in druntime:
>
> https://github.com/dlang/druntime/pull/3117
>
> Lots and lots of projects in the D ecosystem are going to need similar patches to be compatible with DIP 1028. And I don't think it's realistic to expect people to make these changes without some kind of help from the tooling.
>
> [1] https://github.com/dlang/dmd/pull/11176

I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation.

On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
>
> I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation.

Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed @safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly.

On Monday, 25 May 2020 at 16:29:24 UTC, Atila Neves wrote:
> I share your concerns on this, but disagree on the likelihood of reviews having gone by under the assumption of @system by default. I doubt most people even thought about @safe/@trusted/@system, and that's assuming anyone reviewed the code in the first place.
>
> A few years ago I submitted several PRs to Phobos to mark all unittests that could with @safe explicitly. I'd say that was a good example of nobody reviewing them for their @systemness.

I don't think a community run project without a lead or team behind it is a very good example. No one is going to spend their free time reviewing code ensuring it is memory safe. I've seen it such that unit tests in phobos/druntime weren't being run on some platforms, and there were tests that failed when run.

On Monday, 25 May 2020 at 19:44:00 UTC, Paul Backus wrote:
> On Monday, 25 May 2020 at 19:35:04 UTC, Clarice wrote:
>>
>> I really appreciate your contributions. (The latter of which will be quite handy.) But isn't the whole point of Walter's implementation of @safe-as-default is that FFIs are going to be assumed @safe? So they should still compile; that is unless you're referring to the need for re-evaluation.
>
> Yes, that's Walter's current plan. I am hoping that he'll change his mind about allowing un-annotated FFIs to be assumed @safe, but even if he doesn't, having the compiler print a warning will allow those of us who care about safety to find them and handle them properly.

I'm still unsure which implementation is the right one, but I think this middle-ground you're attempting to get merged is a good compromise. So, again, thank you!

The DIP is trying to accomplish (copied from its Rationale):

A. Costs of unsafe code have become ever more apparent and expensive, and @safe has grown more capable. Users expect safety to be opt-out, not opt-in.

B. Most code should be naturally safe, and system code should be relatively rare. It makes sense that the common case - safe code - should be the default and not require an annotation.

Any judgment on the regime of extern(C) functions should be judged as follows: "To what extent does the chosen regime of C functions push the rationale forward?"

The decision fails to support (A) because it keeps safety of C functions opt-in. It also fails to support (B) because it keeps code that does not require an annotation yet is not safe.

A good process would simply go back to the drawing board once the matter is exposed. It is very easy for reasonable people to agree that there is no part of rationale that justifies the decision. Either redefine the rationale (adding e.g. "keep some applications compilable without change" as an important goal), or change the handing of C functions.

The core argument recently detailed is:

A. Define "greenwashing" as a quick workaround to get code to work.
B. Describe in excruciating detail the experience with Java exceptions.
C. Establish a vague analogy with people placing @trusted on C declarations.
D. Assume without stating that the user would do "greenwashing" but the compiler doing it surreptitiously is somehow exempt.
E. Conclude that extern(C) code must be trusted.

This argument is already a smorgasbord of fallacies. It iterates a good subset of https://en.wikipedia.org/wiki/List_of_fallacies, and I am not kidding when I say a few edits to those pages with backreferences to this thread would be appropriate.

To start:

False analogy: making extern(C) function system is analogous with the Java exeception specification debacle.

https://en.wikipedia.org/wiki/Argument_from_analogy#False_analogy

This is a false analogy belying what seems to be a misunderstanding of the problem with Java exceptions: it was a maintenance problem, not an introduction problem.

The matter was, during maintenance, routinely code would change the set of exceptions they threw. So maintainers would need to touch code that didn't belong to them. There was also no "shrinking" of exception specification lists because the compiler does not enforce larger-than-necessary lists. So these lists would just grow forever during maintenance, with no real information attached and no usefulness.

None of these apply to @safe. Functions don't change their safety with ease, and when they do you definitely want to be informed. There's no growth of attributes for obvious reasons.

To the extent the analogy applies, the burden of proof is entirely on the DIP, and in this case the differences are so many and large, it's not worth making the argument by analogy in the first place.

The excruciatingly long discussion of Java exception specifications is an instance of another fallacy, misleading vividness:

http://nizkor.com/features/fallacies/misleading-vividness.html

It was expected that any of the bad things about exceptions would count as yet another good argument for the DIP. This constructs a textbook straw man:

https://en.wikipedia.org/wiki/Straw_man

So the argument constructs a straw man (Java exception specifications), discusses it at length on a long tangent (misleading vividness), to then conclude (by false analogy) that we don't want extern(C) functions to be @system.

What's missing is, of course, an explanation on how the decision supports the very rationale of the DIP. Which it doesn't - it factually works straight against it.

I decided to make this public only once the second boot fell (read on):

On 5/23/20 11:28 PM, Walter Bright wrote:
> I'd like to emphasize:
> 
> 1. It is not possible for the compiler to check any declarations where the implementation is not available. Not in D, not in any language. Declaring a declaration safe does not make it safe.

Strawman: https://en.wikipedia.org/wiki/Straw_man

Nobody asked for that, and listing it as an argument is pointing at a strawman.

> 2. If un-annotated declarations cause a compile time error, it is highly likely the programmer will resort to "greenwashing" - just slapping @safe on it. I've greenwashed code. Atila has. Bruce Eckel has. We've all done it. Sometimes even for good reasons.

Nirvana fallacy: https://en.wikipedia.org/wiki/Nirvana_fallacy

"Electric cars still use polluting materials and processes, so let's continue making gasoline cars."

"People will add workarounds to code, so let's accept the dangerous code anyway."

> 3. Un-annotated declarations are easily detectable in a code review.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

(Um... there's no grep for "find me all un-annnotated declarations")

> 4. Greenwashing is not easily detectable in a code review.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

(Um... grep '@safe|@trusted')

> 5. Greenwashing doesn't fix anything. The code is not safer. It's an illusion, not a guarantee.

Shifting the burden of proof from the DIP to the reviewer: https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)

The DIP needs to prove it does not perform greenwashing itself. Which it factually does - it paints all C code as safe with no user intervention.
"Quod licet bovis non licet Iovis" much? That smacks of special pleading:

https://en.wikipedia.org/wiki/Special_pleading

Of course we have the strawman: https://en.wikipedia.org/wiki/Straw_man. Nobody asked for greenwashing.

> 6. If someone cares to annotate declarations, it means he has at least thought about it, because he doesn't need to. Hence it's more likely to be correct than when greenwashed.

So... requiring adding annotations to extern(C) is good?

> 7. D should *not* make it worthwhile for people to greenwash code.

Proof by assertion: https://en.wikipedia.org/wiki/Proof_by_assertion

No accounting for the fact that the very built-in rule does exactly greenwashing as the post seems to define it.

> It is, in a not-at-all obvious way, safer for C declarations to default to being safe.

In ensemble, all this adds up to a good Kettle logic fallacy:

https://en.wikipedia.org/wiki/Kettle_logic

For the record, I think adoption of this rule works against the very thing that the DIP is trying to accomplish and will also cause a serious PR blow.

Wowza, wow. The voice of reason returns!