On Monday, 6 April 2020 at 01:43:58 UTC, Timon Gehr wrote:
> On 06.04.20 02:09, Jonathan Marler wrote:
>> On Saturday, 4 April 2020 at 06:53:57 UTC, Walter Bright wrote:
>>> [....]
>>> Now, as to what to do. I spent a few minutes and added `@system:` in to the C headers in druntime for windows, posix, and linux. Done. I hope someone else will do it for freebsd, etc., if not I'll go do that to.
>>>
>> 
>> The thought crossed my mind that this same reasoning could be applied as a counter-argument for this DIP:
>> 
>> ... So functions are not safe by default?
>> 
>> Now, as to what to do. I spent a few minutes and added
>> `@safe:` in to the source files in druntime for windows, posix,
>> and linux. Done. I hope someone else will do it for freebsd,
>> etc., if not I'll go do that to.
>> ...
>
> So you annotated all those extern(C) function prototypes as @safe? :)
>

That paragraph wasn't something I was saying, I took what Walter said (see above) and replaced a couple words to show the argument he was using to say extern functions should be considered @safe by default could also be used to argue against the DIP itself.

A good way to test the merit of an argument is to apply it to multiple scenarios.  For example, you could say "I like apples because they are red."  To test if that's the real reason you like apples you can try the same argument with other scenarios.  You could try "I like tomatoes because they are red", but if you don't acutally like tomatoes, which are also red, then that means the "redness" of objects may not have much to do with why you like apples.  Redness may contribute to your "likeness" but there are probably more important factors.

In this situation Walter said (paraphrased) "If you want to to make your extern functions @system by default, put them in their own file and put "@system:" at the top".  But the whole point of this DIP is to change the default from @system to @safe, so the same argument he used would be an argument against his own DIP, "If you want to make your functions @safe by default, put them in their own file and put "@safe:" at the top".

This leads me to believe that he hasn't shared or identified the real reason he doesn't agree with what people are saying about defaulting to @system on unverifiable functions.  It's like he is saying "I like apples because they are red", and then turning around later and saying "I don't like tomatoes because they are red".  Either defaults matter or they don't, if they don't then you are saying your own DIP doesn't matter, if they do, then I wouldn't dismiss what people are saying about the default @safeness of unverifiable functions.

On Monday, 6 April 2020 at 12:35:20 UTC, Steven Schveighoffer wrote:
> Is this really your argument? Do we need Scott Meyers to explain such "esoteric" compiler errors like:
>
> Error: cannot call @system function memcpy from @safe function main.
>
> I think this is not an hour-long talk, but a 10-second talk: "C functions aren't checked by the D compiler, so they are @system by default." Done.

We can do this even in the compiler.  Since `@safe` checks cannot be performed on a bodyless function, it should be an error to declare a bodyless `extern(C)` function without either `@trusted` or `@system` attribute explicitly marked.

And it should be straightforward to report this error, something like:

    Error: default @safe checks cannot be performed on extern function `{name}`.
    Please indicate explicitly whether this function is @system or @trusted.

Honestly, I think we'd be far more in need of a Scott Meyers to explain extern C functions being treated as safe by default (and how to avoid unintended bugs caused by this) than by the straightforward fact (well accepted e.g. by Rust users) that extern functions cannot be assumed safe, because, well, how do you prove it?

On Mon, Apr 06, 2020 at 06:45:14PM +0000, Joseph Rushton Wakeling via Digitalmars-d wrote: [...]
> Honestly, I think we'd be far more in need of a Scott Meyers to explain extern C functions being treated as safe by default (and how to avoid unintended bugs caused by this) than by the straightforward fact (well accepted e.g. by Rust users) that extern functions cannot be assumed safe, because, well, how do you prove it?

I can see it already.  DIP 1028 gets merged, and the following DConf Scott Meyers comes to give a talk about how @safe in D is full of holes and doesn't fulfill its promises.


T

-- 
Famous last words: I *think* this will work...

On Monday, 6 April 2020 at 19:12:18 UTC, H. S. Teoh wrote:
> On Mon, Apr 06, 2020 at 06:45:14PM +0000, Joseph Rushton Wakeling via Digitalmars-d wrote: [...]
>> Honestly, I think we'd be far more in need of a Scott Meyers to explain extern C functions being treated as safe by default (and how to avoid unintended bugs caused by this) than by the straightforward fact (well accepted e.g. by Rust users) that extern functions cannot be assumed safe, because, well, how do you prove it?
>
> I can see it already.  DIP 1028 gets merged, and the following DConf Scott Meyers comes to give a talk about how @safe in D is full of holes and doesn't fulfill its promises.
>
>
> T

Better yet, email Scott Meyers and had him explain to Walter why extern C functions being mark as safe by default is a terrible idea.

-Alex

On Monday, 6 April 2020 at 19:12:18 UTC, H. S. Teoh wrote:
> On Mon, Apr 06, 2020 at 06:45:14PM +0000, Joseph Rushton Wakeling via Digitalmars-d wrote: [...]
>> Honestly, I think we'd be far more in need of a Scott Meyers to explain extern C functions being treated as safe by default (and how to avoid unintended bugs caused by this) than by the straightforward fact (well accepted e.g. by Rust users) that extern functions cannot be assumed safe, because, well, how do you prove it?
>
> I can see it already.  DIP 1028 gets merged, and the following DConf Scott Meyers comes to give a talk about how @safe in D is full of holes and doesn't fulfill its promises.
>
>
> T

LOL!

On 4/6/20 11:35 AM, aliak wrote:
>> e.g:
>>
>> static if(someBool)
>>    return x;
>> return y; // error statement not reachable
>>
>> inout int[] arr;
>> writeln(arr); // OK
>> writeln(arr.filter!(a => a %5 == 0)); // Error: only parameters or stack based variables can be inout
> 
> I'm not sure those are technically sound. Maybe more incomplete implementations of the features?

The issue is that filter returns a wrapper type that contains the original range. Since inout(int)[] is really the only working inout range in existence, it needs to be stored that way. But the compiler disallows it. So you see weird problems like this.

-Steve

On 4/6/20 2:28 PM, Timon Gehr wrote:
> On 06.04.20 01:38, Steven Schveighoffer wrote:
>> I disagree with disallowing @safe as a specific marking on extern(C) code. You can write @safe extern(C) functions in D, and it makes no sense to require that they are @trusted at the prototype.
> 
> The linker can hijack them. The function signature of a @trusted function should be @safe anyway.

The linker can always hijack it. Even without intention. Having it marked @trusted isn't any better than having it marked @safe in that case.

> 
>> Assuming @safe, no. Explicitly @safe OK, you marked it, you own it.
>> ...
> 
> @safe:
> 
> // a lot of code
> // ...
> 
> extern(C) corrupt_all_the_memory();

Why would you do this when @safe is the default?

> 
> When did @safe become a matter of "it's your own fault if you shoot yourself in the foot, this memory corruption you are having is a good thing because it will teach you not to make more mistakes in the future"? If something may break @safe-ty as it trusts the programmer to get it right, it ought to be @trusted.

If you mark a @safe extern(C) function @trusted, how does that help? I'm talking about extern(C) functions that are checked by the compiler as @safe. Why should I have to mark the prototype of that function @trusted? How does that prevent problems? It's functionally equivalent. It's not something that changes when the original extern(C) function for some reason becomes @system (unlike extern(D) code).

I agree with the concept that it's impossible for the compiler to know that an extern(C) prototype is @safe, but I think it's more bureaucratic than effective to make you use @trusted instead of @safe. It's like a law that has roots in good policy, but results in a lot of frivolous enforcement.

>> We have similar problems with inout -- preventing obvious incorrect cases makes sense, until it doesn't.
> 
> This is not analogous. Here, the problem is that the "obvious incorrect cases" where not actually incorrect.

No, they were obvious. If you have an inout return but no inout variables, the "what goes in comes out" doesn't make any sense. Just use const in that case.

But it turns out it's just easier for generic code to have these cases devolve into the equivalents that just use const/immutable, so you don't have to use static if's everywhere.

> I think you can just grep for the error messages and then remove the checks.

I think there are more things than that. For example, inside non-inout code, you need to figure out what inout means, and enforce that.

-Steve

On 07.04.20 23:14, Steven Schveighoffer wrote:
> On 4/6/20 2:28 PM, Timon Gehr wrote:
>> On 06.04.20 01:38, Steven Schveighoffer wrote:
>>> I disagree with disallowing @safe as a specific marking on extern(C) code. You can write @safe extern(C) functions in D, and it makes no sense to require that they are @trusted at the prototype.
>>
>> The linker can hijack them. The function signature of a @trusted function should be @safe anyway.
> 
> The linker can always hijack it.

I guess I used the wrong word. What I meant was that there is zero checking that the extern(C) function you are calling was actually checked to be @safe. For extern(D), the language at least makes a small effort to ensure the @safe qualifier has some meaning, but given that return types are not mangled, that also seems like a lie.

> Even without intention. Having it marked @trusted isn't any better than having it marked @safe in that case.
> ...

Memory corruption in a @safe context should be traceable to @trusted code. It's the entire point.

>>
>>> Assuming @safe, no. Explicitly @safe OK, you marked it, you own it.
>>> ...
>>
>> @safe:
>>
>> // a lot of code
>> // ...
>>
>> extern(C) corrupt_all_the_memory();
> 
> Why would you do this when @safe is the default?
> ...

To show it's broken.

Besides that, it's not currently the default, and if it is, you might want to temporarily switch to @system and back. In any case, @safe code by definition is code written by untrusted programmers. Why do you insist there should be a good reason? Maybe the programmer was a monkey. Or malicious.

>>
>> When did @safe become a matter of "it's your own fault if you shoot yourself in the foot, this memory corruption you are having is a good thing because it will teach you not to make more mistakes in the future"? If something may break @safe-ty as it trusts the programmer to get it right, it ought to be @trusted.
> 
> If you mark a @safe extern(C) function @trusted, how does that help? I'm talking about extern(C) functions that are checked by the compiler as @safe. Why should I have to mark the prototype of that function @trusted?

Because the extern(C) function can be changed to be @system. Hence you must trust the maintainer of the prototype to keep it in sync with the implementation.

> How does that prevent problems?

The point is to be able to trace any problems to some @trusted annotations. @safe alone doesn't completely prevent memory safety problems, but it advertises giving you a way to completely avoid being blamed for them. (Except for your choice of programming language, I suppose.)

> It's functionally equivalent.

$ grep @trusted *.d

Also, if you had in fact successfully identified a case where @safe and @trusted are functionally equivalent, why would that not ring alarm bells? If you write @safe, you don't actually mean @trusted.

> It's not something that changes when the original extern(C) function for some reason becomes @system (unlike extern(D) code).
> 
> I agree with the concept that it's impossible for the compiler to know that an extern(C) prototype is @safe, but I think it's more bureaucratic than effective to make you use @trusted instead of @safe. It's like a law that has roots in good policy, but results in a lot of frivolous enforcement.
> ...

I really don't understand why anyone would argue in favor of a policy that allows code to become less safe when you annotate it @safe.

>>> We have similar problems with inout -- preventing obvious incorrect cases makes sense, until it doesn't.
>>
>> This is not analogous. Here, the problem is that the "obvious incorrect cases" where not actually incorrect.
> 
> No, they were obvious.

They were not incorrect, and the bad reasoning about those cases was caused by a lack of understanding of formal logic and type theory. This is also the underlying cause for the inout-related type system holes.

On 4/7/20 10:01 PM, Timon Gehr wrote:
> On 07.04.20 23:14, Steven Schveighoffer wrote:
>> On 4/6/20 2:28 PM, Timon Gehr wrote:
>>> On 06.04.20 01:38, Steven Schveighoffer wrote:
>>>> I disagree with disallowing @safe as a specific marking on extern(C) code. You can write @safe extern(C) functions in D, and it makes no sense to require that they are @trusted at the prototype.
>>>
>>> The linker can hijack them. The function signature of a @trusted function should be @safe anyway.
>>
>> The linker can always hijack it.
> 
> I guess I used the wrong word. What I meant was that there is zero checking that the extern(C) function you are calling was actually checked to be @safe. For extern(D), the language at least makes a small effort to ensure the @safe qualifier has some meaning, but given that return types are not mangled, that also seems like a lie.

If you change the return type, then there can be detectable differences at runtime. If you change the @safe vs. @trusted vs. @system, everything works *exactly* as before (unless the linker stops it).

The biggest problem is code that works but is wrong in terms of safety. In that case, @safe and @trusted on prototypes are identical.

> 
>> Even without intention. Having it marked @trusted isn't any better than having it marked @safe in that case.
>> ...
> 
> Memory corruption in a @safe context should be traceable to @trusted code. It's the entire point.

And in this case, there is no trusted code. It's all @safe, but you have a boy-who-cried-wolf effect on the @trusted prototype. "Oh, you can ignore those prototypes because the compiler made me do it." In fact I'm assuming you will see stuff like:

// really @safe
@trusted extern(C) ...

causing a reviewer to pass over those as possible problems.

I'm not saying @safe marking of prototypes isn't prone to issues, I'm saying forcing @trusted markings doesn't change that fact.

> 
>>>
>>>> Assuming @safe, no. Explicitly @safe OK, you marked it, you own it.
>>>> ...
>>>
>>> @safe:
>>>
>>> // a lot of code
>>> // ...
>>>
>>> extern(C) corrupt_all_the_memory();
>>
>> Why would you do this when @safe is the default?
>> ...
> 
> To show it's broken.
> 
> Besides that, it's not currently the default, and if it is, you might want to temporarily switch to @system and back.

In that case, all is well! you properly marked the right ones @system.

In the current regime, @safe: at the top does the same thing you are trying to argue against. So there won't be existing (good) code that does this.

In the new regime, @safe is the default, so you wouldn't need to put @safe: at the top.

Any time the compiler forces you to mark things differently than the truth, you become more numb to these compiler warnings, and don't put any stock into their significance.

> In any case, @safe code by definition is code written by untrusted programmers. Why do you insist there should be a good reason? Maybe the programmer was a monkey. Or malicious.

@safe code is mechanically checked. Why shouldn't I be able to declare that when it's true? And if you make me mark it @trusted, and it for some reason becomes @system (because it's not @trustable, a very unlikely occurrence), having them marked @trusted doesn't help. You still have to go find the prototypes (all of them) and mark them @system instead.

> 
>>>
>>> When did @safe become a matter of "it's your own fault if you shoot yourself in the foot, this memory corruption you are having is a good thing because it will teach you not to make more mistakes in the future"? If something may break @safe-ty as it trusts the programmer to get it right, it ought to be @trusted.
>>
>> If you mark a @safe extern(C) function @trusted, how does that help? I'm talking about extern(C) functions that are checked by the compiler as @safe. Why should I have to mark the prototype of that function @trusted?
> 
> Because the extern(C) function can be changed to be @system. Hence you must trust the maintainer of the prototype to keep it in sync with the implementation.

So you are saying this scenario is OK:

Hm... my @safe function is turning into @system. But that's OK because everyone had to mark their prototypes @trusted! So now it becomes their fault I changed it.

I don't get how this is helpful. I don't get how this is somehow logically superior to the same people being able to mark the functions @safe. In both cases, you are pulling the rug from underneath them.

It sounds more like a "good" non-monkey programmer should mark all extern(C) function prototypes @system, regardless of the actual safety of the function, and require @trusted escapes, because they can change at any time without warning and they might get blamed. This is what I meant by a bureaucratic solution.

>> How does that prevent problems?
> 
> The point is to be able to trace any problems to some @trusted annotations. @safe alone doesn't completely prevent memory safety problems, but it advertises giving you a way to completely avoid being blamed for them. (Except for your choice of programming language, I suppose.)

In my interpretation, @safe means the code inside the function was mechanically checked. This doesn't change when you are writing prototypes for your @safe functions.

> 
>> It's functionally equivalent.
> 
> $ grep @trusted *.d

And get a large noise/signal ratio of frivolous @trusted markings for extern(C) functions that are really @safe but were forced by the compiler to mark them @trusted. How does this help find the problem?

> 
> Also, if you had in fact successfully identified a case where @safe and @trusted are functionally equivalent, why would that not ring alarm bells? If you write @safe, you don't actually mean @trusted.

Functionally equivalent in that you can interchange them and it doesn't change anything in terms of ability to call or compile the functions (marking the prototypes, that is).

But to a reviewer, they are not functionally equivalent. @safe says, the function was compiled as a @safe function, @trusted says it was compiled as a @trusted function. At this level the difference between the two is informational, not functional. That's why they are functionally equivalent. I think we should allow the distinction on the prototype because we allow it on the implementation.

And technically, if you don't trust the prototype writer to get the safety right, you should verify he got the parameters and return type right as well. So really it should be:

grep extern(C) *.d

> 
>> It's not something that changes when the original extern(C) function for some reason becomes @system (unlike extern(D) code).
>>
>> I agree with the concept that it's impossible for the compiler to know that an extern(C) prototype is @safe, but I think it's more bureaucratic than effective to make you use @trusted instead of @safe. It's like a law that has roots in good policy, but results in a lot of frivolous enforcement.
>> ...
> 
> I really don't understand why anyone would argue in favor of a policy that allows code to become less safe when you annotate it @safe.

I don't understand why someone arguing that actual @safe functions cannot be marked @safe because they later could become @system would argue that as long as the prototypes are marked @trusted that's OK to switch to @system because it's someone else's fault. In neither case is the function author blameless because others were stupid to trust him with their prototypes.

But with the ability to mark things @safe, you have the ability to tell the truth about the actual implementation.

What if the @safe prototype is auto-generated? Then whenever it changes to @system, the prototype will be changed. This means, you have a memory issue, you search for @trusted, find all these prototypes that are actually prototypes for @safe functions auto generated. You now get into the habit of ignoring all @trusted prototypes as possible issues because it's just noise. Better to focus on the @trusted implementations, which is where the real problem can happen.

> 
>>>> We have similar problems with inout -- preventing obvious incorrect cases makes sense, until it doesn't.
>>>
>>> This is not analogous. Here, the problem is that the "obvious incorrect cases" where not actually incorrect.
>>
>> No, they were obvious.
> 
> They were not incorrect, and the bad reasoning about those cases was caused by a lack of understanding of formal logic and type theory. This is also the underlying cause for the inout-related type system holes.

I'll defer to you on type system stuff, and admit that my arguments for inout not working in the cases that cause problems were flawed. But they made obvious sense to me. I see the same philosophy in my bad consideration of cases for inout in your arguments for this @trusted requirement. Which is why I brought up the analogy.

-Steve

On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
> On 4/7/20 10:01 PM, Timon Gehr wrote:
>> [...]
> [...]

It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?

You can't mechanically check an extern(C) function declaration with no source code. And @trusted on a mechanically checked extern(C) function definition doesn't help anyone.