On 4/8/20 9:35 AM, aliak wrote:
> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>> [...]
>> [...]
> 
> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?

No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.

> 
> You can't mechanically check an extern(C) function declaration with no source code. And @trusted on a mechanically checked extern(C) function definition doesn't help anyone.

If you have an extern(C) function that is mechanically checked @safe, should you have to mark the prototype @trusted? This is the question we are discussing.

My opinion is that requiring a marking of @trusted doesn't make anything safer or more correct. If the function all of a sudden becomes @system (truly @system, and not semantically safe), then the @trusted is still a lie.

The result is just going to be people ignoring @trusted C prototypes beyond the prototype definition itself, so it makes @trusted less of a red flag than it should be.

If you see:

@trusted extern(C) void memcpy(void *dest, void *src, size_t size)

Then this is obviously wrong, and should be @system. It wouldn't be any different for a @safe marking.

If you see:

@trusted extern(C) void copy(ubyte[] dest, ubyte[] src)

Then you are going to go check it, see that it's really @safe in the implementation, and start ignoring such prototypes as flags. Maybe you comment on it so next time you don't waste time going to check that one:

// really @safe in implementation
@trusted extern(C) void copy(ubyte[] dest, ubyte[] src)

Now, if it all of a sudden changes to @system, your @trusted tag is still ignored, just like it was marked @safe. That is my point. It doesn't change what is interpreted by a reviewer to require @trusted, it just adds busywork to shut up the compiler.

In the end, the difference between @safe and @trusted on an extern(C) function prototype is documentation. functionally, they are equivalent.

-Steve

On Wednesday, 8 April 2020 at 13:58:12 UTC, Steven Schveighoffer wrote:
> On 4/8/20 9:35 AM, aliak wrote:
>> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>>> [...]
>>> [...]
>> 
>> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?
>
> No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.

Ok 👍My bad then.

>
>> 
>> You can't mechanically check an extern(C) function declaration with no source code. And @trusted on a mechanically checked extern(C) function definition doesn't help anyone.
>
> If you have an extern(C) function that is mechanically checked @safe, should you have to mark the prototype @trusted? This is the question we are discussing.

Ah, then yeah, as mentioned above, I'm in your boat; if it's mechanically checked then there's no need to mark it @trusted, that'd just be a waste of time in the hunt for memory corruption.

>
> My opinion is that requiring a marking of @trusted doesn't make anything safer or more correct. If the function all of a sudden becomes @system (truly @system, and not semantically safe), then the @trusted is still a lie.
>
> The result is just going to be people ignoring @trusted C prototypes beyond the prototype definition itself, so it makes @trusted less of a red flag than it should be.
>
> If you see:
>
> @trusted extern(C) void memcpy(void *dest, void *src, size_t size)
>
> Then this is obviously wrong, and should be @system. It wouldn't be any different for a @safe marking.

Yes it would? If this was @safe I wouldn't need to check its usage sights because I'm supposed to trust that @safe can only be applied to functions that are mechanically checked. If it's @trusted then I know I need to check it's usage sights because it's not mechanically guaranteed to be a safe function.

>
> If you see:
>
> @trusted extern(C) void copy(ubyte[] dest, ubyte[] src)
>
> Then you are going to go check it, see that it's really @safe in the implementation, and start ignoring such prototypes as flags. Maybe you comment on it so next time you don't waste time going to check that one:

Yes, if I go to that definition and see that the body is @safe then I'll be annoyed. If that's a prototype to a function defined in a C library then it should not be allowed to be marked @safe.

>
> // really @safe in implementation
> @trusted extern(C) void copy(ubyte[] dest, ubyte[] src)
>
> Now, if it all of a sudden changes to @system, your @trusted tag is still ignored, just like it was marked @safe. That is my point. It doesn't change what is interpreted by a reviewer to require @trusted, it just adds busywork to shut up the compiler.
>
> In the end, the difference between @safe and @trusted on an extern(C) function prototype is documentation. functionally, they are equivalent.
>
> -Steve

On Wednesday, 8 April 2020 at 13:58:12 UTC, Steven Schveighoffer wrote:
> On 4/8/20 9:35 AM, aliak wrote:
>> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>>> [...]
>>> [...]
>> 
>> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?
>
> No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.
>

Which Jonathan?  I'm on the side that if the compiler can verify it, it should be @safe.  The extern'ness of a function seems to be orthogonal as to whether it can be verified.  Having a function body seems to be the primary criteria.

On 4/8/20 11:56 AM, Jonathan Marler wrote:
> On Wednesday, 8 April 2020 at 13:58:12 UTC, Steven Schveighoffer wrote:
>> On 4/8/20 9:35 AM, aliak wrote:
>>> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>>>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>>>> [...]
>>>> [...]
>>>
>>> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?
>>
>> No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.
>>
> 
> Which Jonathan?  I'm on the side that if the compiler can verify it, it should be @safe.  The extern'ness of a function seems to be orthogonal as to whether it can be verified.  Having a function body seems to be the primary criteria.
> 

I was referring to the other one ;)

But we are talking about prototypes not functions. If the function is implemented in D and @safe, should you be required to mark the prototype @trusted? If so, what benefit does that provide?

I don't think the benefit is worth the cost of annoyance -- it ends up the same either way.

I just thought of another way to fix this whole mess (even Walter's @safe by default for extern(C)).

I don't want to post it deep inside this thread, so I'll move it to the top.

-Steve

On 3/25/20 10:10 AM, Steven Schveighoffer wrote:
> In response to Walter's response to ag*, I would say that there is a fatal problem with automatically allowing extern(C) function prototypes (and really, anything that does not mangle @safe) to be default @safe.
> 
> The reason is simple -- the change is silent and automatically marks everything @safe that has not been checked.
> 
> I would argue that if the compiler is going to make things @safe by default, then things that are not marked and are not @safe should not compile AT ALL COSTS. Otherwise the value of @safe is completely lost.
> 
> The DIP should be rejected IMO unless all functions with no mechanism to mangle @safe into the name (e.g. extern(C), extern(C++), etc) that have no implementation are either:
> 
> a) required to be marked, or
> b) default @system.
> 
> Everything else in the DIP is possibly annoying to deal with but at least doesn't silently destroy the meaning of @safe.
> 
> I will note that I support the notion of @safe by default. I would be in favor of the DIP as long as this fatal flaw is not included.

I thought of a third option that could work:

When an extern(C) @safe function is compiled, produce 2 symbols that point at the same function text, one that is the normal extern(C) function symbol name (_foo), and one that is mangled with something like e.g. _dsafe_foo.

When there is an extern(C) @safe prototype, only look for the _dsafe_foo version. If there is an extern(C) @trusted or or @system prototype, then look for the normal _foo symbol.

What does this accomplish?

1. you can keep @safe by default extern(C) functions, as they now will not link if the real function is @system or built in something other than D.
2. @trusted "overrides" still work. In other words the function is implemented in C but can technically be @trusted because it doesn't involve memory safety problems.
3. Calls from other languages that bind to C still work. In other words, you can still call foo from C/C++ through the normal C symbol.
4. Any changes to the actual implementation that cause the prototype to be @trusted or @system will now fail to link as the _dsafe_foo symbol goes away.

I don't know how this might work for extern(C++), and I'm also not sure how the symbol emission works, but I feel like this should be possible. It would be a benefit even if we don't make @safe the default for extern(C).

-Steve

On 4/8/20 11:52 AM, aliak wrote:
>> If you see:
>>
>> @trusted extern(C) void memcpy(void *dest, void *src, size_t size)
>>
>> Then this is obviously wrong, and should be @system. It wouldn't be any different for a @safe marking.
> 
> Yes it would? If this was @safe I wouldn't need to check its usage sights because I'm supposed to trust that @safe can only be applied to functions that are mechanically checked. If it's @trusted then I know I need to check it's usage sights because it's not mechanically guaranteed to be a safe function.

There may be something you are missing. You can mark e.g. memcpy @safe, and the compiler is fine with it. It will still link, and now be callable inside @safe code.

The connection between the prototype and the implementation is MANUALLY maintained for extern(C) code.

So if you saw @safe here, you should not be satisfied with it, as the marking is probably wrong, but will still compile. You shouldn't say "well, it's marked @safe, so memcpy must be @safe".

This is the whole problem with extern(C) prototypes -- you are allowed to lie without consequence.

But in a lot of cases, D-implemented extern(C) code *is* @safe, so it wouldn't be a lie.

-Steve

On Wednesday, 8 April 2020 at 17:01:29 UTC, Steven Schveighoffer wrote:
> On 4/8/20 11:52 AM, aliak wrote:
>>> If you see:
>>>
>>> @trusted extern(C) void memcpy(void *dest, void *src, size_t size)
>>>
>>> Then this is obviously wrong, and should be @system. It wouldn't be any different for a @safe marking.
>> 
>> Yes it would? If this was @safe I wouldn't need to check its usage sights because I'm supposed to trust that @safe can only be applied to functions that are mechanically checked. If it's @trusted then I know I need to check it's usage sights because it's not mechanically guaranteed to be a safe function.
>
> There may be something you are missing. You can mark e.g. memcpy @safe, and the compiler is fine with it. It will still link, and now be callable inside @safe code.

I was under the assumption we were talking about how to go forward from whatever the current status quo was.

Though, I did not know that was how it worked now because frankly I never tried marking an extern(C) function (declaration, not definition) as @safe - never even occurred to me.

If that's possible now then it should be fixed as I can't see how that's correct behaviour.

>
> The connection between the prototype and the implementation is MANUALLY maintained for extern(C) code.
>
> So if you saw @safe here, you should not be satisfied with it, as the marking is probably wrong, but will still compile. You shouldn't say "well, it's marked @safe, so memcpy must be @safe".
>
> This is the whole problem with extern(C) prototypes -- you are allowed to lie without consequence.
>
> But in a lot of cases, D-implemented extern(C) code *is* @safe, so it wouldn't be a lie.
>
> -Steve

On 08.04.20 17:56, Jonathan Marler wrote:
> On Wednesday, 8 April 2020 at 13:58:12 UTC, Steven Schveighoffer wrote:
>> On 4/8/20 9:35 AM, aliak wrote:
>>> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>>>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>>>> [...]
>>>> [...]
>>>
>>> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?
>>
>> No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.
>>
> 
> Which Jonathan?  I'm on the side that if the compiler can verify it, it should be @safe.  The extern'ness of a function seems to be orthogonal as to whether it can be verified.  Having a function body seems to be the primary criteria.
> 

To be very clear, Steven says the following is desirable to allow as a possibility:

---
module a;

extern(C) corrupt_memory()@system{
    // ...
}
---
---
module b;
extern(C) corrupt_memory()@safe;

void main()@safe{
    corrupt_memory();
}
---

A "prototype" is a function declaration without a function body.

Related: https://www.theregister.co.uk/2016/02/04/underhand_c_2015/

On 4/8/20 2:57 PM, Timon Gehr wrote:
> On 08.04.20 17:56, Jonathan Marler wrote:
>> On Wednesday, 8 April 2020 at 13:58:12 UTC, Steven Schveighoffer wrote:
>>> On 4/8/20 9:35 AM, aliak wrote:
>>>> On Wednesday, 8 April 2020 at 13:09:33 UTC, Steven Schveighoffer wrote:
>>>>> On 4/7/20 10:01 PM, Timon Gehr wrote:
>>>>>> [...]
>>>>> [...]
>>>>
>>>> It really feels like you and Timon are talking past each other based on mixing up extern(C) declarations and extern(C) definitions?
>>>
>>> No, I think we both understand that difference. I actually understand the point of view of Timon and Jonathan and agree with that point of view, I just think the restriction isn't helpful, and is just going to cause busywork for no true benefit.
>>>
>>
>> Which Jonathan?  I'm on the side that if the compiler can verify it, it should be @safe.  The extern'ness of a function seems to be orthogonal as to whether it can be verified.  Having a function body seems to be the primary criteria.
>>
> 
> To be very clear, Steven says the following is desirable to allow as a possibility:
> 
> ---
> module a;
> 
> extern(C) corrupt_memory()@system{
>      // ...
> }
> ---
> ---
> module b;
> extern(C) corrupt_memory()@safe;
> 
> void main()@safe{
>      corrupt_memory();
> }
> ---
> 
> A "prototype" is a function declaration without a function body.
> 
> Related: https://www.theregister.co.uk/2016/02/04/underhand_c_2015/

To be clear, Timon says the following EQUIVALENT situation is desirable to allow as a possibility, and somehow this is different than the above:

extern(C) void corrupt_memory() @trusted; // go ahead, trust this function named corrupt_memory.

Simply because the unseen function is marked @trusted (when it is really @system). In other words, the function author made zero guarantees about memory safety, and the prototype author just trusts him blindly.

And really, what I'm saying is that I want:

---
module a;
extern(C) void perfectly_safe @safe {
 // ...
}
---
---
module b;
extern(C) void perfectly_safe() @safe;
---

To be possible.

If we can't have that then you have to lie (say it's @trusted when it's actually mechanically checked).

You have to have the first (very very unlikely) case allowed in order to have the second case allowed. Disallowing one disallows the other.

Since the situation with @trusted requirements isn't any safer (I'd argue it's less safe since it dilutes the meaning of @trusted), I'd say don't put up red tape that doesn't help.

-Steve

On 08.04.20 15:09, Steven Schveighoffer wrote:
> On 4/7/20 10:01 PM, Timon Gehr wrote:
>> On 07.04.20 23:14, Steven Schveighoffer wrote:
>>> On 4/6/20 2:28 PM, Timon Gehr wrote:
>>>> On 06.04.20 01:38, Steven Schveighoffer wrote:
>>>>> I disagree with disallowing @safe as a specific marking on extern(C) code. You can write @safe extern(C) functions in D, and it makes no sense to require that they are @trusted at the prototype.
>>>>
>>>> The linker can hijack them. The function signature of a @trusted function should be @safe anyway.
>>>
>>> The linker can always hijack it.
>>
>> I guess I used the wrong word. What I meant was that there is zero checking that the extern(C) function you are calling was actually checked to be @safe. For extern(D), the language at least makes a small effort to ensure the @safe qualifier has some meaning, but given that return types are not mangled, that also seems like a lie.
> 
> If you change the return type, then there can be detectable differences at runtime. If you change the @safe vs. @trusted vs. @system, everything works *exactly* as before (unless the linker stops it).
> 
> The biggest problem is code that works but is wrong in terms of safety. In that case, @safe and @trusted on prototypes are identical.
> ...

Wrong. In terms of _language_ and _type system_ design what's most important is that if code is marked @safe, is actually memory safe conditional on programmers getting the @trusted annotations right. That's the promise that the language is giving.

>>
>>> Even without intention. Having it marked @trusted isn't any better than having it marked @safe in that case.
>>> ...
>>
>> Memory corruption in a @safe context should be traceable to @trusted code. It's the entire point.
> 
> And in this case, there is no trusted code.

You are trusting the prototype to be correct.

> It's all @safe, but you have a boy-who-cried-wolf effect on the @trusted prototype. "Oh, you can ignore those prototypes because the compiler made me do it." In fact I'm assuming you will see stuff like:
> 
> // really @safe
> @trusted extern(C) ...
> 
> causing a reviewer to pass over those as possible problems.
> ...

And of course if the module has no @trusted code at all the reviewer will be extra careful?

> I'm not saying @safe marking of prototypes isn't prone to issues,

Purely @safe code is not supposed to be prone to issues.

> I'm saying forcing @trusted markings doesn't change that fact.
> ...

@trusted means "there may be issues here". @safe means "no issues here".

>>
>>>>
>>>>> Assuming @safe, no. Explicitly @safe OK, you marked it, you own it.
>>>>> ...
>>>>
>>>> @safe:
>>>>
>>>> // a lot of code
>>>> // ...
>>>>
>>>> extern(C) corrupt_all_the_memory();
>>>
>>> Why would you do this when @safe is the default?
>>> ...
>>
>> To show it's broken.
>>
>> Besides that, it's not currently the default, and if it is, you might want to temporarily switch to @system and back.
> 
> In that case, all is well! you properly marked the right ones @system.
> ...

So we're right back to trusting the programmer even though that programmer did not actually write any @trusted annotations.

> In the current regime, @safe: at the top does the same thing you are trying to argue against. So there won't be existing (good) code that does this.
> 
> In the new regime, @safe is the default, so you wouldn't need to put @safe: at the top.
> ...

It does not have to be at the top of the file.

> Any time the compiler forces you to mark things differently than the truth, you become more numb to these compiler warnings, and don't put any stock into their significance.
> ...

@trusted is the truth, @safe is the lie.

>> In any case, @safe code by definition is code written by untrusted programmers. Why do you insist there should be a good reason? Maybe the programmer was a monkey. Or malicious.
> 
> @safe code is mechanically checked. Why shouldn't I be able to declare that when it's true?

Because the prototype is not mechanically checked. Programmers who are not allowed to write @trusted code have no business writing @safe extern(C) prototypes.

> And if you make me mark it @trusted, and it for some reason becomes @system (because it's not @trustable, a very unlikely occurrence), having them marked @trusted doesn't help. You still have to go find the prototypes (all of them) and mark them @system instead.
> ...

It's the prototype maintainer's responsibility to ensure it does not happen.

>>
>>>>
>>>> When did @safe become a matter of "it's your own fault if you shoot yourself in the foot, this memory corruption you are having is a good thing because it will teach you not to make more mistakes in the future"? If something may break @safe-ty as it trusts the programmer to get it right, it ought to be @trusted.
>>>
>>> If you mark a @safe extern(C) function @trusted, how does that help? I'm talking about extern(C) functions that are checked by the compiler as @safe. Why should I have to mark the prototype of that function @trusted?
>>
>> Because the extern(C) function can be changed to be @system. Hence you must trust the maintainer of the prototype to keep it in sync with the implementation.
> 
> So you are saying this scenario is OK:
> 
> Hm... my @safe function is turning into @system. But that's OK because everyone had to mark their prototypes @trusted! So now it becomes their fault I changed it.
> ...

It is their fault. Why did they make their @safety conditional on the signature of an extern(C) function that apparently was outside their own control?

> I don't get how this is helpful. I don't get how this is somehow logically superior to the same people being able to mark the functions @safe. In both cases, you are pulling the rug from underneath them.
> ...

But the code at the prototype is not lying about it.

> It sounds more like a "good" non-monkey programmer should mark all extern(C) function prototypes @system, regardless of the actual safety of the function, and require @trusted escapes, because they can change at any time without warning and they might get blamed.

If that is in fact a possibility they should absolutely do that. Otherwise they need to have conventions in place that prevent it from happening. If the @safety of a piece of code is conditional on such conventions, it should be marked @trusted.

>> ...
> What if the @safe prototype is auto-generated? Then whenever it changes to @system, the prototype will be changed. This means, you have a memory issue, you search for @trusted, find all these prototypes that are actually prototypes for @safe functions auto generated. You now get into the habit of ignoring all @trusted prototypes as possible issues because it's just noise. Better to focus on the @trusted implementations, which is where the real problem can happen.
> ...

https://www.theregister.co.uk/2016/02/04/underhand_c_2015/