On 4/3/2020 5:44 PM, Jonathan M Davis wrote:
> Well, I don't think that you've actually acknowledged any of it, and what
> responses you do have make it seem like you're not aware of it or are
> ignoring it.

I have replied to it more than once.

On 4/3/2020 2:42 PM, Joseph Rushton Wakeling wrote:
> Now I'm not suggesting that D should do it because Rust does ;-) But their reasoning seems sound, and I don't see an obvious reason for assuming @safe-ty of functions that the D compiler cannot verify.

In both Rust and D the reliance is on the programmer when calling functions in another language. D gives you the option to allow the programmer to treat them as safe if he wants to.

On 3/27/2020 2:02 PM, Jonathan M Davis wrote:
> I don't know if we can reasonably change how @trusted is treated in name
> mangling at this point, but I definitely think that it was a mistake to
> distinguish between @safe and @trusted with name mangling.

If @trusted wasn't part of the mangling, one could not turn the mangling back into the function signature.

On Saturday, April 4, 2020 12:53:57 AM MDT Walter Bright via Digitalmars-d wrote:
> On 4/3/2020 2:06 PM, Steven Schveighoffer wrote:
> > I want to make sure you understand that we are not talking about extern(C) functions that are written in D.
>
> I understand totally what you are talking about.
>
> > But what should absolutely not compile is:
> >
> > extern(C) int free(void *);
> >
> > void foo(int *ptr) // now inferred @safe
> > {
> >
> >     free(ptr);
> >
> > }
>
> I understand your proposal. You want C functions without bodies to be @system.

Anything else would go against what @safe is supposed to be promising. @safe means that the compiler verified the function for memory safety, which it obviously didn't do in the case of extern(C) declarations. You're needlessly putting a hole in the @safety system as part of making @safe the default. We already have enough problems with @trusted being used incorrectly without the compiler blindly applying it to declarations by assuming that they're @safe with no verification whatsoever.

> > The fact that we cannot control where/how people define their prototypes means we have to be firm on this. They need to opt-in to @safe with extern(C), it cannot be default.
>
> On the other hand, special cases like this tend to cause unexpected problems in the future. Experience pretty much guarantees it. It's likely to be tricky to implement as well.
>
> People remember simple rules. They don't remember rules with odd exceptions to them, that always winds up with trouble and bug reports. Simple rules applied evenly lead to a compiler that works and is reliable. I'm afraid the weight of all the special rules will crush D.

I would have thought that a rule that function declarations which are not extern(D) would be @system by default would be very clear and simple. But if it's not, would it be more straightforward to just make all function declarations be treated as @system by default? Or would it be simpler to just say that anything that isn't extern(D) is @system by default? Or would it be simpler to just outright require that all non-extern(D) functions be explicitly marked by the programmer instead of assuming anything?

There has got to be a straightforward rule here that doesn't involve the compiler assuming that code is @safe when it hasn't verified it, and the programmer hasn't told the compiler that it's @trusted. If this DIP is accepted as-is, it will no longer be the case that you can find all memory safety problems problems by searching for @trusted code. All non-extern(D) functions will be a potential problem and will have to be checked for whether they've been explicitly marked with any attributes by the programmer or were simply assumed to be @safe by the compiler, whereas right now, they have to be marked with @trusted in order to treated as @safe, just like any function body where the compiler can't verify that it's memory safe.

> Now, as to what to do. I spent a few minutes and added `@system:` in to the C headers in druntime for windows, posix, and linux. Done. I hope someone else will do it for freebsd, etc., if not I'll go do that to.

If the programmer has to explicitly mark extern(C) declarations as @system just so that they're not @safe, then you're basically turning @trusted on its head. Instead of the compiler guaranteeing that something is memory safe and only treating something that it can't guarantee is memory safe as memory safe if the programmer marks it as @trusted, the compiler is then assuming that these functions are @safe, because it can't verify that they're not, forcing the programming to mark them with @system when they're not. I really don't understand how that can be viewed as acceptable given than @safe is supposed to be about compiler-verified memory safety.

And remember that not all extern(C) declarations are in druntime. Plenty of programmers interface with other C libraries. So, going through druntime and making sure that all of the extern(C) declarations in it are marked appropriately doesn't solve the problem. It just makes sure that those particular extern(C) declarations aren't being incorrectly treated as @safe by the compiler.

It very much sounds like you're putting a hole in @safe, because you think that it's simpler to have a hole than it is to have the compiler tell the programmer that they need to actually verify extern(C) declarations for memory safety. This situation reminds me of how you tried to make it so that array bounds checking went away even in @safe code. If you remove the checks, you remove the guarantees of memory safety.

>  > is going to cause huge issues.
>
> I doubt that for the simple reason that @system by default has not caused huge issues.
>
> The rule is simple:
>
> "For a D module with a bunch of C declarations in it, start it with `@system:`."
>
> It's not a hard rule to check. It's one line. D's C interface has always relied on the user to get right. Recall that C doesn't do any name mangling at all:
>
>    ----- mylib.di
>      extern (C) int addOne(int i);
>
>    ----- mylib.c
>      double addOne(double d) { return d + 1; }
>
>
> That'll link fine and yet fail at runtime. Oops! Calling it @system will not help at all. If the C implementation is bad, there's not a damn thing D can do about it, with or without @system. It's always going to be this way.

The key difference is that right now, the programmer has to actually tell the compiler that a particular C declaration is @trusted for it to be treated as @safe. If they fail to take the time to mark an extern(C) declaration approriately or they miss it for whatever reason, the code using it will have to deal with the fact that it wasn't marked properly. There will be errors when @safe code tries to use it, and the programmers working on that code will know that they have to make sure that what they're doing can be @trusted. With this DIP, as it currently stands, those mistakes will then be invisible, and it's much less likely that they will be caught. It goes completely against the idea that the compiler only treats code as @safe if it can actually verify it that is or if the programmer has told the compiler that it is.

Of course, the programmer can use @trusted incorrectly and screw themselves over, but then at least when the program has problems due to memory safety bugs, you only have to track down the @trusted code to see what code you have to examine for memory safety bugs. None of it is invisible, and the compiler is not claiming that anything is @safe when it's not.

Personally, I would much rather see @safe not be the default than to have holes like this put in it. Yes, as things stand, too much code is left as @system, and too often, programmers don't take the time to make sure that their code is @safe where it can be. But at least the compiler isn't claiming that something is @safe when it hasn't actually verified that it is. And I really don't see why it would be a big problem to either treat all non-extern(D) function declarations as @system by default (or even all function declarations as @system by default if that's simpler) or to just simply require that the programmer explicitly mark them. It would avoid introducing holes into @safe while still allowing us to have @safe be the default, and it seems really simple to explain and easy to understand. Instead, with this DIP, we'll have to be explaining to people why they have to be extremely wary of any extern(C) declarations in their code leading to @system code being treated as @safe without the compiler saying anything about it.

- Jonathan M Davis

On Saturday, 4 April 2020 at 06:53:57 UTC, Walter Bright wrote:
> On 4/3/2020 2:06 PM, Steven Schveighoffer wrote:
>> I want to make sure you understand that we are not talking about extern(C) functions that are written in D.
>
> I understand totally what you are talking about.
>
>
>> But what should absolutely not compile is:
>> 
>> extern(C) int free(void *);
>> 
>> void foo(int *ptr) // now inferred @safe
>> {
>>     free(ptr);
>> }
>
> I understand your proposal. You want C functions without bodies to be @system.
>
>
>> The fact that we cannot control where/how people define their prototypes means we have to be firm on this. They need to opt-in to @safe with extern(C), it cannot be default.
>
> On the other hand, special cases like this tend to cause unexpected problems in the future. Experience pretty much guarantees it. It's likely to be tricky to implement as well.

How are rules like:
1. All D functions are safe by default
2. All extern functions are system by default

Complicated or special cased? They look simpler than D’s conversion rules or function overloading rules. At the end, any rule can be called a “special case”.

>
> People remember simple rules. They don't remember rules with


Also 60-80% of people stick with defaults. The one thing I consistently see people get wrong in APIs and UX is that they make the choice that does the most damage the default.

I’ll leave this here for insight in to research on defaults: https://uxdesign.cc/design-by-default-the-impact-of-using-default-options-in-user-centered-design-926c4d24385c

> odd exceptions to them, that always winds up with trouble and

I think calling “you need to explicitly mark uncheckable functions as safe” an odd exception is highly exaggerating the situation.

> bug reports. Simple rules applied evenly lead to a compiler that works and is reliable. I'm afraid the weight of all the special rules will crush D.
>
> Now, as to what to do. I spent a few minutes and added `@system:` in to the C headers in druntime for windows, posix, and linux. Done. I hope someone else will do it for freebsd, etc., if not I'll go do that to.
>
> > is going to cause huge issues.
>
> I doubt that for the simple reason that @system by default has not caused huge issues.
>
> The rule is simple:
>
> "For a D module with a bunch of C declarations in it, start it with `@system:`."
>
> It's not a hard rule to check. It's one line. D's C interface has always relied on the user to get right. Recall that C doesn't do any name mangling at all:
>
>   ----- mylib.di
>     extern (C) int addOne(int i);
>
>   ----- mylib.c
>     double addOne(double d) { return d + 1; }
>
>
> That'll link fine and yet fail at runtime. Oops! Calling it @system will not help at all. If the C implementation is bad, there's not a damn thing D can do about it, with or without @system. It's always going to be this way.

And there’s not a damn thing D needs to do about bad C code. But D pretending all C code is safe is a disservice to users who care about their code being safe - and at the end of the day just plainly inaccurate.

On Saturday, 4 April 2020 at 06:53:57 UTC, Walter Bright wrote:

> D's C interface has always relied on the user to get right. Recall that C doesn't do any name mangling at all:
>
>   ----- mylib.di
>     extern (C) int addOne(int i);
>
>   ----- mylib.c
>     double addOne(double d) { return d + 1; }
>
>
> That'll link fine and yet fail at runtime. Oops! Calling it @system will not help at all. If the C implementation is bad, there's not a damn thing D can do about it, with or without @system. It's always going to be this way.

That's not what we are discussing: we are discussing about "memory safety", and marking an extern(C) declaration with @trusted simply tells the reviewer to verify the C API, and check that the programmer has done no mistakes as "really", the C function can't corrupt memory. The same for @trusted wrapper.

I don't want to add more burden to reviewers (check that @system: is present, or what-so-ever different solution) I want MORE mechanical checks and LESS reviewer work.

It's the review of written code the bottleneck in most of companies, not writing code. That's also why I don't like  DIP 1032, readability is much more important over syntactic sugars rules.

On Fri, Apr 03, 2020 at 11:53:57PM -0700, Walter Bright via Digitalmars-d wrote: [...]
> I understand your proposal. You want C functions without bodies to be @system.
[...]
> On the other hand, special cases like this tend to cause unexpected problems in the future. Experience pretty much guarantees it. It's likely to be tricky to implement as well.
> 
> People remember simple rules. They don't remember rules with odd exceptions to them, that always winds up with trouble and bug reports. Simple rules applied evenly lead to a compiler that works and is reliable.

The rule:

	extern(D) => @safe by default
	extern(C) => @system by default

hardly sounds "odd" to me.  It almost verbally describes what C is, and what we want D to be, there's nothing easier to remember.


> I'm afraid the weight of all the special rules will crush D.

Now *that's* an odd statement, considering that you recently just posted that memory safety by default is the way to go, and now you're proposing to add a huge big hole to @safe, and even more ironically, this in the name of making D code safer.

	// Current situation: user forgot to write @system on an
	// extern(C) function, the code fails to compile, the user is
	// informed of it, and makes the appropriate fix:
	extern(C) int myfunc();	// @system by default
	void main() @safe {
		myfunc();	// compile error
	}

	// Proposed situation: user forgot to write @system on an
	// extern(C) function, the code compiles beautifully, the user
	// is not informed of any potential problem, and didn't fix it
	// until it explodes in the customer's production server:
	extern(C) int myfunc();	// @safe by default, but actually @system
	void main() @safe {
		myfunc();	// no compile error, @safe is bypassed
	}

Yep, this certainly makes D all the more memory-safe, and D certainly won't be crushed by the weight of all the newly-added @safety loopholes, nope, not at all! :-P


[...]
> The rule is simple:
> 
> "For a D module with a bunch of C declarations in it, start it with `@system:`."
[...]

Since the rule is this simple, and can be mechanically automated, why is it still left as a burden on the user to do it?  What we're proposing is simply to make this rule mechanically enforced, instead of coding by convention, which we all know all too well how it will end up.  Whatever happened to mechanically-verified correctness vs. coding by convention?


T

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth

On Saturday, 4 April 2020 at 11:57:50 UTC, H. S. Teoh wrote:
> [...]

Here's how I see function safety attributes:

           verify body is @safe?  | callable from @safe code?
-------------------------------------------------------------------
@system  |        NO              |            NO
@trusted |        NO              |            YES
@safe    |        YES             |            YES

The compiler can only verify whether a function is @safe if it has a body.  When it doesn't, it is solely up to the programmer to indicate whether it should be callable from @safe code.

So what about defaults?  If we enable @safe by default on "functions with bodies", then we are telling the compiler to "verify" everything is safe by default.  However, if we enable @safe by default on "functions without bodies", then we are telling the compiler to assume everything is "safe to call" by default.  The two are verify different changes and should not be conflated.  Changing the default for functions with bodies makes some sense:

  function has body => verify it is @safe => callable from @safe code

For function's without bodies, not so much:

  function has NO body => CANNOT verify it is @safe => ??? callable from @safe ???


P.S. Based on my table above, I don't think it makes sense to mark any function without a body as @safe, rather, they are either @system or @trusted.

P.S. Notice that there is a potential 4th attribute that verifies the body is safe, but does not allow it to be called from @safe code.  It's possible for a function to only do @safe things, but also be "unsafe" to call.  However, D has taken the route that if a function's body is @safe, it should also be safe to call it from @safe code. This decision indicates that D ignores function boundaries when it comes to safety.  To me this indicates that functions without bodies should not be marked safe by default, because D treats code safety the same whether or not it's in another function, and the compiler would never assume a block of code is safe without analyzing it.

On 4/4/20 2:53 AM, Walter Bright wrote:
> On 4/3/2020 2:06 PM, Steven Schveighoffer wrote:
>> But what should absolutely not compile is:
>>
>> extern(C) int free(void *);
>>
>> void foo(int *ptr) // now inferred @safe
>> {
>>     free(ptr);
>> }
> 
> I understand your proposal. You want C functions without bodies to be @system.

This is the bare minimum proposal. There are various options as I laid out.

My preference is that ALL extern(C), extern(C++), extern(Objective-c) functions are @system by default, with or without body. This is the least intrusive and most consistent proposal that does not gut all safety guarantees in D.

>> The fact that we cannot control where/how people define their prototypes means we have to be firm on this. They need to opt-in to @safe with extern(C), it cannot be default.
> 
> On the other hand, special cases like this tend to cause unexpected problems in the future. Experience pretty much guarantees it. It's likely to be tricky to implement as well.

This is like saying we shouldn't keep loaded guns in a locked safe because we don't keep our shirts in a locked safe, because it's too complicated and "tricky to implement". Yep, it's not as easy. That's because it's dangerous.

> People remember simple rules. They don't remember rules with odd exceptions to them, that always winds up with trouble and bug reports. Simple rules applied evenly lead to a compiler that works and is reliable. I'm afraid the weight of all the special rules will crush D.

This is not a "special rule" or even a complicated one. It's really simple -- extern(C) functions cannot be trusted, so they need to be @system by default. H.S. Teoh put it perfectly:

> The rule:
> 
> 	extern(D) => @safe by default
> 	extern(C) => @system by default
> 
> hardly sounds "odd" to me.  It almost verbally describes what C is, and
> what we want D to be, there's nothing easier to remember.

Continuing...

> Now, as to what to do. I spent a few minutes and added `@system:` in to the C headers in druntime for windows, posix, and linux. Done. I hope someone else will do it for freebsd, etc., if not I'll go do that to.

I don't even have to look to know that you didn't get them all. There are peppered extern(C) prototypes all over Druntime and Phobos. I pointed out one in another post which you have not replied to. This does not even mention all D code in all repositories which frequently add an extern(C) prototype for functions needed. Our current documentation says [https://dlang.org/spec/interfaceToC.html#calling_c_functions]:

> C functions can be called directly from D. There is no need for wrapper functions, argument swizzling, and the C functions do not need to be put into a separate DLL.
> 
> The C function must be declared and given a calling convention, most likely the "C" calling convention, for example:
> 
> extern (C) int strcmp(const char* string1, const char* string2);
> and then it can be called within D code in the obvious way:
> import std.string;
> int myDfunction(char[] s)
> {
>     return strcmp(std.string.toStringz(s), "foo");
> }

Hey look, there's a safety violation right there! It doesn't say, "import core.stdc.string where I've helpfully already pre-marked your system functions for you" it says, just spit out a prototype (without a @system tag) and you are good to go.

Let's update that documentation, and then wait for the questions "why does it say to use @system?"

"Oh, that's because D decided to trust all C calls as perfectly memory safe, so it's on you to tell the compiler it's not safe. Make sure you do that."

WAT.

> 
>  > is going to cause huge issues.
> 
> I doubt that for the simple reason that @system by default has not caused huge issues.

@system by default is fine! It doesn't violate safety because you have to opt-in to trusting code. @safe by default for code that CANNOT BE CHECKED is wrong. Just plain wrong, and breaks safety completely.

@safe by default for code that CAN be checked is fine, because the compiler will check it. If it's not safe, it won't compile. This is why it's ok to mark @safe by default D functions with implementation, and even extern(D) prototypes (since the name mangling takes into account safety).

> 
> The rule is simple:
> 
> "For a D module with a bunch of C declarations in it, start it with `@system:`."

This is your "simpler" solution? "Don't use the default because it's completely wrong"

And that is not the only place C prototypes are declared. People who have a C function they need to call follow the spec, and add a prototype into their module that is full of D code. Putting @system: at the top isn't a good idea.

> 
> It's not a hard rule to check. It's one line. D's C interface has always relied on the user to get right. Recall that C doesn't do any name mangling at all:
> 
>    ----- mylib.di
>      extern (C) int addOne(int i);
> 
>    ----- mylib.c
>      double addOne(double d) { return d + 1; }
> 
> 
> That'll link fine and yet fail at runtime. Oops! Calling it @system will not help at all. If the C implementation is bad, there's not a damn thing D can do about it, with or without @system. It's always going to be this way.

This is a red herring -- code that doesn't work is not code that I care about. ALL unmarked extern(C) prototypes in existence today THAT ARE CORRECTLY WRITTEN are @system!!! Making them "magically" switch to @safe is wrong, BUT THEY WILL STILL WORK. However D safety will be utterly gutted and destroyed. I can't say this enough.

Consider a scenario:

Before the switch:

Some library has code that does i/o. They use prototypes to interface with the system calls as that's what the spec calls for:

extern(C) size_t read(int fd, void* buf, size_t length);

size_t doRead(int fd, void[] arr)
{
   return read(int fd, &arr[0], arr.length);
}

Now, let's see what user code can do here:

void main()
{
   int[10] arr;
   doRead(0, arr[]);
}

This runs and builds, and is technically fine! But it's not @safe. Which is OK because the user didn't opt-in to safety. This ALSO compiles:

void main()
{
   int[][10] arr;
   doRead(0, arr[]);
}

This compiles and builds and is NOT fine. It's now reading POINTERS out of stdin. It's still not @safe, and the user did not declare it safe, so it's on him (maybe he knows what he's doing).

Now, let's move to the future where your new DIP is the default.

BOTH versions of user code COMPILE, and are treated as @safe!!! Why? simply because the read prototype is now considered @safe.

This SILENTLY still works, and the library function doRead now is incorrectly @safe. Perhaps the user knew what he was doing when he was reading pointers from stdin. Maybe it's OK for now, but the library DEFINITELY is wrong for other uses.

You see, the problem isn't that "someone didn't do it right", it's that the thing that was right before is now wrong. Instantly, code that was completely correct in terms of memory safety is now completely incorrect. And it still silently builds and runs exactly as it did before.

Changes like this should REQUIRE a deprecation and error period.

But the easier thing is to simply avoid marking things safe that aren't safe. With that one change, this whole DIP becomes a benefit rather than the great destructor of all safe code in existence.

Please reconsider your position. This is so important, I think a virtual live discussion is in order if you are not convinced by this. Let me know, I'm working from home for 3 weeks already, I'm fine having one any time.

-Steve

On Saturday, 4 April 2020 at 16:44:48 UTC, Steven Schveighoffer wrote:
> [snip]

For all the discussion about @safe meaning that the compiler has verified that a function is @safe, that's not really what @safe does. @safe is a blacklist, preventing some operations in @safe functions, not a white list that only allows verifiable ones in @safe functions. Granted, I think a whitelist would have been a better idea if it was done from the start, but that's not really what we have and I'm not sure the current conversation appreciates that difference. @safe (currently) doesn't actually verify that everything is safe, rather it just says you can't do these unsafe things. Maybe extern(C) functions with no body should be added to the @safe blacklist? That would mean that if @safe is the default, then those functions won't compile. The user will need to use them through a @trusted interface. Now, I don't like that solution by itself, but at least it is consistent with how @safe is designed (as a blacklist).