July 26, 2021
On 7/25/2021 8:40 PM, Timon Gehr wrote:
> Note that the @trusted lambda idiom is _basically always_ incorrect @trusted code. Some people do it anyway, because it's convenient.


https://github.com/dlang/dlang.org/pull/3077
July 26, 2021

On Monday, 26 July 2021 at 15:54:06 UTC, Steven Schveighoffer wrote:

>

Consider that the posix function read has the specification that it will read data from a file descriptor, put the data into a passed-in buffer, up to the amount of bytes indicated in the third parameter. Its prototype is:

@system extern(C) int read(int fd, void *ptr, size_t nBytes);

Without reading the code of read, you must conclude from the specification that it does what it says it should do, and not say, ignore nBytes and just use the pointed-at data for as many bytes as it wants. [...]
It's no different from favoriteNumber.

The difference between POSIX read and favoriteNumber is that you can read the source code of favoriteNumber. It's literally right there, in the same module. That's the entire reason why you can be certain it returns 42.

If favoriteNumber and favoriteElement were in different modules, your argument would be correct, because favoriteElement could no longer be certain about which version of favoriteNumber it was calling.

July 27, 2021
On Sunday, 25 July 2021 at 21:42:16 UTC, Ali Çehreli wrote:
> [snip]
> Existing code:
>
> 1) Replace @safe keywords with @trusted where the compiler complains. (No safety lost because @trusted will be behaving exactly like today's @safe.)
>
> 2) Add @system where the compiler complains. This is because all code is @trusted by default and @trusted is safe.
>
> Ali

I'd much rather add new attributes for new behavior than break existing code. If you want @safe to not be able to call @trusted functions, then introduce a new attribute with that behavior.

I think too much is trying to get shoehorned into the @safe/@trusted/@system dichotomy.

I have favored the whitelist/blacklist approach that had been discussed on the forums previously (ignoring bikeshedding on names). @whitelist would only allow operations known to be memory safe (so no @trusted escape hatch). @blacklist would be the opposite of what is currently @safe, in that things that are not allowed in @safe would have to be in a @blacklist function (or block). This would contrast with @system, which basically allows everything in them. Unmarked @system functions would be assumed to be @blacklist, unless they can be inferred to be @whitelist.

Whether @whitelist is orthogonal to @blacklist (such that everything that is not @whitelist must be @blacklist), I am not 100% sure on, but I would lean to putting things in @blacklist only if they are not allowed in @safe currently, which may allow for a middle ground. Regardless, a @whitelist function can only call @whitelist functions. A @blacklist function can call any kind of function it wants.

@safe functions would be able to call @whitelist functions, but nothing would stop them from calling @trusted @blacklist functions. @trusted or @system functions could call either @whitelist or @blacklist functions (the use of these might make it easier to narrow down where safety issues are). However, if either @safe/@trusted/@system functions call @blacklist functions (or include blocks) then they become @blacklist (either explicitly or inferred). So this would distinguish an @safe function that is @whitelist from one that is @blacklist. An @safe function that is @whitelist is verified by the compiler not to only use operations that the compiler verifies are memory safe, while one that is @blacklist will have at some point called @trusted functions that use @blacklist behavior and needs to be manually verified.

This approach doesn't break any code and allows for a lot of the flexibility that I think people want with the @safe system.


July 27, 2021

On Tuesday, 27 July 2021 at 16:36:48 UTC, jmh530 wrote:

>

On Sunday, 25 July 2021 at 21:42:16 UTC, Ali Çehreli wrote:

>

[snip]
Existing code:

  1. Replace @safe keywords with @trusted where the compiler complains. (No safety lost because @trusted will be behaving exactly like today's @safe.)

  2. Add @system where the compiler complains. This is because all code is @trusted by default and @trusted is safe.

Ali

I'd much rather add new attributes for new behavior than break existing code. If you want @safe to not be able to call @trusted functions, then introduce a new attribute with that behavior.

I think too much is trying to get shoehorned into the @safe/@trusted/@system dichotomy.

Presently D is shoehorning two different usages into @safe/@trusted/@system:

  1. the documented usage: @safe for checked @safe functions, @system for unrestricted functions, and @trusted for a manually-reviewed interface between the two. (complaint: @trusted functions check too little)

  2. a usage people have moved to, where the manually-reviewed interfaces are now @trusted function literals embedded inside @safe functions. (complaint: '@trusted blocks' are too hard to review and their containing @safe functions are misleadingly 'safe')

So that's already two ways to write SafeD code and both are found wanting. With @whitelist/@blacklist there'd be a third way. At that point, why not a fourth? I'm sure some people would like a single @unsafe attribute, and maybe it could be justified by a late discovery of a serious complaint with @whitelist/@blacklist. And then we could have a function signature containing @trusted @whitelist @unsafe, and then we could have a "The Nightmare of Memory Safety in D" video at cppcon.

Opposite the Scylla of "breaking code" isn't safe open water, but the Charbydis of "having to explain SafeD to a skeptical newbie." Why five attributes? Because three wasn't enough.

The immediate result of this DIP would also be three SafeD styles:

  1. legacy code doing it as previously documented, #1 above

  2. legacy code doing it with '@trusted blocks', #2 above

  3. the new way of @safe functions without compromise, @trusted functions containing @system blocks, and @system functions

But as #3 solves the problems people had with #1 and #2, the farther-off result is just one SafeD again, this time also with the documentation matching what experts actually do.

July 28, 2021

On Monday, 26 July 2021 at 16:45:07 UTC, jfondren wrote:

>

On Monday, 26 July 2021 at 16:26:53 UTC, claptrap wrote:

>

Its a pointless exercise because your example is a red herring, but this breaks it.
...
And TBH I'm not even sure what your overall point is.

It's a response to overly strong claims about what this DIP will achieve:

https://forum.dlang.org/post/fnhvydmbguyagcmaepih@forum.dlang.org

>

It is a response to the claim that "the compiler's assertions regarding your remaining @safe code might actually mean something." They mean exactly the same thing with your proposal as they do without it: that the @safe portion of the program does not violate the language's memory-safety invariants directly.

If you're saying the proposed "system blocks inside trusted functions" provide no advantage over teh current "trusted lambdas inside safe functions" yes thats true. But I think the point is trusted functions get more checking. Even if you say well you can achieve the same by just using a trusted lambda inside a safe function its not the same once you consider what people actually do.

If you have just one trusted function in your app, then switching to this new regime would automatically give you more checking.

You have to take into account how people will actually behave, even if you can technically achieve the same thing with the current system.

>

And it's again from the perspective of someone reviewing a patch rather than someone troubleshooting a bug. Once there is a memory error in your program, then @safe helps you by telling you to look elsewhere for the direct violation. If you are reviewing patches with an eye towards not including a memory error in your program, then @safe matters a lot less.

This is still mischaracterizing the problem, if you add safe code and it causes a memory error to occur in trusted or system code, the problem was already there. You're not adding a memory error, just changing the conditions so it is triggered.

It would be nice to have a system that would help with problems like that but I think its actually unreasonable, and probably impossible. And it's probably counterproductive to use examples like that to guide the design process. You're designing for constraints that cant be met.

July 28, 2021

On Monday, 26 July 2021 at 18:59:45 UTC, Paul Backus wrote:

>

The difference between POSIX read and favoriteNumber is that you can read the source code of favoriteNumber. It's literally right there, in the same module. That's the entire reason why you can be certain it returns 42.

If favoriteNumber and favoriteElement were in different modules, your argument would be correct, because favoriteElement could no longer be certain about which version of favoriteNumber it was calling.

If you consider the source to be the spec, then that contradicts your earlier suggestion that favoriteNumber can be changed -- its source is the spec, so changing the source to return something other than 42 will violate the spec.

If you consider the source to be the spec and you think changing the spec at will is OK, then we have different philosophies on what a code review and "good software" means.

-Steve

July 28, 2021

On Wednesday, 28 July 2021 at 11:12:14 UTC, Steven Schveighoffer wrote:

>

On Monday, 26 July 2021 at 18:59:45 UTC, Paul Backus wrote:

>

The difference between POSIX read and favoriteNumber is that you can read the source code of favoriteNumber. It's literally right there, in the same module. That's the entire reason why you can be certain it returns 42.

If favoriteNumber and favoriteElement were in different modules, your argument would be correct, because favoriteElement could no longer be certain about which version of favoriteNumber it was calling.

If you consider the source to be the spec, then that contradicts your earlier suggestion that favoriteNumber can be changed -- its source is the spec, so changing the source to return something other than 42 will violate the spec.

If you consider the source to be the spec and you think changing the spec at will is OK, then we have different philosophies on what a code review and "good software" means.

Again, I am in 100% in agreement with you that favoriteElement is not "good software" and should not pass code review. I have never claimed otherwise. (In fact, I spent the rest of that post talking about how we can reject functions like favoriteElement in code review!)

That does not change the fact that in the current version of my example module, favoriteElement is memory safe--meaning, it cannot possibly cause undefined behavior when called with safe values. It is possible for bad software to be memory safe.

It seems to me like we have been talking past each other for most of this discussion, with differences in background assumptions obscured by words like "correct", "valid", "safe", etc. In the future, I will try to be much more careful about defining my terms.

July 28, 2021

On Tuesday, 27 July 2021 at 21:05:52 UTC, jfondren wrote:

>

[snip]

Presently D is shoehorning two different usages into @safe/@trusted/@system:

  1. the documented usage: @safe for checked @safe functions, @system for unrestricted functions, and @trusted for a manually-reviewed interface between the two. (complaint: @trusted functions check too little)

  2. a usage people have moved to, where the manually-reviewed interfaces are now @trusted function literals embedded inside @safe functions. (complaint: '@trusted blocks' are too hard to review and their containing @safe functions are misleadingly 'safe')

I don't use @trusted function literals. I don't have a sense of how popular this usage is. In the context of my suggestion, an @trusted literal that is @blacklist would result in @safe that is @blacklist, which would make it a candidate for review.

>

So that's already two ways to write SafeD code and both are found wanting. With @whitelist/@blacklist there'd be a third way. At that point, why not a fourth? I'm sure some people would like a single @unsafe attribute, and maybe it could be justified by a late discovery of a serious complaint with @whitelist/@blacklist. And then we could have a function signature containing @trusted @whitelist @unsafe, and then we could have a "The Nightmare of Memory Safety in D" video at cppcon.
[snip]

The underlying problem is how to fix design issues with @safe/@trusted/@system while minimizing code breakage. Only the bare minimum to address those issues would make sense. Adding an @unsafe that overlaps with existing or proposed functionality wouldn't make sense.

July 28, 2021

On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:

>

If you're saying the proposed "system blocks inside trusted functions" provide no advantage over teh current "trusted lambdas inside safe functions" yes thats true. But I think the point is trusted functions get more checking. Even if you say well you can achieve the same by just using a trusted lambda inside a safe function its not the same once you consider what people actually do.

If you have just one trusted function in your app, then switching to this new regime would automatically give you more checking.

You have to take into account how people will actually behave, even if you can technically achieve the same thing with the current system.

If I understand correctly, there are two problems being diagnosed in this discussion (overall--not just in your post), and two solutions being proposed:

  • Problem #1: most uses of function-level @trusted are mistakes, and allow too much code to go without automatic checks.

  • Proposed Solution #1: get rid of function-level @trusted as it currently exists. Instead, automatic safety checks will only be disabled in specially-marked blocks (like in Rust). This will encourage programmers to disable automatic checking only for the specific lines of code where it is actually necessary, rather than for the entire function.

  • Problem #2: @trusted lambdas make code harder to review and maintain because they have implicit dependencies on the surrounding @safe code.

  • Proposed Solution #2: ban @trusted lambdas, and force programmers to put their @trusted code into separate functions, which communicate with @safe code explicitly via arguments and return values. (See: Walter's PR).

There is logic to both of these proposals, but their solutions conflict: one pushes for less use of function-level @trusted, the other for more use of it.

July 28, 2021

On Wednesday, 28 July 2021 at 12:49:28 UTC, Paul Backus wrote:

>

On Wednesday, 28 July 2021 at 08:40:47 UTC, claptrap wrote:

>

If I understand correctly, there are two problems being diagnosed in this discussion (overall--not just in your post), and two solutions being proposed:

  • Problem #1: most uses of function-level @trusted are mistakes, and allow too much code to go without automatic checks.

  • Proposed Solution #1: get rid of function-level @trusted as it currently exists. Instead, automatic safety checks will only be disabled in specially-marked blocks (like in Rust). This will encourage programmers to disable automatic checking only for the specific lines of code where it is actually necessary, rather than for the entire function.

  • Problem #2: @trusted lambdas make code harder to review and maintain because they have implicit dependencies on the surrounding @safe code.

  • Proposed Solution #2: ban @trusted lambdas, and force programmers to put their @trusted code into separate functions, which communicate with @safe code explicitly via arguments and return values. (See: Walter's PR).

There is logic to both of these proposals, but their solutions conflict: one pushes for less use of function-level @trusted, the other for more use of it.

Agree 100%.

Maybe you can just ban trusted lambdas from having access to the surrounding scope?

Or ban unsafe blocks from having access to the surrounding scope, but then you need to have a way to pass stuff in, maybe like...

@system (int[] foo, float bar) { ... }

unsafe blocks could be lowered to a trusted lambda maybe?