January 16, 2020
On Wednesday, 15 January 2020 at 18:38:25 UTC, Johannes Pfau wrote:
> Am Wed, 15 Jan 2020 19:06:11 +0100 schrieb ag0aep6g:
>
>> 
>> The real purpose of @trusted in that example is to allow the @system block in the body, and to signal to reviewers and maintainers that the whole function is unsafe despite the mechanical checks that are done on most of the lines. To a user, @trusted functions would still be the same as @safe ones.
>> 
>> Unfortunately, adding the mechanical checks of @safe to @trusted would mean breaking all @trusted code that exists. So implementing that scheme seems unrealistic.
>> 
>
>
> I think it shouldn't be much of a problem, as there is a very nice transition path:
>
> * Add @system block support
> * Add -transition=systemBlocks which enforces @system blocks in trusted
> functions
> * Users gradually add @system blocks to their trusted functions, until
> everything compiles with -transition=systemBlocks. If you did not add all
> blocks yet, your code will still compile fine without -
> transisition=systemBlocks
> * -transition=systemBlocks becomes default
>
>> But as Steven says, it can be done when we use @trusted blocks instead of @system blocks and @safe instead of @trusted on the function. I.e.:
>> 
>>      @safe fun ()
>>      {
>>          // lines that the compiler accepts as @safe go here @trusted {
>>              // these lines are allowed to use @system code
>>          }
>>          // only @safe lines here again
>>      }
>> 
>> It weakens the meaning of @safe somewhat, but it's often treated that way already. There's clearly a need.
>
>
> I don't really like this. It makes @trusted functions completely useless legacy cruft. And there's no longer any way to annotate a function as 'this is 100% safe code', so then you'll have to check every safe function as thoroughly as trusted functions.

Oh, I hate that discussion.
@trusted functions were useless from the start. They are just the same as @safe from a caller perspective. And @trusted blocks are nothing new either, only the syntax is more ugly because anonymous lambdas are used to simulate them:

 () @trusted {
     // @system stuff here
 }()

And the argument that @trusted functions could better be found and that that would increase the safety is invalid. @trusted blocks can be searched just as good, and any function containing them still need to be treated with caution.
January 16, 2020
On Thursday, 16 January 2020 at 15:02:10 UTC, Ola Fosheim Grøstad wrote:
> On Thursday, 16 January 2020 at 14:46:15 UTC, Paul Backus wrote:
>> On Thursday, 16 January 2020 at 10:58:33 UTC, IGotD- wrote:
>>> Then we can remove @safe all together and trust the programmer to only use the safe subset of D.
>>
>> The difference between "a small subset of the program must be manually verified in order to guarantee memory safety" and "the entire program must be manually verified in order to guarantee memory safety" is not insignificant.
>
> If memory-safety is the default then you only need to mark functions and code blocks as unsafe. No need for @trusted or @safe.

This is a non-sequitur. Did you mean to respond to a different message?
January 16, 2020
On Thursday, 16 January 2020 at 15:12:07 UTC, Paul Backus wrote:
> On Thursday, 16 January 2020 at 15:02:10 UTC, Ola Fosheim Grøstad wrote:
>> If memory-safety is the default then you only need to mark functions and code blocks as unsafe. No need for @trusted or @safe.
>
> This is a non-sequitur. Did you mean to respond to a different message?

It wasn't really clear what "IGotD-" meant. Although I suspect he was ironic, but if taken literally it would be fair to say that what D has is pretty much the same as this: assume all code is written as memory safe code and add an escape that allows for writing unsafe constructs adorned with a comment that says that code is trusted... except you also need to mark functions as unsafe. Not really a big shift from Rust, except Rust provides dedicated typing-constructs for doing unsafe operations like dealing with uninitialized variables.
January 16, 2020
On 1/15/20 1:06 PM, ag0aep6g wrote:
> On 15.01.20 17:54, Joseph Rushton Wakeling wrote:
>> On Wednesday, 15 January 2020 at 14:30:02 UTC, Ogi wrote:
> [...]
>>> @safe fun() {
>>>     //safe code here
>>>     @trusted {
>>>         //those few lines that require manual checking
>>>     }
>>>     //we are safe again
>>> }
> [...]
>> So here's the problem with this approach (which was mentioned by several people in the discussion): the actual safety of a function like this is usually down to the combination of the lines that (in your example) are both inside and outside the @trusted block.
> 
> Yup. But a proposal could specify that that's the intended meaning for an @safe function that contains @trusted blocks. Whereas it's more of a cheat when we use @trusted nested functions like that.
> 
> [...]
>> So, a better approach would be for the function to be marked up like this:
>>
>> @trusted fun ()    // alerts the outside user
>> {
>>      // lines that on their own are provably safe go here
>>      @system {
>>          // these lines are allowed to use @system code
>>      }
>>      // only provably safe lines here again
>> }
>>
>> ... and the compiler's behaviour would be to explicitly verify standard @safe rules for all the lines inside the @trusted function _except_ the ones inside a @system { ... } block.
>>
>> Cf. Steven Schveighoffer's remarks here: https://forum.dlang.org/post/qv7t8b$2h2t$1@digitalmars.com
>>
>> This way the function signature gives a clear indicator to the user which functions are provably @safe, and which are safe only on the assumption that the developer has done their job properly.
> 
> I don't think that's what Steven had in mind. In that world, @safe would be very, very limited, because it couldn't be allowed to call @trusted functions. That means @safe would only apply to trivial functions, and @trusted would assume the role that @safe has today. But you'd have to wrap every call from an @trusted to another @trusted function in an @system block. It wouldn't be practical.

> The real purpose of @trusted in that example is to allow the @system block in the body, and to signal to reviewers and maintainers that the whole function is unsafe despite the mechanical checks that are done on most of the lines. To a user, @trusted functions would still be the same as @safe ones.

I'll interject here. I was thinking actually exactly along the lines of Joseph's code, and exactly what you are saying as well. And I think Joe's ideas are along the same lines, just they were misinterpreted here.

There are two things to look at for safety. One is that a function is safe or not safe (that is, it has a safe implementation, even if there are calls to system functions, so therefore is callable from mechanically checked safe code). This is the part where the compiler uses function attributes to determine what is callable and what is not.

The second is how much manual review is needed for the code. This is a signal to the reviewer/reader. In the current regime, the two reasons for marking are muddled -- we don't have a good way to say "this needs manual checking, but I also want the benefits of mechanical checking". This is why I proposed a change to trusted code STILL being mechanically checked, unless you want an escape. This would allow you to mark all code that needs manual review trusted, even if it's mechanically checked (it still needs review if the system-calling parts can muck with the data).

There may be a case as well to make data only accessible from system escapes, because the semantics of the data affect the memory safety of an aggregate.

> 
> Unfortunately, adding the mechanical checks of @safe to @trusted would mean breaking all @trusted code that exists. So implementing that scheme seems unrealistic.

Yeah, most likely. We would possibly need a fourth attribute for this purpose, or we can continue to rely on @safe/@trusted meaning what they mean today (it's doable, though confusing).

> 
> But as Steven says, it can be done when we use @trusted blocks instead of @system blocks and @safe instead of @trusted on the function. I.e.:
> 
>      @safe fun ()
>      {
>          // lines that the compiler accepts as @safe go here
>          @trusted {
>              // these lines are allowed to use @system code
>          }
>          // only @safe lines here again
>      }
> 
> It weakens the meaning of @safe somewhat, but it's often treated that way already. There's clearly a need.

@safe code is tremendously hampered without @trusted. You could only do things like simple math. As soon as you start needing things like memory allocation, or i/o, you need escapes. That is the reality we have.

-Steve
January 16, 2020
On 1/16/20 6:50 AM, Joseph Rushton Wakeling wrote:

> Obviously in general one should not assume virtue on the part of library developers.  But OTOH in a day-to-day practical working scenario, where one has to prioritize how often one wants to deep-dive into implementation details -- versus just taking a function's signature and docs at face value and only enquiring more deeply if something breaks -- it's always useful to have a small hint about the best vs. worst case scenarios.
> 
> It's not that @safe provides a stronger guarantee than @trusted, it's that @trusted makes clear that you are definitely in worst-case territory.  It's not a magic bullet, it's just another data point that helps inform the question of whether one might want to deep-dive up front or not (a decision which might be influenced by plenty of other factors besides memory safety concerns).
> 

In fact, because of how the system works, @safe code is LESS likely to mean what you think.

If you see a @safe function, it just means "some of this is mechanically checked". It doesn't mean that the function is so much more solid than a @trusted function that you can skip the review. It can have trusted escapes that force the whole function into the realm of needing review.

I would say that today, @safe and @trusted are indistinguishable from each other as a user of the function.

If we moved to a scheme more like I was writing about in the post you quoted, then they actually do start to take on a more solid meaning. It's still not fool-proof -- @safe functions can call @trusted functions, which can call @system functions. BUT if everyone does the job they should be doing, then you shouldn't be able to call @trusted functions and corrupt memory, and you should not have to necessarily review @safe functions. There are still cases where you still have to review functions that are @safe which do not have inner functions that are trusted. These are cases where data that is usually accessible to safe functions can cause memory problems in conjunction with trusted functions. When you need to break the rules, it's very hard to contain where the rule breaking stops.

-Steve
January 16, 2020
On Thursday, 16 January 2020 at 15:30:45 UTC, Steven Schveighoffer wrote:
> The second is how much manual review is needed for the code. This is a signal to the reviewer/reader. In the current regime, the two reasons for marking are muddled -- we don't have a good way to say "this needs manual checking, but I also want the benefits of mechanical checking". This is why I proposed a change to trusted code STILL being mechanically checked, unless you want an escape. This would allow you to mark all code that needs manual review trusted, even if it's mechanically checked (it still needs review if the system-calling parts can muck with the data).

As was pointed out @trusted does not achieve much more than a comment, so why not just have a statement- / operator-level escape using a distinguishable and greppable marker like @@. Then you can just prepend that to all function calls or operations that are unsafe:

safe_function(…){
     ptr = …
     //TRUSTED: this is safe because x, y, z
     @@free(ptr);
}

Then leave on any existing mechanical checks and keep adding @@ until it passes.

January 16, 2020
On 1/16/20 10:46 AM, Ola Fosheim Grøstad wrote:
> On Thursday, 16 January 2020 at 15:30:45 UTC, Steven Schveighoffer wrote:
>> The second is how much manual review is needed for the code. This is a signal to the reviewer/reader. In the current regime, the two reasons for marking are muddled -- we don't have a good way to say "this needs manual checking, but I also want the benefits of mechanical checking". This is why I proposed a change to trusted code STILL being mechanically checked, unless you want an escape. This would allow you to mark all code that needs manual review trusted, even if it's mechanically checked (it still needs review if the system-calling parts can muck with the data).
> 
> As was pointed out @trusted does not achieve much more than a comment, so why not just have a statement- / operator-level escape using a distinguishable and greppable marker like @@. Then you can just prepend that to all function calls or operations that are unsafe:
> 
> safe_function(…){
>       ptr = …
>       //TRUSTED: this is safe because x, y, z
>       @@free(ptr);
> }
> 
> Then leave on any existing mechanical checks and keep adding @@ until it passes.
> 

Enforcing the name of the comment helps the greppability. And putting @trusted on the entire function helps avoid grep finding blocks whereby you now have to dig back up to find the context.

@safe functions with no trusted escapes have a much different level of review required. If you can assume all the inputs are valid, then the function shouldn't need review -- even if it calls trusted functions.

I was looking for a distinction from @safe functions like this, and @safe ones with escapes.

I do like the @@ syntax, though @system free(ptr); isn't much worse.

-Steve
January 16, 2020
On Thursday, 16 January 2020 at 15:46:01 UTC, Ola Fosheim Grøstad wrote:
> Then leave on any existing mechanical checks and keep adding @@ until it passes.

Note: for this to work in the general case (with more advanced verification) you most likely need to add an assume construct to tell the compiler what invariants have been upheld by the escaped/unknown code that was called.




January 16, 2020
On Thursday, 16 January 2020 at 15:30:45 UTC, Steven Schveighoffer wrote:
[...]
>> But as Steven says, it can be done when we use @trusted blocks instead of @system blocks and @safe instead of @trusted on the function. I.e.:
>> 
>>      @safe fun ()
>>      {
>>          // lines that the compiler accepts as @safe go here
>>          @trusted {
>>              // these lines are allowed to use @system code
>>          }
>>          // only @safe lines here again
>>      }
>> 
>> It weakens the meaning of @safe somewhat, but it's often treated that way already. There's clearly a need.
>
> @safe code is tremendously hampered without @trusted. You could only do things like simple math. As soon as you start needing things like memory allocation, or i/o, you need escapes. That is the reality we have.

Three ways to design @safe/@trusted:

1) @safe cannot call @trusted. I agree that this would be virtually useless.
2) @safe can call @trusted. @trusted functions must be self-contained safe. This is how @trusted is currently meant to work.
3) @safe code can contain @trusted parts (blocks/lambdas). Those @trusted parts may rely on the surrounding @safe code for safety. The @safe parts are effectively in a third state: Mechanical checks are performed, but manual checks are still needed to verify the interaction with the @trusted parts.

Whenever we disagree, it's about #2 vs #3. #1 is not on the table, as far as I'm concerned.

#3 is how you're treating @trusted in practice. I don't think doing that is strictly speaking valid today. But if we do get @trusted blocks, that's an opportunity to make it valid. I.e., we'd be going from #2 to #3. And that's what I meant by "[weakening] the meaning of @safe".

In contrast, with @system blocks (in @trusted functions) we could make #2 more practical.

To reiterate, I'm ok with both @system blocks and @trusted blocks. I'd even be ok with changing the spec to #3 without getting any kind of block syntax. But that would be harder to formalize, and there's no point in doing it if we want to transition to block syntax anyway.

The only thing I oppose is working with #3 while keeping the officially documented rules at #2. Something should be done about that one way or the other. And if block syntax gets introduced, then that's the time to strike.
January 16, 2020
On Thursday, 16 January 2020 at 15:29:58 UTC, Ola Fosheim Grøstad wrote:
> On Thursday, 16 January 2020 at 15:12:07 UTC, Paul Backus wrote:
>
> It wasn't really clear what "IGotD-" meant. Although I suspect he was ironic, but if taken literally it would be fair to say that what D has is pretty much the same as this: assume all code is written as memory safe code and add an escape that allows for writing unsafe constructs adorned with a comment that says that code is trusted... except you also need to mark functions as unsafe. Not really a big shift from Rust, except Rust provides dedicated typing-constructs for doing unsafe operations like dealing with uninitialized variables.

Yes, kind of.

It similar what ag0aep6g described in alternative 3.

On Thursday, 16 January 2020 at 16:56:17 UTC, ag0aep6g wrote:
> 3) @safe code can contain @trusted parts (blocks/lambdas). Those @trusted parts may rely on the surrounding @safe code for safety. The @safe parts are effectively in a third state: Mechanical checks are performed, but manual checks are still needed to verify the interaction with the @trusted parts.

but the difference is that we don't need @trusted. We can let @system rely on the safety of the surrounding safe block. I can't see how this really different from having that extra @trusted attribute. I might have forgotten to tell that we need this extra @system {} block.