Consider the following code excerpted from std.array.join:

  static U trustedCast(U, V)(V v) @trusted { return cast(U) v; }
  return trustedCast!RetType(result);

This is because the compiler would complain that the following line would not be @safe:

  return cast(RetType)(result);

The rationale is "I know it's safe, so I'll create an @trusted wrapper to eliminate the error." What comes next is "that's cumbersome. How about a better syntax:"

  trusted {
     return cast(RetType)(result);
  }

? It's the rationale behind "unsafe" blocks that appear in other languages. It seems like a perfectly reasonable request.

The trouble with it is, what if the cast is converting from an integer to a pointer? That could lead to memory corruption. The code allows a potentially memory corrupting operation to be inserted into code that is otherwise @safe.

The only way to deal with it is to then manually review everything about 'RetType' and 'result' to prove to oneself that it is not converting random bit patterns into pointers. In other words, one is manually reviewing @safe code for memory corruption errors.

This is an abject failure of @safe, @trusted, and @system.

The solution is to regard @trusted as a means of encapsulating unsafe operations, not escaping them. Encapsulating them means that the interface from the @trusted code is such that it is usable from safe code without having to manually review the safe code for memory safety. For example (also from std.array):

  static void trustedMemcopy(T[] dest, T[] src) @trusted
  {
    assert(src.length == dest.length);
    memcpy(dest.ptr, src.ptr, src.length * T.sizeof);
  }

I don't have to review callers of trustedMemory() because it encapsulates an unsafe operation (memcpy) with a safe interface.

The reason @trusted applies only to functions, and not to blocks of code, is that functions are the construct in D that provides an interface. Arbitrary blocks of code do not have a structured interface. Adding @trusted { code } support will encourage incorrect uses like the opening example. The existence of @trusted blocks will require review of every line of code in the function that encloses it, and transitively every function that calls it!

Adding @trusted as a function attribute, on the other hand, only requires review of the function's interface to determine if it is acceptable to use in safe code. Safety review of its callers is unnecessary.

The fact that @trusted is contained in small block doesn't mean rest of @safe function doesn't need to be reviewed. Only difference is "review all manually" vs "review all manually with some help of compiler".

On 2/5/15 3:43 PM, Dicebot wrote:
> The fact that @trusted is contained in small block doesn't mean rest of
> @safe function doesn't need to be reviewed. Only difference is "review
> all manually" vs "review all manually with some help of compiler".

As Steve and others have shown, unsafe code under the disguise of @trusted affects everything. You are fostering an illusion. -- Andrei

On Thursday, 5 February 2015 at 23:43:19 UTC, Dicebot wrote:
> The fact that @trusted is contained in small block doesn't mean rest of @safe function doesn't need to be reviewed. Only difference is "review all manually" vs "review all manually with some help of compiler".

I believe that was also the reasoning behind H.S.Teoh proposal for refining @trusted semantics - to keep help from compiler but mark functions as @trusted to make it clear that it still needs to be reviewed in total.

On 2/5/2015 3:43 PM, Dicebot wrote:
> The fact that @trusted is contained in small block doesn't mean rest of @safe
> function doesn't need to be reviewed. Only difference is "review all manually"
> vs "review all manually with some help of compiler".

I did a review of all uses of @trusted in std.array:

  https://issues.dlang.org/show_bug.cgi?id=14127

About 90% of them resulted in the injection of unsafe code into safe functions, requiring a safety review of those allegedly mechanically checkable functions.

This is an abject failure of the technique of using @trusted as an escape than as encapsulation.

By definition, if an @trusted function presents itself with a safe interface, the calling code does not have to be reviewed. And reviewing the interface is a heluva lot easier than the whole rest of the code.

On Thursday, 5 February 2015 at 23:49:05 UTC, Andrei Alexandrescu wrote:
> On 2/5/15 3:43 PM, Dicebot wrote:
>> The fact that @trusted is contained in small block doesn't mean rest of
>> @safe function doesn't need to be reviewed. Only difference is "review
>> all manually" vs "review all manually with some help of compiler".
>
> As Steve and others have shown, unsafe code under the disguise of @trusted affects everything. You are fostering an illusion. -- Andrei

Steve has shown exactly the opposite, proving quite some of my points ;) But it is not uncommon to see what one wants to see in identical piece of information.

On Friday, 6 February 2015 at 00:04:26 UTC, Walter Bright wrote:
> On 2/5/2015 3:43 PM, Dicebot wrote:
>> The fact that @trusted is contained in small block doesn't mean rest of @safe
>> function doesn't need to be reviewed. Only difference is "review all manually"
>> vs "review all manually with some help of compiler".
>
> I did a review of all uses of @trusted in std.array:
>
>   https://issues.dlang.org/show_bug.cgi?id=14127
>
> About 90% of them resulted in the injection of unsafe code into safe functions, requiring a safety review of those allegedly mechanically checkable functions.

Yes, that was intended and not accidental. Again, we were dealing with limited set of faulty tools. Things got inevitably hacky.

> By definition, if an @trusted function presents itself with a safe interface, the calling code does not have to be reviewed. And reviewing the interface is a heluva lot easier than the whole rest of the code.

I know this definition. It have tried it in practice and concluded as being absolutely useless. There is no way I am going to return back to this broken concept - better to ignore @safe completely as misfeature if you insist on doing things that way.

On 2/5/2015 4:13 PM, Dicebot wrote:
> I know this definition. It have tried it in practice and concluded as being
> absolutely useless. There is no way I am going to return back to this broken
> concept - better to ignore @safe completely as misfeature if you insist on doing
> things that way.

I'm sorry I haven't been able to convince you. I don't have any more arguments other than just repeating myself.

Moving forward, I must insist that use of @trusted in such a way as to make its surrounding context not mechanically checkable is no longer acceptable in Phobos.

If need be, I will rewrite std.array myself to address this.

But D is a systems programming language, not a B&D language, and anyone will be free to ignore @safe and continue to use the other benefits of D, or use @safe in a way that conforms to their own formulation of best practices.

On 2/5/15 5:12 PM, Walter Bright wrote:
> On 2/5/2015 4:13 PM, Dicebot wrote:
>> I know this definition. It have tried it in practice and concluded as
>> being
>> absolutely useless. There is no way I am going to return back to this
>> broken
>> concept - better to ignore @safe completely as misfeature if you
>> insist on doing
>> things that way.
>
> I'm sorry I haven't been able to convince you. I don't have any more
> arguments other than just repeating myself.
>
> Moving forward, I must insist that use of @trusted in such a way as to
> make its surrounding context not mechanically checkable is no longer
> acceptable in Phobos.
>
> If need be, I will rewrite std.array myself to address this.
>
> But D is a systems programming language, not a B&D language, and anyone
> will be free to ignore @safe and continue to use the other benefits of
> D, or use @safe in a way that conforms to their own formulation of best
> practices.

I second this. -- Andrei

On Thu, Feb 05, 2015 at 11:50:05PM +0000, Dicebot via Digitalmars-d wrote:
> On Thursday, 5 February 2015 at 23:43:19 UTC, Dicebot wrote:
> >The fact that @trusted is contained in small block doesn't mean rest of @safe function doesn't need to be reviewed. Only difference is "review all manually" vs "review all manually with some help of compiler".
> 
> I believe that was also the reasoning behind H.S.Teoh proposal for refining @trusted semantics - to keep help from compiler but mark functions as @trusted to make it clear that it still needs to be reviewed in total.

This whole discussion has been very tiring and frustrating, because we keep talking past each other. The reason we keep talking past each other is because we keep conflating several distinct, though related, issues.

1) I think we all basically agree that std.file is a mess and some solution is needed to fix this mess. I don't think anyone is actually advocating *for* the code that's currently in std.file right now. So can we pretty please agree that this is a given, and stop using it to beat each other over the head?

2) I think we also all basically agree that the *intent* of @trusted is to be an encapsulation mechanism, or to present a safe API, or however you want to describe it. I.e., if a function is marked @safe, then it must be impossible to cause it to do something unsafe by passing it the wrong arguments. Whatever it does under the hood should not have any observable unsafe effect on the outside world. I'm pretty sure everyone also agrees with this; I don't think anyone is advocating that we should changee this intent. So can we please also take this as a given, and stop repeating it at each other as if we don't all already know it?

This leaves the core of the contention, which is that @safe/@trusted, as currently implemented, presents maintenance difficulties. Walter & Andrei don't seem to be convinced that this is a problem, but the evidence is right before us. We Phobos committers did not deliberately set out to sabotage @trusted by introducing blatant abuses of it just for fun (see (1) and (2) above). The current ugly (and arguably totally wrong) hacks are a symptom of the underlying maintenance difficulty that the current system presents. It is a desperate attempt to contain a maintenance problem that's growing out of hand.

You probably don't believe me, but let's assume that we all agree on (1) and (2), and let's suppose that everything is implemented the way Walter & Andrei envision it. That is, no ugly hacks like what's currently in std.file, and @trusted is only ever used on functions that present a @safe API. What maintenance issues might there be? There are at least the following (and probably more):

- First, let's get one thing straight: the body of a function is rarely
  only written once and never touched again later. In the real world,
  function bodies tend to be written piecemeal -- you write an initial
  implementation, then refine it, then commit it, then somebody comes
  along later and fixes a bug, somebody else implements an enhancement,
  etc.. I.e., code is in a constant flux of changes. Especially in a
  public project like Phobos, these changes are pouring in everyday from
  all manner of contributors of varying levels of competence. No single
  person can possibly keep up with every code change that gets merged.

Now, with this fact as background, let's consider:

- The body of a @trusted function is completely unrestricted, and can
  contain any manner of @system operation. The compiler doesn't give you
  any help in ensuring there aren't careless mistakes that slipped in.
  Any unsafe operation you can imagine can be freely thrown into such a
  function, and the compiler will happily accept it and allow @safe code
  to freely call it.

- The fact that the compiler does zero verification of the body of a
  @trusted function, should already be a sign of deficiency, since the
  responsibility of checking correctness of the *entire* function body
  now falls upon fallible human beings. How sure are we that even the
  initial review was foolproof? At the very least, one would hope that
  the compiler could give us some help where it can, and leave the parts
  it cannot to a few isolated places where human inspection can be more
  focused on.

- Now, couple this with the volume of changes pouring into Phobos
  constantly, which means that inadvertent mistakes will creep into the
  code, either in the function body directly, or indirectly into any of
  the functions that the @trusted function calls. Some of these mistakes
  may break the @trusted promise of the function, but currently there is
  NO WARNING for this. How confident are we that changes to the function
  after the initial review did not break the @trusted promise?

- The currently recommended approach is to thoroughly review all changes
  to @trusted functions before merging. Given that just today, somebody
  merged a PR that has no unittests for verifying the correctness of a
  bugfix and preventing regressions, how likely is it that changes to
  @trusted function will be thoroughly reviewed for all possible
  implications before merging?

- Not to mention, this is a direct change to a function, that is
  immediately visible from the function itself.  We haven't considered
  yet the implications of changes to helper functions called by @trusted
  functions, which may have safety implications on previously-verified
  @trusted functions. If a direct change passes review without a proper
  accompanying unittest, how much less will changes to functions
  *indirectly* called by @trusted functions cause reviewers to inspect
  the @trusted function that calls them?

  Sure, we *could* institute a draconian policy that we must keep track
  of every change to every function that might be called by a @trusted
  function, and review all those @trusted callers everytime any one of
  these functions change. That would ensure the promise of @trusted is
  not broken. That would also ensure that nobody will maintain Phobos,
  because the burden of such a review requirement is far too onerous to
  be practical.

So basically, once a function is marked @trusted, we basically have to accept its safety based on faith and hope. Since reviewing the entire call tree of the function is impractical, we just need to have faith that whoever reviewed the initial version of the function 5 months ago did his job correctly, and we hope that nothing has changed since that may have broken the safety of the function. External @safe callers of the function simply take its safety on faith, because it presents a @safe API, and there's no reason whatsoever to doubt whether or not it is truly safe. In any case, since the compiler happily accepts @safe code calling @trusted code, and we can't live our lives in fear that perhaps a recent PR has introduced a change that broke the safety promise of said @trusted code, we just have to take it on faith that the @trusted code is, in fact, @safe.

Meanwhile, the changes continue to pour in, and the compiler doesn't even check the most basic @safe-ty concerns in the body of the @trusted function. When a change touches something that causes a previously @safe operation in said function body to become @system, the compiler happily accepts it without any warning or indication whatsoever that perhaps there is some kind of mistake, a previously @safe primitive has now become un-@safe, and we'd better re-review the @trusted function to ensure that no new safety holes have been introduced.  But surely, this isn't a problem at all, since we reviewed this @trusted function 5 months ago, and nothing could possibly have gone wrong in the interim, -- we believe.

(After all, our reviewers are doing their jobs properly, right? That is, the onerous obligation to review not only changes to @trusted functions, but every change to every function that occurs in the downstream call tree of every @trusted function. In a *volunteer* project, where people only work on what interests them. Yes, this just fills me with confidence that there's absolutely nothing wrong with @trusted. What do you mean, the emperor has no clothes?)


T

-- 
Государство делает вид, что платит нам зарплату, а мы делаем вид, что работаем.