On Wednesday, 27 May 2020 at 00:50:26 UTC, Paul Backus wrote:
> On Tuesday, 26 May 2020 at 22:52:09 UTC, ag0aep6g wrote:
>> But yeah, the @trusted function might rely on the @safe function returning 42. And when it suddenly returns 43, all hell breaks loose. There doesn't need to be any monkey business with unsafe aliasing or such. Just an @safe function returning an unexpected value.
>>
>> I suppose the only ways to catch that kind of thing would be to forbid calling @safe (and other @trusted?) functions from @trusted (and @system?) code, or to mandate that the exact behavior of @safe functions (including their return values) cannot be relied upon for safety. Those would be really, really tough sells.
>
> All that's necessary is to have the @trusted function check that the assumption it's relying on is actually true:
>
> @safe int foo() { ... }
>
> @trusted void bar() {
>     int fooResult = foo();
>     assert(fooResult == 42);
>     // proceed accordingly
> }
>
> If the assumption is violated, the program will crash at runtime rather than potentially corrupt memory.

Exactly: the whole point in trusted it’s that it MUST check the parameters it’s using to call system code, and with an assert that’s just fine in this case, as it’s checking invariants in the program logic.

The only way for a safe program to corrupt memory should be inside the external system code binary: a bug in a used system library, a misunderstanding of the API documentation, and so on.

On 27.05.20 02:50, Paul Backus wrote:
> On Tuesday, 26 May 2020 at 22:52:09 UTC, ag0aep6g wrote:
>> But yeah, the @trusted function might rely on the @safe function returning 42. And when it suddenly returns 43, all hell breaks loose. There doesn't need to be any monkey business with unsafe aliasing or such. Just an @safe function returning an unexpected value.
>>
>> I suppose the only ways to catch that kind of thing would be to forbid calling @safe (and other @trusted?) functions from @trusted (and @system?) code, or to mandate that the exact behavior of @safe functions (including their return values) cannot be relied upon for safety. Those would be really, really tough sells.
> 
> All that's necessary is to have the @trusted function check that the assumption it's relying on is actually true:
> 
> @safe int foo() { ... }
> 
> @trusted void bar() {
>      int fooResult = foo();
>      assert(fooResult == 42);
>      // proceed accordingly
> }
> 
> If the assumption is violated, the program will crash at runtime rather than potentially corrupt memory.

I.e., "the exact behavior of @safe functions (including their return values) cannot be relied upon for safety". I think it's going to be hard selling that to users. Especially, because there is no such requirement when calling @system functions.

Say you have this code:

    void f() @trusted
    {
        import core.stdc.string: strlen;
        import std.stdio: writeln;
        char[5] buf = "foo\0\0";
        char last_char = buf.ptr[strlen(buf.ptr) - 1];
        writeln(last_char);
    }

That's ok, right? `f` doesn't have to verify that `strlen` returns a value that is in bounds. It's allowed to assume that `strlen` counts until the first null byte.

Now you realize that you can calculate the length of the string more safely than C's strlen does, so you change the code to:

    size_t my_strlen(ref char[5] buf) @safe
    {
        foreach (i; 0 .. buf.length) if (buf[i] == '\0') return i;
        return buf.length;
    }
    void f() @trusted
    {
        import std.stdio: writeln;
        char[5] buf = "foo\0\0";
        char last_char = buf.ptr[my_strlen(buf) - 1];
        writeln(last_char);
    }

Nice. Now you're safe even if you forget to put a null-terminator into the buffer.

But oh no, `my_strlen` is @safe. That means `f` cannot assume that the returned value is in bounds. It now has to verify that. Somehow, it's harder to call the @safe function correctly than the @system one.

What user is going to remember those subtleties?

On 5/26/2020 10:23 AM, Timon Gehr wrote:
> On 26.05.20 15:19, ag0aep6g wrote:
>> On 26.05.20 11:35, Walter Bright wrote:
>>> On 5/26/2020 12:07 AM, Timon Gehr wrote:
>>>> I don't think so. @trusted code can't rely on @safe code behaving a certain way to ensure memory safety, it has to be defensive.
>>>
>>> I agree. The trusted code here is not passing safe arguments to g(), but it is trusted to do so.
>>
>> Nice. Timon and Walter agree on something related to safety.
> 
> (We agree on many things. It just does not seem that way because I seldomly get involved when I agree with a decision.)

It is indeed nice when we agree. I appreciate it.

On 5/27/20 2:36 AM, ag0aep6g wrote:
> On 27.05.20 02:50, Paul Backus wrote:
>> On Tuesday, 26 May 2020 at 22:52:09 UTC, ag0aep6g wrote:
>>> But yeah, the @trusted function might rely on the @safe function returning 42. And when it suddenly returns 43, all hell breaks loose. There doesn't need to be any monkey business with unsafe aliasing or such. Just an @safe function returning an unexpected value.
>>>
>>> I suppose the only ways to catch that kind of thing would be to forbid calling @safe (and other @trusted?) functions from @trusted (and @system?) code, or to mandate that the exact behavior of @safe functions (including their return values) cannot be relied upon for safety. Those would be really, really tough sells.
>>
>> All that's necessary is to have the @trusted function check that the assumption it's relying on is actually true:
>>
>> @safe int foo() { ... }
>>
>> @trusted void bar() {
>>      int fooResult = foo();
>>      assert(fooResult == 42);
>>      // proceed accordingly
>> }
>>
>> If the assumption is violated, the program will crash at runtime rather than potentially corrupt memory.
> 
> I.e., "the exact behavior of @safe functions (including their return values) cannot be relied upon for safety". I think it's going to be hard selling that to users. Especially, because there is no such requirement when calling @system functions.
> 
> Say you have this code:
> 
>      void f() @trusted
>      {
>          import core.stdc.string: strlen;
>          import std.stdio: writeln;
>          char[5] buf = "foo\0\0";
>          char last_char = buf.ptr[strlen(buf.ptr) - 1];
>          writeln(last_char);
>      }
> 
> That's ok, right? `f` doesn't have to verify that `strlen` returns a value that is in bounds. It's allowed to assume that `strlen` counts until the first null byte.
> 
> Now you realize that you can calculate the length of the string more safely than C's strlen does, so you change the code to:
> 
>      size_t my_strlen(ref char[5] buf) @safe
>      {
>          foreach (i; 0 .. buf.length) if (buf[i] == '\0') return i;
>          return buf.length;
>      }
>      void f() @trusted
>      {
>          import std.stdio: writeln;
>          char[5] buf = "foo\0\0";
>          char last_char = buf.ptr[my_strlen(buf) - 1];
>          writeln(last_char);
>      }
> 
> Nice. Now you're safe even if you forget to put a null-terminator into the buffer.
> 
> But oh no, `my_strlen` is @safe. That means `f` cannot assume that the returned value is in bounds. It now has to verify that. Somehow, it's harder to call the @safe function correctly than the @system one.

I think this is not the way to view it. @safe code still should do what it's supposed to do. It's not any harder *or any easier* to call @safe code.

You are going to have to trust that whatever functions you call (@trusted, @safe, or @system) are following their spec. @safe has additional restrictions, so you can assume more. But the semantic meaning of things cannot be checked by the compiler, and those are the interesting things that cause bugs.

A more realistic example, and one that gets me all the time is something like indexOf. Does it return input.length or -1 if the item isn't found? Using the wrong expectation can lead to bad consequences. So it's important, no matter the safety of the indexOf function, to know what it's supposed to do, and base your review of trusted code on that knowledge.

Of course, with something like that, one could be extra cautious, and assert the value is within bounds if it's not the sentinel. You could be even more cautious and check that the index found has the sought-after element. And that's probably the right defensive way to do this. But who's going to do that? Most people will work under the assumption that indexOf does what it says it's going to do, and not worry about unittesting it on every call.

-Steve

On Wednesday, 27 May 2020 at 12:48:46 UTC, Steven Schveighoffer wrote:
> On 5/27/20 2:36 AM, ag0aep6g wrote:
>>> [...]
's relying on is actually true:
>> [...]
>
> I think this is not the way to view it. @safe code still should do what it's supposed to do. It's not any harder *or any easier* to call @safe code.
>
> [...]



{
  const i = cast(ssize_t) indexof(x, E);
  if (i < 0 || i > x.dim)
  {
    // no luck.
  }
  else
  {
    index is in bounds so use it.
  }
}

On 27.05.20 14:48, Steven Schveighoffer wrote:
> On 5/27/20 2:36 AM, ag0aep6g wrote:
[...]
>> But oh no, `my_strlen` is @safe. That means `f` cannot assume that the returned value is in bounds. It now has to verify that. Somehow, it's harder to call the @safe function correctly than the @system one.
> 
> I think this is not the way to view it. @safe code still should do what it's supposed to do. It's not any harder *or any easier* to call @safe code.
> 
> You are going to have to trust that whatever functions you call (@trusted, @safe, or @system) are following their spec. @safe has additional restrictions, so you can assume more. But the semantic meaning of things cannot be checked by the compiler, and those are the interesting things that cause bugs.
> 
> A more realistic example, and one that gets me all the time is something like indexOf. Does it return input.length or -1 if the item isn't found? Using the wrong expectation can lead to bad consequences. So it's important, no matter the safety of the indexOf function, to know what it's supposed to do, and base your review of trusted code on that knowledge.
> 
> Of course, with something like that, one could be extra cautious, and assert the value is within bounds if it's not the sentinel. You could be even more cautious and check that the index found has the sought-after element. And that's probably the right defensive way to do this. But who's going to do that? Most people will work under the assumption that indexOf does what it says it's going to do, and not worry about unittesting it on every call.

Just to be clear: I agree with you. Requiring @trusted to be so super-defensive doesn't seem viable.

On Wednesday, 27 May 2020 at 06:36:14 UTC, ag0aep6g wrote:
> I.e., "the exact behavior of @safe functions (including their return values) cannot be relied upon for safety". I think it's going to be hard selling that to users. Especially, because there is no such requirement when calling @system functions.

I think you are focused in so closely on this particular example that you are losing track of the bigger picture.

Knowing that a function is @safe guarantees you that (modulo errors in @trusted code) it will not violate memory safety when called with safe arguments. That is the *only* thing @safe guarantees.

If you want to guarantee anything else about what the function does, there are many tools you can use to do so. You can use in and out contracts to establish preconditions and postconditions. You can use assert or enforce in calling code to check that the arguments and/or return value meet some criteria. You can have it accept or return types with invariants. In the case of strlen, you can rely on the wording of the C standard which states that "The strlen function returns the number of characters that precede the terminating null character" to know that if you call it with a null-terminated string (its precondition), the result will be in-bounds (its postcondition).

There's nothing weird or unusual or "a hard sell" about this. Programming is always like this, in any language, memory-safe or not.

On 27.05.20 15:43, Paul Backus wrote:
> I think you are focused in so closely on this particular example that you are losing track of the bigger picture.

That's entirely possible.

> Knowing that a function is @safe guarantees you that (modulo errors in @trusted code) it will not violate memory safety when called with safe arguments. That is the *only* thing @safe guarantees.

Agreed. The question is: What else can @trusted code rely on, beyond the guarantees of @safe?

[...]
> In the case of strlen, you can rely on the wording of the C standard which states that "The strlen function returns the number of characters that precede the terminating null character" to know that if you call it with a null-terminated string (its precondition), the result will be in-bounds (its postcondition).

Ok. An @trusted function can rely on the C standard. I take it that an @trusted function can also rely on other documentation of @system functions. So far I'm with you.

From your previous post I figured that this is your position towards calling @safe from @trusted:

    @trusted code is not allowed to rely on the documented
    return value of an @safe function. The @trusted function
    must instead verify that the actually returned value is
    safe to use.

I'm not sure if I'm representing you correctly, but that position makes sense to me. At the same time, it doesn't seem feasible, because I don't see how we're going to get users to adhere to that.

The way I see it we can either make the rules for @trusted so arcane that practically no one will be able to follow them, or we accept that a change in @safe code can lead to memory corruption. At the moment I'm leaning towards the latter.

On Wednesday, 27 May 2020 at 15:14:17 UTC, ag0aep6g wrote:
> On 27.05.20 15:43, Paul Backus wrote:
>> I think you are focused in so closely on this particular example that you are losing track of the bigger picture.
>
> That's entirely possible.
>
>> Knowing that a function is @safe guarantees you that (modulo errors in @trusted code) it will not violate memory safety when called with safe arguments. That is the *only* thing @safe guarantees.
>
> Agreed. The question is: What else can @trusted code rely on, beyond the guarantees of @safe?

If the specific functions it's calling make any additional guarantees above and beyond what @safe requires, it can rely on those as well.

The question, then, is what constitutes a "guarantee."

> [...]
>> In the case of strlen, you can rely on the wording of the C standard which states that "The strlen function returns the number of characters that precede the terminating null character" to know that if you call it with a null-terminated string (its precondition), the result will be in-bounds (its postcondition).
>
> Ok. An @trusted function can rely on the C standard. I take it that an @trusted function can also rely on other documentation of @system functions. So far I'm with you.

It can, to the extent that you trust the code to conform to the documentation.

Personally, I am willing to trust libc to conform to the C standard, which means that a buggy or non-conforming libc will be able to cause memory corruption in @safe programs that I write. If you want to take a hard-line stance, you should not trust documentation at all.

Note that trusting the D compiler to conform to the D language standard is also, in some sense, "trusting documentation." A bug in the compiler can always introduce memory corruption to @safe code. So "never trust documentation under any circumstances" is not really a tenable position in practice.

> From your previous post I figured that this is your position towards calling @safe from @trusted:
>
>     @trusted code is not allowed to rely on the documented
>     return value of an @safe function. The @trusted function
>     must instead verify that the actually returned value is
>     safe to use.

This is my position on *any* function calling *any other* function. Even in 100% @system code, I must (for example) check the return value of malloc if I want to rely on it not being null.

> I'm not sure if I'm representing you correctly, but that position makes sense to me. At the same time, it doesn't seem feasible, because I don't see how we're going to get users to adhere to that.

Why does it not seem feasible? Checking return values is defensive programming 101. People already do this sort of thing all the time.

I agree that writing correct @trusted code is difficult, and that people are going to make mistakes--just like writing correct C code is difficult, and people make mistakes trying to. I don't think there's anything we can do in the language itself to fix that, other than making it easy to make the @trusted parts of the code as small as possible.

On 5/27/20 8:52 AM, Stefan Koch wrote:
> On Wednesday, 27 May 2020 at 12:48:46 UTC, Steven Schveighoffer wrote:
>> On 5/27/20 2:36 AM, ag0aep6g wrote:
>>>> [...]
> 's relying on is actually true:
>>> [...]
>>
>> I think this is not the way to view it. @safe code still should do what it's supposed to do. It's not any harder *or any easier* to call @safe code.
>>
>> [...]
> 
> 
> 
> {
>    const i = cast(ssize_t) indexof(x, E);
>    if (i < 0 || i > x.dim)
>    {
>      // no luck.
>    }
>    else
>    {
>      index is in bounds so use it.
>    }
> }

Again, I also think this is valid @trusted code:

const i = indexof(x, E);

if(i != -1){
   // use it
}

or

if(i != x.length)
{
   // use it
}

depending on the spec for that function.

I don't think it's 100% necessary to be defensive on all semantics assuming they are not implemented properly.

People just aren't going to write what you wrote in the name of @safe.

They *could* write:

size_t i = indexof(x, E);
if(i < x.length) {
}

But most people aren't going to do that either.

-Steve