On Monday, 25 November 2019 at 20:32:03 UTC, Doc Andrew wrote:
> On Monday, 25 November 2019 at 00:10:41 UTC, Ola Fosheim Grøstad wrote:
>> Looks more like it is defining what authors should not do in unsafe code for Rust to make optimizations under the assumption that unsafe code is wellbehaved. Then they provide an interpreter that dynamically borrow checks tests by running the code... In order to capture situations that rust cannot deal with statically. Or something to that effect.
>>
>> TLDR; they specify undefined behaviour in order to enable optimization.
>>
>
> Yeah, that's about as much as I was able to take from it yesterday. I wasn't sure if the stacked borrow technique might be useful for catching some of the errors that Timon pointed out with the interface between @live and @system code. I think most of us would consider the UB an error, rather than an issue with optimization?

Well, my take speedreading the paper was that they try to arrive at a "memory model" for Rust that enables aliasing optimizations while not limiting unsafe code too much. So they try out different models by building the model into the interpreter and run it on many programs to see how limiting their new rules are.

Anyway, one of the authors has blogged about what he was interested in exploring here:
https://www.ralfj.de/blog/2019/05/21/stacked-borrows-2.1.html

On 11/22/2019 12:08 AM, Johannes Pfau wrote:
> But only if you don't know for sure already that the pointer can't
> possibly be null. With non-null types (by default) such situations are
> rare.
> 
> You should really have a look at TypeScript / Kotlin some time. In
> Javascript, aborting scripts due to null-pointers in untested codepaths
> has be a main reason of bugs and TypeScript really nicely solved this. I
> think non-null types + these automated conversion from nullable to non-
> null are the most important innovations for OOP code in recent years and
> the feature i miss most when doing OOP in D.
> 

I won't say no, but it is off topic for this thread and there's too much on my plate in the near future to think about that (O/B, and move semantics).

On 11/23/2019 3:40 PM, Timon Gehr wrote:
> I don't understand why this is not a concern for _your_ designs, which freely admit to being uncheckable and unsound. You can't say "@safe means memory safe and this is checked by the compiler" and at the same time "@live @safe relies on unchecked conventions to ensure memory safety across the @live boundary".

All languages that have an "unsafe" annotation rely on unchecked conventions when the boundary is crossed.


> I don't think this is the case. The GC-allocated raw pointer allows aliasing while @live does not allow aliasing. They have incompatible aliasing restrictions. It's like having a mutable and an immutable reference to the same memory location.

If I may sum this up, it is you wish to not follow O/B rules in @live functions when you've got a GC pointer. I suspect it is possible to segregate such operations into separate, non-@live functions, and I concede that you'll find this inconvenient.

In your scheme, this issue is resolved by distinguishing the two by annotating the non-GC pointers with `scope`.


> What about the fact that it is _optional_ for a /caller/ to respect the smart pointer's desired ownership restrictions? That's very restrictive for the smart pointer! It won't be able to provide @safe borrowing functionality.

I understand that the salient difference between your scheme and mine is you attach it to the pointer type while mine attaches it to the function. Pretty much all of this discussion is about consequences of this difference, so I don't think it is necessary to go through it point-by-point agreeing with you on those consequences.

But yours has problems, too, so the question is which method is better?

Some problems:

1. Doesn't seem to be a way to prevent un-scope pointers from existing and being unsafe.

2. The splitting pointers down the middle into two categories: scope and un-scope. This is a very intrusive and drastic change. The only saving grace of the existing `scope` and `return` annotations is the compiler can infer the vast bulk of them. Will people accept manually adding `scope` to a big chunk of their types? I don't know.

On 26.11.19 10:34, Walter Bright wrote:
> On 11/23/2019 3:40 PM, Timon Gehr wrote:
>> I don't understand why this is not a concern for _your_ designs, which freely admit to being uncheckable and unsound. You can't say "@safe means memory safe and this is checked by the compiler" and at the same time "@live @safe relies on unchecked conventions to ensure memory safety across the @live boundary".
> 
> All languages that have an "unsafe" annotation rely on unchecked conventions when the boundary is crossed.
> ...

I was very careful _not_ to cross that boundary. For your argument to make sense, crossing the boundary between @live and non-@live needs to be @system.

You can't say that just because @trusted exists, @safe doesn't need to do any checking either. Calling @live from non-@live or the other way around amounts to an unsafe type cast.

> 
>> I don't think this is the case. The GC-allocated raw pointer allows aliasing while @live does not allow aliasing. They have incompatible aliasing restrictions. It's like having a mutable and an immutable reference to the same memory location.
> 
> If I may sum this up, it is you wish to not follow O/B rules in @live functions when you've got a GC pointer.

I don't really want @live functions. I want O/B, but only where it makes sense.

> I suspect it is possible to segregate such operations into separate, non-@live functions,

It is not possible. What if you store a std.container.Array as a field of a class object and want to borrow the internal data? If you null out the class pointer after borrowing the internal array data, the GC may deallocate the internal array data. @live can't protect against such interactions.

> and I concede that you'll find this inconvenient.
> ...

Inconvenient and unsound. I would have to write @system code, and the compiler wouldn't even alert me that there is a problem if I annotate it @safe.

> In your scheme, this issue is resolved by distinguishing the two by annotating the non-GC pointers with `scope`.
> ...

Yes, but `scope` does not track the allocator. `scope` restricts lifetime, and possibly aliasing. As I also said, alternatively, and perhaps preferably, we could annotate aliasing restrictions separately, but the accepted DIP1021 already introduces some checking against aliasing outside @live code.

> 
>> What about the fact that it is _optional_ for a /caller/ to respect the smart pointer's desired ownership restrictions? That's very restrictive for the smart pointer! It won't be able to provide @safe borrowing functionality.
> 
> I understand that the salient difference between your scheme and mine is you attach it to the pointer type while mine attaches it to the function.

You attach it to the function, but the meaning pertains to pointers. This should be a red flag, but apparently you don't think so. Then, you allow @live and non-@live code to interact in a @safe context. This is highly unsound because those calls change the interpretation of pointer types. Those are unsafe pointer casts.

> Pretty much all of this discussion is about consequences of this difference,

Not true. Even if I accept that a function attribute is a reasonable way to go, @live comes short.

> so I don't think it is necessary to go through it point-by-point agreeing with you on those consequences.
> ...

This makes no sense to me.  It seems rather weird to be debating the merits of proposals without actually taking into account the consequences of each proposal.

> But yours has problems, too, so the question is which method is better?
> ...

I don't care about "my" method vs "your" method. I want a method that makes sense, does not break @safe or GC and improves @safe. This can just as well be some new combination.

The issue was that you weren't responding to any of my points until I made some concrete proposal. I am not stuck to that. We can improve it.

> Some problems:
> 
> 1. Doesn't seem to be a way to prevent un-scope pointers from existing and being unsafe.
> ...

It doesn't statically prevent memory corruption in @system code. I don't think this is a "problem".

> 2. The splitting pointers down the middle into two categories: scope and un-scope.

It appears this split exists after https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1021.md

> This is a very intrusive and drastic change.

I think it's less so than splitting functions into two mutually incompatible categories.

> The only saving grace of the existing `scope` and `return` annotations is the compiler can infer the vast bulk of them.

But now there's DIP1021, which checks (unsoundly) that `scope` pointers don't alias. This is not my invention and I would be on board with reverting the decision on DIP1021.

> Will people accept manually adding `scope` to a big chunk of their types? I don't know.

If those types want manage memory manually and expose the internal memory directly to a @safe caller, yes.
If the @safe caller wants to pass around manually-managed memory it will have to annotate stuff as `scope`. This is also true with @live.


If you want to avoid intrusive and drastic language changes, what about reverting DIP1021, moving aliasing checks to run time? We could add opBorrow and opUnborrow primitives. opBorrow would return a scoped value borrowed from the receiver and opUnborrow would be called once the last borrow ends. This would even be quite a bit more precise than doing everything in the type system, because you would only prevent invalidation, not all mutation.

On Tuesday, 26 November 2019 at 13:07:07 UTC, Timon Gehr wrote:
> Yes, but `scope` does not track the allocator. `scope` restricts lifetime, and possibly aliasing. As I also said, alternatively, and perhaps preferably, we could annotate aliasing restrictions separately, but the accepted DIP1021 already introduces some checking against aliasing outside @live code.

One possibility for thread local objects:

Add "lifetime-names" and let scope be one, and deduce as many as possible. Then have a stack of thread local allocators (i.e. arena).

Then let lifetime-name track the allocator-position on that stack.

Then only allow the popping of an allocator when there are no pointers carrying the life-time name with allocator-stack-depth 0 (top).

The problem is shared objects...

On Tuesday, 26 November 2019 at 09:34:55 UTC, Walter Bright wrote:
>
> I understand that the salient difference between your scheme and mine is you attach it to the pointer type while mine attaches it to the function. Pretty much all of this discussion is about consequences of this difference, so I don't think it is necessary to go through it point-by-point agreeing with you on those consequences.
>

I think another great difference is that in your proposal a pointer has ownership of the memory but I think that structs should have ownership over resources and pointers are only borrows (like references in Rust), this way is more useful and lets you track ownership over other resources other than memory.

Another point is that on Timon's proposal scope pointers can be members of structs so you can have a safe slice that borrows from a container.

On 11/26/2019 9:31 AM, victoroak wrote:
> I think another great difference is that in your proposal a pointer has ownership of the memory but I think that structs should have ownership over resources and pointers are only borrows (like references in Rust), this way is more useful and lets you track ownership over other resources other than memory.
> 
> Another point is that on Timon's proposal scope pointers can be members of structs so you can have a safe slice that borrows from a container.

That's the way both work.

After having some time to think about the messages here and what the current specification document says, I have gotten a bit concerned about the complexity involved in having multiple states a pointer can be in.

Alternatively what I have derived from these discussions is that what is being described is essentially a head const reference type tied to lifetime (i.e. DIP25).

With a head const type, I think that we might be able to tick all the boxes without much iteration:

- Non-null
- Always points to valid memory
- Pointer cannot be modified _anywhere_ (this includes dynamic arrays appending, but not subslicing)
- Not possible to free (can't & or cast to void*)
- Tied to a point in the stack at CT with predetermined point of time deallocation

I wrote up some code which basically acts as a proposal for it (using ``const ref`` as syntax, but that is not important).

https://gist.github.com/rikkimax/4cb2cc8ddcac33c1a9bb20de432f9dea

Am I completely bonkers to think that this might be a workable solution?

On Wednesday, 27 November 2019 at 10:24:57 UTC, rikki cattermole wrote:
> Am I completely bonkers to think that this might be a workable solution?

You need to ensure at least the following:

- nobody reads the pointer and puts it somewhere else
- nobody reads any pointer reachable from the root pointer and puts it somewhere else
- the root pointer does not point to any datatype that can be modified/extended with new pointers (like a dynamic array of pointers or a non-const pointer)

But that isn't enough, is it? You could have an integer-id as parameter that index a global array with pointers... so it also has to be @pure.

To get to something useful the common assumption should be that you need lifetime-names/variables under the hood. No point in wasting time thinking otherwise... unless someone has written a paper on it.

On Wednesday, 27 November 2019 at 11:12:55 UTC, Ola Fosheim Grøstad wrote:
> To get to something useful the common assumption should be that you need lifetime-names/variables under the hood. No point in wasting time thinking otherwise... unless someone has written a paper on it.

That said, lifetime-names works ok with GC. The lifetime of GC objects lasts until the end of the program, so you can just mark it as "*" or "forever"...