On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:
> That's what ref counting is about.

That would require pervasive ref-counting.

On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
> http://wiki.dlang.org/DIP77

In general, this is way too much focused on reference counting. The DIP does define "RCO" in more general terms, but still, it feels like the wrong approach.

You also seem to look at Rust's type system from the same POV ("only one mutable reference at a time, for efficiency"), but that's not the point at all: it's about safety in general, not efficiency for RC. It encompasses so much more: memory safety, race-free sharing of data, avoiding iterator invalidation... DIP77 cannot help with general iterator invalidation at all.

Another thing that just feels wrong is the automagic copying. We're passing something by `ref`, and the compiler inserts a copy behind our back?! And this isn't even visible to the user...

Because the topic is correctness, @safe/@system seems the correct way to approach the problem. The compiler should just treat unsafe pass-by-ref as @system, as Martin suggests. Basically, instead of inserting a copy, pass-by-ref will be @system, when no copy would need to be inserted, it will be @safe.

I also think that the unsafety detection heuristic is too simplistic. When scope/return is extended to pointers and other kinds of references, there would probably be way too many false positives. A more sophisticated analysis is necessary that can more realistically determine when mutable aliasing can occur. E.g., it can also take uniqueness into account.

Besides, the address and slice operators also need to be taken into account, and these can appear outside of function calls.

On Thursday, 9 April 2015 at 22:06:01 UTC, Walter Bright wrote:
> I was curious how Rust handled the global variable issue. Turns out it doesn't - mutable global variables are marked as "unsafe" and the checker gives up on it.

I think it's safe to say that if even Rust with its linear type system and advanced borrow checker cannot deal with it, noone can, short of whole program analysis. And it's easy to understand why once you realize that it's all about aliasing. That's why I also suggest to make borrowing from mutable globals unsafe.

On 4/10/15 2:16 AM, ixid wrote:
>> http://wiki.dlang.org/DIP77
>
> We seem to be incrementally edging towards Rust's memory management
> system or something similar and retracing their steps along the path to
> get there. Hypothetically how close could we get to being able to opt
> into a Rust world from the GC world or C worlds of memory management?

We hope to do better than Rust. -- Andrei

On 4/10/2015 4:49 AM, Michel Fortin wrote:
> ...or maybe not. Let's add another field to Big:
>
>      struct Handle
>      {
>          // shouldn't be able to make copies of this
>          @disable this(this) {}
>      }
>
>      struct Big
>      {
>          RCArray!T array;
>          int[1000] ints;
>          Handle handle;
>      }
>
> Now Big's implicit postblit becomes disabled too (because Handle can't handle
> copying), therefore it is no longer a RCO object and the compiler will no longer
> create temporary copies. Now you've trashed memory-safety.

The copy would be made of array, not Big.

On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
> On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
>> http://wiki.dlang.org/DIP77
>
> In general, this is way too much focused on reference counting. The DIP does
> define "RCO" in more general terms, but still, it feels like the wrong approach.
>
> You also seem to look at Rust's type system from the same POV ("only one mutable
> reference at a time, for efficiency"), but that's not the point at all: it's
> about safety in general, not efficiency for RC. It encompasses so much more:
> memory safety, race-free sharing of data, avoiding iterator invalidation...
> DIP77 cannot help with general iterator invalidation at all.

D uses ranges, not iterators.


> Another thing that just feels wrong is the automagic copying. We're passing
> something by `ref`, and the compiler inserts a copy behind our back?! And this
> isn't even visible to the user...
>
> Because the topic is correctness, @safe/@system seems the correct way to
> approach the problem. The compiler should just treat unsafe pass-by-ref as
> @system, as Martin suggests. Basically, instead of inserting a copy, pass-by-ref
> will be @system, when no copy would need to be inserted, it will be @safe.

A difficulty with that is the semantics will change as the compiler improves and gets better at realizing no copy would be needed. Having the @safety be "implementation defined" would be unhappy.


> I also think that the unsafety detection heuristic is too simplistic. When
> scope/return is extended to pointers and other kinds of references, there would
> probably be way too many false positives. A more sophisticated analysis is
> necessary that can more realistically determine when mutable aliasing can occur.
> E.g., it can also take uniqueness into account.
>
> Besides, the address and slice operators also need to be taken into account, and
> these can appear outside of function calls.

I expect any use of RCO's necessarily means the implementation of the RCO is unsafe (after all, it calls free()), but a safe interface is provided. The safe interface to an RCO is tightly controlled with 'return ref'.

On Friday, 10 April 2015 at 17:43:00 UTC, Walter Bright wrote:
> On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>> On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
>>> http://wiki.dlang.org/DIP77
>>
>> In general, this is way too much focused on reference counting. The DIP does
>> define "RCO" in more general terms, but still, it feels like the wrong approach.
>>
>> You also seem to look at Rust's type system from the same POV ("only one mutable
>> reference at a time, for efficiency"), but that's not the point at all: it's
>> about safety in general, not efficiency for RC. It encompasses so much more:
>> memory safety, race-free sharing of data, avoiding iterator invalidation...
>> DIP77 cannot help with general iterator invalidation at all.
>
> D uses ranges, not iterators.

I used "iterator invalidation" because it is an established term. You get the same problems with ranges, once the contents aren't GC managed. And even if they are, your code is merely memory safe, but not necessarily correct. stdin.byLine() without idup is a prime example.

>
>
>> Another thing that just feels wrong is the automagic copying. We're passing
>> something by `ref`, and the compiler inserts a copy behind our back?! And this
>> isn't even visible to the user...
>>
>> Because the topic is correctness, @safe/@system seems the correct way to
>> approach the problem. The compiler should just treat unsafe pass-by-ref as
>> @system, as Martin suggests. Basically, instead of inserting a copy, pass-by-ref
>> will be @system, when no copy would need to be inserted, it will be @safe.
>
> A difficulty with that is the semantics will change as the compiler improves and gets better at realizing no copy would be needed. Having the @safety be "implementation defined" would be unhappy.

Agreed, it needs to be specified. I'm aware that this is difficult. Though, why can't we use the same approach we use in other cases? Be restrictive at first, later accept more, as we refine the specification... At some point we'll find the sweet spot between false negatives and complexity of specification, and can stabilize on it, so that other compilers have a non-moving target.

>
>
>> I also think that the unsafety detection heuristic is too simplistic. When
>> scope/return is extended to pointers and other kinds of references, there would
>> probably be way too many false positives. A more sophisticated analysis is
>> necessary that can more realistically determine when mutable aliasing can occur.
>> E.g., it can also take uniqueness into account.
>>
>> Besides, the address and slice operators also need to be taken into account, and
>> these can appear outside of function calls.
>
> I expect any use of RCO's necessarily means the implementation of the RCO is unsafe (after all, it calls free()), but a safe interface is provided. The safe interface to an RCO is tightly controlled with 'return ref'.

Uhm... how is this related to what I wrote?

Example:

void foo() @safe {
    RCArray!int arr = [0,1,2];
    {
        int* p = &arr[0];  // legal under new scope rules
        RCArray!int other;
        arr = other;
        writeln(*p);       // OOPS
    }
}

This shows that with the extended scope rules (i.e. scope not restricted to `ref`), the borrowing can happen everywhere, not just on function calls. By your rules, every address and slice operator applied to a mutable local would need to trigger the copying, because the resulting (borrowed) reference exists in the same scope as the original variable (owner).

I'd prefer instead to make writing to the owner unsafe, as long as borrowed references to it exist (this also includes potential writes, like passing a mutable ref to another function). This doesn't require copying behind the user's back, and allows the user to choose the solution best suited for the situation: manual copying, deferring the write to the owner, @trusted...

On 04/09/2015 01:10 AM, Walter Bright wrote:
> http://wiki.dlang.org/DIP77

In the first problem example:

 struct S {
     RCArray!T array;
 }
 void main() {
     auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
     foo(s, s.array[0]);           // pass by ref
 }
 void foo(ref S s, ref T t) {
     s.array = RCArray!T([]);      // drop the old s.array
     t.doSomething();              // oops, t is gone
 }

What do you do to pin s.array?

auto tmp = s;

or

auto tmp = s.array;

On 04/10/2015 12:29 PM, Walter Bright wrote:
>>
>> So someone passes an RCO via ref to avoid the inc/dec, and because
>> that imposes
>> safety issues we turn it into some sort of pass by value under the hood,
>> defeating the purpose, and provide an opt-out via @system opAssign.
> 
> Or you could pass it by const ref (which is what Rust essentially does).

Maybe I'm missing something, but this proposal seems to make `ref RCO` fairly useless, because it creates a copy anyhow.

>> Wouldn't it more straightforward to make pass-by-ref unsafe (@system)
>> for RCOs?
> 
> That's what we have now. It's not good enough.

Assigning a RefCounted is marked @system, pass-by-ref is @safe.
What's missing, you want to be able to use RefCounted in @safe code? Why
not pass it by value then? That would pin the object and you could elide
the additional inc/dec just like you propose to elide the temporary copies.
It's more efficient to pass a smart pointer like RefCounted by-value anyhow.

On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
> Example:
>
> void foo() @safe {
>      RCArray!int arr = [0,1,2];
>      {
>          int* p = &arr[0];  // legal under new scope rules

This would be a bad design of an RCO. RCO's must be constructed to not allow pointers to the payload other than by ref.

>          RCArray!int other;
>          arr = other;
>          writeln(*p);       // OOPS
>      }
> }