On Wednesday, 4 March 2015 at 08:13:33 UTC, deadalnix wrote:
> On Wednesday, 4 March 2015 at 03:46:36 UTC, Zach the Mystic wrote:
>> That's fine. I like DIP25. It's a start towards stronger safety guarantees. While I'm pretty sure the runtime costs of my proposal are lower than yours, they do require compiler hacking, which means they can wait.
>
> I don't think that it is fine.
>
> At this point we need to :
>  - Not free anything as long as something is alive.
>  - Can't recycle memory.
>  - Keep track of allocated chunk to be able to free them (ie implementing malloc on top of malloc).
>
> It means that RC is attached to an ever growing arena. Code that would manipulate RCArray and append to it on a regular manner must expect some impressive memory consumption.
>
> Even if we manage to do this in phobos (I'm sure we can) it is pretty much guaranteed at this point that noone else will, at least safely. The benefit is reduced because of the bookeeping that need to be done for memory to be freed in addition to reference count themselves.
>
> The #1 argument for DIP25 compared to alternative proposal was its simplicity. I assume at this point that we have empirical evidence that this is NOT the case.

To me, DIP25 is just the first step towards an ownership system. The only language additions you need to it are "out!" parameters, to track escapes to other parameters, "static" parameters (previously called "noscope"), to say that the parameter won't be copied to a global, and one more function attribute (for which I can reuse "noscope" as @noscope) which says the return value will nto be allocated on the heap. All of these will be rare, as they aim to target the exceptional cases rather than the norms ("scope" would be the norm. Hence "@noscope" to target the rare cases):

Examples:

T* fun(return T* a, T* b, T**c);

This signature would indicate complete ownership transferred from `a` to the return value, since only `a` can be returned (see why below)

T* gun(return out!b T* a, T** b);

`a` is declared to be copied both to the return value and to `b`. Therefore it is not owned. (If you're following my previous definition of `out!` in DIP71, you'll notice I moved `out!` to the source parameter rather than the target, but the point is the same.)

T* hun(return T* a) @noscope {
  if(something)
    return a;
  else return new T;
}

Again, no ownership. If you *might* return a heap or global, the function must be marked @noscope (Again I've readapted the word to a new meaning from dIP71. I'm using `static` now for `noscope's original meaning.)

Another example:

T* jun(return static T* a) {
  static T* t;
  t = a;
  return a;
}

Again, no ownership, because of the `static` parameter attribute. In a previous post, you suggested that such an attribute was unnecessary, but an ownership system would require that a given parameter `a` which was returned, not also be copied to a global at the same time. So `static` tells the compiler this, and thus cancels ownership.

My point is that DIP25's `return` parameters are the beginning of an ownership system. An option to specify that the function *will* return a given `return` parameter as opposed to *might* return it is the only thing needed. Hence the additions named above. (Also, `pure` functions will need no `static` parameter attributes, and functions both `pure` and `@nogc` will not need )

With the exception of some minor cosmetic changes, all this is in, or at least hinted at, in my previously posted Reference Safety System:

http://forum.dlang.org/post/offurllmuxjewizxedab@forum.dlang.org

The only thing which bears reiterating is that with better attribute inference, the whole system becomes invisible for most uses.

On Wednesday, 4 March 2015 at 17:13:13 UTC, Zach the Mystic wrote:
> (Also, `pure` functions will need no `static` parameter attributes, and functions both `pure` and `@nogc` will not need )

...will not need `@noscope` either.

On Wednesday, 4 March 2015 at 09:06:01 UTC, Walter Bright wrote:
> On 3/4/2015 12:13 AM, deadalnix wrote:
>> The #1 argument for DIP25 compared to alternative proposal was its simplicity. I
>> assume at this point that we have empirical evidence that this is NOT the case.
>
> The complexity of a free list doesn't remotely compare to that of adding an ownership system.

My reference safety system has ownership built in, more-or-less for free:

http://forum.dlang.org/post/offurllmuxjewizxedab@forum.dlang.org

See also my reply to deadalnix:

http://forum.dlang.org/post/oyaoibmwybzfkhhufpow@forum.dlang.org

On 3/4/15 10:42 AM, Andrei Alexandrescu wrote:
> On 3/4/15 12:55 AM, Ivan Timokhin wrote:
>> Excuse me if I miss something obvious, but:
>>
>>      void main()
>>      {
>>          auto arr = RCArray!int([0]);
>>          foo(arr, arr[0]);
>>      }
>>
>>      void foo(ref RCArray!int arr, ref int val)
>>      {
>>          {
>>              auto copy = arr; //arr's (and copy's) reference counts
>> are both 2
>>              arr = RCArray!int([]); // There is another owner, so arr
>>                                     // forgets about the old payload
>>          } // Last owner of the array ('copy') gets destroyed and happily
>>            // frees the payload.
>>          val = 3; // Oops.
>>      }
>
> That's a problem, thanks very much for pointing it out. -- Andrei

Again, I think this is an issue with the expectation of RCArray. You cannot *save* a ref to an array element, only a ref to the array itself, because you lose control over the reference count.

I don't think arr[0] should correctly bind to foo's second argument.

-Steve

On Wednesday, 4 March 2015 at 07:50:50 UTC, Manu wrote:
> Well you can't get to a subcomponent if not through it's owner.
> If the question is about passing RC objects members to functions, then
> the solution is the same as above, the stack needs a reference to the
> parent before it can pass a pointer to it's member down the line for
> the same reasons.

Yeah, or you could mimic such a reference by wrapping the call in an addRef/release cycle, as a performance optimization.

> The trouble then is what if that member pointer escapes? Well I'd
> imagine that it needs to be a scope pointer (I think we all agree RC
> relies on scope). So a raw pointer to some member of an RC object must
> be scope(*).

I have a whole Reference Safety System which doesn't need explicit scope because it incorporates it implicitly:

http://forum.dlang.org/post/offurllmuxjewizxedab@forum.dlang.org

> That it can't escape, combined with knowledge that the
> stack has a reference to it's owner, guarantees that it won't
> disappear.

I think you and I are on the same page.

On Wednesday, 4 March 2015 at 08:13:33 UTC, deadalnix wrote:
> On Wednesday, 4 March 2015 at 03:46:36 UTC, Zach the Mystic wrote:
>> That's fine. I like DIP25. It's a start towards stronger safety guarantees. While I'm pretty sure the runtime costs of my proposal are lower than yours, they do require compiler hacking, which means they can wait.
>
> I don't think that it is fine.
>
> At this point we need to :
>  - Not free anything as long as something is alive.
>  - Can't recycle memory.
>  - Keep track of allocated chunk to be able to free them (ie implementing malloc on top of malloc).

Well, I don't want to make any enemies. I thought that once the compiler was hacked people could just change their deferred-freeing code.

On Wednesday, 4 March 2015 at 17:22:15 UTC, Steven Schveighoffer wrote:
> Again, I think this is an issue with the expectation of RCArray. You cannot *save* a ref to an array element, only a ref to the array itself, because you lose control over the reference count.

What you need is a special RCSlave type, which is reference counted not to the type of its *own* data, but to its parent's. In this case, a RCArraySlave!(T) holds data of type T, but a pointer to an RCArray, which it decrements when it gets destroyed. This could get expensive, with an extra pointer per instance than a regular T, but it would probably be safe.

On 3/4/15 9:22 AM, Steven Schveighoffer wrote:
> On 3/4/15 10:42 AM, Andrei Alexandrescu wrote:
>> On 3/4/15 12:55 AM, Ivan Timokhin wrote:
>>> Excuse me if I miss something obvious, but:
>>>
>>>      void main()
>>>      {
>>>          auto arr = RCArray!int([0]);
>>>          foo(arr, arr[0]);
>>>      }
>>>
>>>      void foo(ref RCArray!int arr, ref int val)
>>>      {
>>>          {
>>>              auto copy = arr; //arr's (and copy's) reference counts
>>> are both 2
>>>              arr = RCArray!int([]); // There is another owner, so arr
>>>                                     // forgets about the old payload
>>>          } // Last owner of the array ('copy') gets destroyed and
>>> happily
>>>            // frees the payload.
>>>          val = 3; // Oops.
>>>      }
>>
>> That's a problem, thanks very much for pointing it out. -- Andrei
>
> Again, I think this is an issue with the expectation of RCArray. You
> cannot *save* a ref to an array element, only a ref to the array itself,
> because you lose control over the reference count.
>
> I don't think arr[0] should correctly bind to foo's second argument.

Yah, this is a fork in the road: either we solve this with DIP25 + implementation, or we add stricter static checking disallowing two lent references to data in the same scope.

Andrei

On Wednesday, 4 March 2015 at 18:05:52 UTC, Zach the Mystic wrote:
> On Wednesday, 4 March 2015 at 17:22:15 UTC, Steven Schveighoffer wrote:
>> Again, I think this is an issue with the expectation of RCArray. You cannot *save* a ref to an array element, only a ref to the array itself, because you lose control over the reference count.
>
> What you need is a special RCSlave type, which is reference counted not to the type of its *own* data, but to its parent's. In this case, a RCArraySlave!(T) holds data of type T, but a pointer to an RCArray, which it decrements when it gets destroyed. This could get expensive, with an extra pointer per instance than a regular T, but it would probably be safe.

Another solution is to get compiler help. If you know the lifetime of a sub-reference `p.t` to be shorter than of its Rc'd parent `p`, the compiler can wrap its `p.t's lifetime in an addRef/release cycle for P. This works in calling a function:

fun(p, p.t);

Let's say that you know that `p.t` won't escape (a different question). The compiler doesn't need to know about `p.t` to wrap the whole function like this:

p.opAddRef(); // or equivalent
fun(p, p.t);
p.opRelease();

It just needs to know that `p.t's lifetime is shorter than `p's.

On Wednesday, 4 March 2015 at 17:13:13 UTC, Zach the Mystic wrote:
> Another example:
>
> T* jun(return static T* a) {
>   static T* t;
>   t = a;
>   return a;
> }
>
> Again, no ownership, because of the `static` parameter attribute. In a previous post, you suggested that such an attribute was unnecessary, but an ownership system would require that a given parameter `a` which was returned, not also be copied to a global at the same time. So `static` tells the compiler this, and thus cancels ownership.

Actually, I think you convinced me before that `static` (or `noscope`) parameters wouldn't carry their weight. Instead, copying a parameter reference to a global variable is unsafe by default. Wrap it in a `@trusted` lambda if you know what you're doing. (Trusted lambdas are assumed to copy no reference parameters.) In this way, you can assume ownership. Any unsafe global escapes are just ignored. ???