On Sunday, 22 February 2015 at 14:41:43 UTC, Peter Alexander wrote:
> On Sunday, 22 February 2015 at 04:19:32 UTC, deadalnix wrote:
>> On Saturday, 21 February 2015 at 22:13:09 UTC, Peter Alexander wrote:
>>> malloc+free can be trusted if wrapped in something like a ref counted pointer, no?
>>
>> Foo bazoom;
>>
>> class Foo {
>>    void bar() {
>>        bazoom = this;
>>    }
>> }
>>
>> void foo() {
>>    RefCounted!Foo f = ...
>>    f.bar();
>>
>>    // bazoom is now a dandling pointer.
>> }
>
> I see, thanks.
>
> Is assigning 'this' from a member function the only problem case?

No. There's also returning the reference from a member function, storing it in a passed-in reference (pointer, ref, out or slice), and passing it to other functions that in turn leak the reference, as well as throwing it. And leaking closures containing the reference.

That's all that I can think of now...

On Sunday, 22 February 2015 at 14:49:37 UTC, Marc Schütz wrote:
> On Sunday, 22 February 2015 at 14:41:43 UTC, Peter Alexander wrote:
>> On Sunday, 22 February 2015 at 04:19:32 UTC, deadalnix wrote:
>>> On Saturday, 21 February 2015 at 22:13:09 UTC, Peter Alexander wrote:
>>>> malloc+free can be trusted if wrapped in something like a ref counted pointer, no?
>>>
>>> Foo bazoom;
>>>
>>> class Foo {
>>>   void bar() {
>>>       bazoom = this;
>>>   }
>>> }
>>>
>>> void foo() {
>>>   RefCounted!Foo f = ...
>>>   f.bar();
>>>
>>>   // bazoom is now a dandling pointer.
>>> }
>>
>> I see, thanks.
>>
>> Is assigning 'this' from a member function the only problem case?
>
> No. There's also returning the reference from a member function, storing it in a passed-in reference (pointer, ref, out or slice), and passing it to other functions that in turn leak the reference, as well as throwing it. And leaking closures containing the reference.
>
> That's all that I can think of now...

Sorry, I meant things other than moving this from a member function to somewhere else, i.e. the RefCounted shouldn't leak the pointer itself in any other way. Just wondering if there are ways to avoid the problem by making 'this' unescapable in certain situations.

On Sunday, 22 February 2015 at 14:54:26 UTC, Peter Alexander wrote:
> On Sunday, 22 February 2015 at 14:49:37 UTC, Marc Schütz wrote:
>> On Sunday, 22 February 2015 at 14:41:43 UTC, Peter Alexander wrote:
>>> On Sunday, 22 February 2015 at 04:19:32 UTC, deadalnix wrote:
>>>> On Saturday, 21 February 2015 at 22:13:09 UTC, Peter Alexander wrote:
>>>>> malloc+free can be trusted if wrapped in something like a ref counted pointer, no?
>>>>
>>>> Foo bazoom;
>>>>
>>>> class Foo {
>>>>  void bar() {
>>>>      bazoom = this;
>>>>  }
>>>> }
>>>>
>>>> void foo() {
>>>>  RefCounted!Foo f = ...
>>>>  f.bar();
>>>>
>>>>  // bazoom is now a dandling pointer.
>>>> }
>>>
>>> I see, thanks.
>>>
>>> Is assigning 'this' from a member function the only problem case?
>>
>> No. There's also returning the reference from a member function, storing it in a passed-in reference (pointer, ref, out or slice), and passing it to other functions that in turn leak the reference, as well as throwing it. And leaking closures containing the reference.
>>
>> That's all that I can think of now...
>
> Sorry, I meant things other than moving this from a member function to somewhere else, i.e. the RefCounted shouldn't leak the pointer itself in any other way. Just wondering if there are ways to avoid the problem by making 'this' unescapable in certain situations.

Ah, ok. But escaping the reference is unavoidable if you don't want to pass the RC object around (and you don't, because it's expensive). And the only safe way to escape it is an ownership system.

On 2/22/15 6:49 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>
> No. There's also returning the reference from a member function, storing
> it in a passed-in reference (pointer, ref, out or slice), and passing it
> to other functions that in turn leak the reference, as well as throwing
> it. And leaking closures containing the reference.
>
> That's all that I can think of now...

Consider

class C { ... client code ... }
alias T = RefCounted!C;
... more client code ...

For reference counting to work transparently, access to the symbol "C" must be restricted. RefCounted obviously needs access to it. Client code should never have access to it, even in the definition of C.

That means:

1. client code must not be able to declare variables of type C or issue calls like "new C" etc.

2. The type of "this" in methods of C must be RefCounted!C, not C.

3. Conversions of C to bases of C and interfaces must be forbidden; only their RefCounted!Base versions must be allowed.

4. Returning references to direct members of C must be restricted the same way they are for structs (see http://wiki.dlang.org/DIP25). A GC class object does not have that restriction.

I think reference counting is an important component of a complete solution to resource management. D should implement world-class reference counting for safe code.

For 1-4 above, although I am a staunch supporter of library-exclusive abstractions, I have reached the conclusion there is no way to implement RC in safe code for D classes without changes to the language. The more we realize that as a community the quicker we can move to effect it.



Andrei

On Sunday, 22 February 2015 at 17:01:45 UTC, Andrei Alexandrescu wrote:
> Consider
>
> class C { ... client code ... }
> alias T = RefCounted!C;
> ... more client code ...
>
> For reference counting to work transparently, access to the symbol "C" must be restricted. RefCounted obviously needs access to it. Client code should never have access to it, even in the definition of C.
>

What ??? That mean writing all library code twice, for client that want GC and for these who don't. That is a looser strategy.

> That means:
>
> 1. client code must not be able to declare variables of type C or issue calls like "new C" etc.
>
> 2. The type of "this" in methods of C must be RefCounted!C, not C.
>
> 3. Conversions of C to bases of C and interfaces must be forbidden; only their RefCounted!Base versions must be allowed.
>
> 4. Returning references to direct members of C must be restricted the same way they are for structs (see http://wiki.dlang.org/DIP25). A GC class object does not have that restriction.
>
> I think reference counting is an important component of a complete solution to resource management. D should implement world-class reference counting for safe code.
>

Sounds like a world class RC would not force all the code to be written twice.

> For 1-4 above, although I am a staunch supporter of library-exclusive abstractions, I have reached the conclusion there is no way to implement RC in safe code for D classes without changes to the language. The more we realize that as a community the quicker we can move to effect it.
>

I don't think we want to implement RC in the language, but implement what would allow to have safe RC as library.

On 2/22/15 10:26 AM, deadalnix wrote:
> On Sunday, 22 February 2015 at 17:01:45 UTC, Andrei Alexandrescu wrote:
>> Consider
>>
>> class C { ... client code ... }
>> alias T = RefCounted!C;
>> ... more client code ...
>>
>> For reference counting to work transparently, access to the symbol "C"
>> must be restricted. RefCounted obviously needs access to it. Client
>> code should never have access to it, even in the definition of C.
>>
>
> What ??? That mean writing all library code twice, for client that want
> GC and for these who don't.

I'm not 100% convinced but it seems to me RC vs. GC is a class design time decision.

> That is a looser strategy.

I'm sure there are tighter ones :o).


Andrei

On Sunday, 22 February 2015 at 20:48:58 UTC, Andrei Alexandrescu wrote:
>> What ??? That mean writing all library code twice, for client that want
>> GC and for these who don't.
>
> I'm not 100% convinced but it seems to me RC vs. GC is a class design time decision.
>

The right strategy for memory management depend on the usage you'll do of an object, not of the object itself.

Or, in other terms, the client code know more about the adapted memory management. Complex library solution probably need to adopt some strategy, but for most library (especially for something like phobos), it make sense to offload that choice on the user.

>> That is a looser strategy.
>
> I'm sure there are tighter ones :o).
>

Padam tshhhhhh !

On Sunday, 22 February 2015 at 17:01:45 UTC, Andrei Alexandrescu wrote:
> On 2/22/15 6:49 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm@gmx.net>" wrote:
>>
>> No. There's also returning the reference from a member function, storing
>> it in a passed-in reference (pointer, ref, out or slice), and passing it
>> to other functions that in turn leak the reference, as well as throwing
>> it. And leaking closures containing the reference.
>>
>> That's all that I can think of now...
>
> Consider
>
> class C { ... client code ... }
> alias T = RefCounted!C;
> ... more client code ...
>
> For reference counting to work transparently, access to the symbol "C" must be restricted. RefCounted obviously needs access to it. Client code should never have access to it, even in the definition of C.
>
> That means:
>
> 1. client code must not be able to declare variables of type C or issue calls like "new C" etc.

No, this would require the class to be specialized for refcounting. But the memory management method needs to be the client code's decision; it can decide to manage some instance by refcounting, some by Unique!C, and leave others to the GC. The class implementer shouldn't need to care about all that.

>
> 2. The type of "this" in methods of C must be RefCounted!C, not C.
>
> 3. Conversions of C to bases of C and interfaces must be forbidden; only their RefCounted!Base versions must be allowed.

These two points have undesirable consequences: All consumers such objects need to be aware of the exact type, which includes the management strategy (RC, Unique, GC). But this is a violation of the principle of separation of concerns: a consumer shouldn't need to have information about the management strategy, it should work equally with `RefCounted!C`, `Unique!C` and bare (GC) `C`, as long as it doesn't take ownership of the resource.

>
> 4. Returning references to direct members of C must be restricted the same way they are for structs (see http://wiki.dlang.org/DIP25). A GC class object does not have that restriction.

This is only a partial solution that doesn't work efficiently with anything other then value members. Slices, pointers and classes require introducing an additional, useless indirection, and that's not the only problem with it.

>
> I think reference counting is an important component of a complete solution to resource management. D should implement world-class reference counting for safe code.
>
> For 1-4 above, although I am a staunch supporter of library-exclusive abstractions, I have reached the conclusion there is no way to implement RC in safe code for D classes without changes to the language. The more we realize that as a community the quicker we can move to effect it.

I'm not sure this is what you're implying, but do you want to restrict it to classes only? Why not structs and slices, too?

Of course I agree that a language change is necessary, but I'm convinced what you suggest above is not the right direction at all. (And I see that deadalnix has already replied and basically said the same thing.)

In general, this and related proposals tend to limit themselves on memory management (as witnessed by the importance that `ref` and `@safe` play in them). This is too narrow IMO. A well thought-out solution can be equally applicable to the broader field of resource management.

On Monday, 23 February 2015 at 14:56:11 UTC, Marc Schütz wrote:
> violation of the principle of separation of concerns: a consumer shouldn't need to have information about the management strategy, it should work equally with `RefCounted!C`, `Unique!C` and bare (GC) `C`, as long as it doesn't take ownership of the resource.

Just be aware that by not making the refcount part of C you either:

1. Need one more indirection.

2. Need a more complicated allocator to avoid alignment padding when C has 32byte/64 byte alignment requirements.

3. Risk having the refcount landing on a different cacheline, causing more cache misses.

If you do excessive refcounting (ARC) and care about performance you actually need to let the implementor of C decide where the RC_counter is embedded...

On Monday, 23 February 2015 at 15:35:52 UTC, Ola Fosheim Grøstad wrote:
> On Monday, 23 February 2015 at 14:56:11 UTC, Marc Schütz wrote:
>> violation of the principle of separation of concerns: a consumer shouldn't need to have information about the management strategy, it should work equally with `RefCounted!C`, `Unique!C` and bare (GC) `C`, as long as it doesn't take ownership of the resource.
>
> Just be aware that by not making the refcount part of C you either:
>
> 1. Need one more indirection.
>
> 2. Need a more complicated allocator to avoid alignment padding when C has 32byte/64 byte alignment requirements.
>
> 3. Risk having the refcount landing on a different cacheline, causing more cache misses.
>

The refcount would be embedded right next to the object by the constructor of `RefCounted!C`, which is responsible for allocating memory for it, using an appropriate allocator.

> If you do excessive refcounting (ARC) and care about performance you actually need to let the implementor of C decide where the RC_counter is embedded...

Of course there can be an additional (template) parameter to `RefCounted`, or it could be a different, user-implemented type altogether.

But my point was that all of `Unique!C`, `RefCounted!C` and `C` should decay to `scope C`. This way, most consumers don't need to be aware of the actual wrapper type (if any) at all.