On Saturday, 27 September 2014 at 19:11:08 UTC, Andrei Alexandrescu wrote:
> On 9/27/14, 1:11 AM, Foo wrote:
>>> Consider:
>>>
>>> struct MyRefCounted
>>>    void opInc();
>>>    void opDec();
>>>    int x;
>>> }
>>>
>>> MyRefCounted a;
>>> a.x = 42;
>>> MyRefCounted b = a;
>>> b.x = 43;
>>>
>>> What is a.x after this?
>>>
>>>
>>> Andrei
>>
>> a.x == 42
>> a.ref_count == 1 (1 for init, +1 for copy, -1 for destruction)
>> b.x == 43
>> b.ref_count == 1 (only init)
>
> So then when does the counter get ever incremented? -- Andrei

increment: by postblit call
decrement: by dtor call

But if you ask me, we should either use a valid library solution like: http://dpaste.dzfl.pl/b146ac2e599a (it is only a draft of 10 min work)
Or we should extend UDA's, so that:
@rc(int) x; is rewritten to: Rc!int x;

Am Thu, 25 Sep 2014 01:27:13 -0700
schrieb Walter Bright <newshound2@digitalmars.com>:

> On 9/25/2014 12:10 AM, Manu via Digitalmars-d wrote:
> >  > I think Microsoft's C++/CLI tried that mixed pointer approach, and it was a
> > disaster. I don't have personal knowledge of this.
> > C++ doesn't have any notion of borrowing. Scope can (will?) fix a whole lot of
> > existing problems, and also allow RC -> raw pointers work nicely.
> 
> Consider that people complain a lot about annotations. See the other thread. Adding the scope annotations everywhere is a LOT of annotations. Do you think people will be happy with that? I don't.
> 
> I remember reading a paper about someone adding pointer annotations to Java. It was a technical success, and a usability failure. People just couldn't be bothered to add the annotations.

You brought up these comparisons with near/far pointer earlier. (They stay as vague.) And now you also argue against borrowing.

Please reconsider, since pointers do differ in their lifetimes. We have that friction already and D is unable to plug the hole. From the top of my head there are several bugs about escaping stack pointers and RC is around the corner.

The extent of what borrowing solves is really yaw dropping when you read the list of use cases and I will surely add scope to every function argument that it applies to (and already do that), because it makes it verifiable safe to call with any pointer type, be it ARC, GC or stack. I consider that more important than const or pure.

And why do you bring up Java? Does Java have any pointer types other than GC? Is it a low level systems programming language? No! So those annotations probably weren't an enabling feature like "scope", right?

-- 
Marco

Marco Leise:

> The extent of what borrowing solves is really yaw dropping
> when you read the list of use cases and I will surely add
> scope to every function argument that it applies to (and
> already do that), because it makes it verifiable safe to call
> with any pointer type, be it ARC, GC or stack. I consider that
> more important than const or pure.

I consider the tracking of memory ownership more important for D than the whole reference counting discussions (despite the two topics are partially linked). Because it's a correctness issue (and will also lead to some performance improvements too).

Bye,
bearophile

On Saturday, 27 September 2014 at 11:54:45 UTC, Dmitry Olshansky wrote:
> 27-Sep-2014 14:23, "Marc Schütz" <schuetzm@gmx.net>" пишет:
>> AFAICS we don't gain anything from this, because it just automates
>> certain things that can already be done manually in a suitably
>> implemented wrapper struct. I don't think automation is necessary here,
>> because realistically, how many RC wrappers will there be? Ideally just
>> one, in Phobos.
>
> You must be missing something big. Ref-counting ain't singular thing, it's a strategy with a multitude of implementations, see my other post.

Ok, you're right about the different possible implementations. Still, your proposal mostly seems to propose separate functions opInc()/opDec() for operations that can just be placed in the wrappers' constructor, destructor and postblit, with no repetition.
(opInitRefcount() is missing in you proposal, strictly speaking; an automatic call to opInc() isn't enough, because initialization could be arbitrarily complex.)

The one thing your proposal enables is wrapper-less reference counting. Is that important?

>> I believe we can achieve the same efficiency without ARC with the help
>> of borrowing and multiple alias this.
>
> Problem is - there is no borrowing yet in the compiler, or maybe you mean something more simple.

No, I'm talking about my proposal.

>
>> Consider the cases where inc/dec
>> can be elided:
>>
>>    RC!int a;
>>    // ...
>>    foo(a);
>>    // ...
>>    bar(a);
>>    // ...
>>
>> Under the assumption that foo() and bar() don't want to keep a copy of
>> their arguments, this is a classical use case for borrowing. No inc/dec
>> is necessary, and none will happen, if RC!int has an alias-this-ed
>> method returning a scoped reference to its payload.
>
> Interesting. However scope must work first, also passing an RC!int by borrowing is this:
>
> void func(scope(A) a)
>
> or what? how does it transform scope? (Sorry I haven't followed your proposals for scope)

That's how it will look from the user's point of view. The RC wrapper's implementer will need to add an alias this that returns the payload with scope:

    struct RC(T) {
        private T payload_;
        scope!this(T) borrow() { return payload_; }
        alias borrow this;
    }

It is then implicitly convertible to scope(T);

>
>
>> On the other hand, foo() and bar() could want to make copies of the
>> refcounted variable. In this case, we still wouldn't need an inc/dec,
>> but we need a way to express that. The solution is another alias-this-ed
>> method that returns a (scoped) BorrowedRC!int, which does not inc/dec on
>> construction/destruction, but does so on copying. (It's probably
>> possible to reuse RC!int for this, a separate type is likely not
>> necessary.)
>
> Who would make sure original RC still exists?

How could it not exist? If it existed immediately before the function call, it will also exist during the call. It can not have gone out of scope before the function returned.

This is already true without borrowing. Borrowing additionally ensures that it even works in special cases, e.g. when the called function makes a copy of the wrapper, and that there will be no references left when the function returns (that is, no references to the wrapper, not the payload).

>
>> The other opportunity is on moving:
>>
>>     void foo() {
>>         RC!int a;
>>         // ....
>>         bar(a);    // last statement in foo()
>>     }
>
> We should already have it with structs by their nature.

Exactly. What I want to say is that here, again, the compiler doesn't need to know that `a` does reference counting in order to generate efficient code. No RC specific optimizations are necessary, thus teaching the compiler about RC doesn't have advantages in this respect.

>
>>
>> Here, clearly `a` isn't used after the tail call. Instead of copy &
>> destroy, the compiler can resort to a move (bare bitcopy). In contrast
>> to C++, this is allowed in D.
>>
>> This covers most opportunities for elision of the ref counting. It only
>> leaves a few corner cases (e.g. `a` no longer used after non-tail calls,
>> accumulated inc/dec as in your example). I don't think these are worth
>> complicating the compiler with ARC.
>
> I don't mind having working scope and borrowing but my proposal doesn't require them.

Well, implement RC in the compiler, and you still have no borrowing. But implement borrowing, and you get efficient RC for free, in addition to all the other nice things it allows.

28-Sep-2014 00:16, Andrei Alexandrescu пишет:
> On 9/27/14, 12:50 PM, Dmitry Olshansky wrote:
>> 27-Sep-2014 23:14, Andrei Alexandrescu пишет:
>>> On 9/27/14, 2:38 AM, Dmitry Olshansky wrote:
>>>>
>>>> Okay it serves no good for me to make these tiny comments while on the
>>>> go.
>>>>
>>>> As usual, structs are value types, so this feature can be mis-used, no
>>>> two thoughts abouts it. It may need a bit of improvement in
>>>> user-friendliness, compiler may help there by auto-detecting common
>>>> misuse.
>>>
>>> I still don't understand what "this feature" is after reading your long
>>> post twice.

Must be my fault, I'll try once again.

>>>
>>> So structs are still value types and you replace postblit/destroy with
>>> calls to opInc/opDec? That's it? How does this enable anything more
>>> interesting than ctors/dtors?
>>>
>>
>> Compiler is aware that opInc and opDec are indeed ref-countinng ops,
>> meaning that opInc + opDec = no op. I claim that this is enough to get
>> "ARC" going.
>
> You give marginal details but still don't describe the thing. When are
> they called and what do they have that ctors/dtors/postblit don't?

The key point is that the change is small, that's why (maybe) it's hard to grasp. The whole thing is a bit of lowering and a "hint" to the compiler. It reuses the same mechanism that structs already have.

Okay a few examples of lowering to get things going:

http://dpaste.dzfl.pl/3722d9d70937

(note I think there could be better lowerings and simpler set of primitives to bootstrap ARC-ed type)

Now why would we need this trivial lowering?

Typical postblit can be anything unless the compiler has full source code, dtor can be anything as well.

With opInc/opDec compiler generates postblit/dtor on his own, in doing so it decorates user-defined dtor that actually clears resources.

What being in control gives to the compiler:

1. Compiler always has the source of generated parts, so they can be inlined (and should be)

2. Can do typical algebra optimization on opInc/opDec, no matter what's inside opInc and opDec (this is a contract between programmer and compiler).
	 e.g opInc(10) followed by OpDec(1) is opInc(9)

3. Also opInc and opDec do not alter object in any capacity, nor are they affected by any method calls on this object (another contract)

4. 1 + 2 = win, as by inlining postblits/dtors we expose opInc/opDecs to the optimizer pass which the would fold them using basic algebra optimizations.

Motivating example:

struct RC(T) { ... } //implemented with or without proposed feature

void foo(RC!int arg); // no source available, sorry;)

void bar(RC!int my)
{	
	foo(my);
	writeln("Bar: ", *my);	
}

which is the same as

{
	{
		auto __tmp = my; // a postblit
		foo(__tmp);
	}//dtor
	writeln("Bar: ", *my);
}

And assuming the same names for inc/dec of count and no exceptions, can be simplified back to:
{
	{
		auto __tmp = my;
		my.opInc(1); // or __tmp.opInc() it's the same count
		foo(__tmp);
		if(my.opDec(1)) my.__dtor();
	}
	writeln("Bar: ", *my);
}

Ideally with assumptions I stated above will look like this:
{
	{
		auto __tmp = my;
		foo(__tmp);
		if(my.opDec(0)) my.__dtor();
	}
	writeln("Bar: ", *my);
}

Now here if compiler can optimize ref-count operations completely on his own without assumptions stated above then we are done and no special opInc/opDec required.

I bet my hat it can't for the simple reason that all bets are off if any of the following is happening:
1. No source available thus e.g. postblit is opaque call
2. Not "everything" in between opInc/opDec can be inlined impairing optimizer (since 'my' is passed to foo, it may be modified)

I might be wrong though - Walter?

>
> FWIW the language always "understands" when to elide postblit/dtor calls.

Yes, which immediately gives us part of ARC advantages such as NRVO elides ref-count bump. The other part comes from batching up multiple incs/decs to one call and eliminating redundant pairs.


I might be wrong but RCObject proposal is harder to construct then this "feather-weight ARC" as it would have to "force" the same lifetime management as we have for structs on some class instances that never had it before!

That being said I think it's valuable to have RCObject and exceptions being ref-counted.

-- 
Dmitry Olshansky

Am 28.09.2014 12:15, schrieb Marco Leise:
> Am Thu, 25 Sep 2014 01:27:13 -0700
> schrieb Walter Bright <newshound2@digitalmars.com>:
>
>> On 9/25/2014 12:10 AM, Manu via Digitalmars-d wrote:
>>>   > I think Microsoft's C++/CLI tried that mixed pointer approach, and it was a
>>> disaster. I don't have personal knowledge of this.
>>> C++ doesn't have any notion of borrowing. Scope can (will?) fix a whole lot of
>>> existing problems, and also allow RC -> raw pointers work nicely.
>>
>> Consider that people complain a lot about annotations. See the other thread.
>> Adding the scope annotations everywhere is a LOT of annotations. Do you think
>> people will be happy with that? I don't.
>>
>> I remember reading a paper about someone adding pointer annotations to Java. It
>> was a technical success, and a usability failure. People just couldn't be
>> bothered to add the annotations.
>
> You brought up these comparisons with near/far pointer earlier.
> (They stay as vague.) And now you also argue against borrowing.
>
> Please reconsider, since pointers do differ in their lifetimes.
> We have that friction already and D is unable to plug the hole.
>  From the top of my head there are several bugs about escaping
> stack pointers and RC is around the corner.
>
> The extent of what borrowing solves is really yaw dropping
> when you read the list of use cases and I will surely add
> scope to every function argument that it applies to (and
> already do that), because it makes it verifiable safe to call
> with any pointer type, be it ARC, GC or stack. I consider that
> more important than const or pure.
>
> And why do you bring up Java? Does Java have any pointer types
> other than GC? Is it a low level systems programming language?
> No! So those annotations probably weren't an enabling feature
> like "scope", right?
>

Depends on how you look at it.

Those type of annotations are used in meta-circular implementations of Java. In those cases Java is a low level systems programming language.

Case in point,

Jikes, http://jikesrvm.org/

Annotations examples, http://jikesrvm.sourceforge.net/apidocs/latest/org/vmmagic/pragma/package-summary.html

Oracle's Hotspot replacement Graal and Substrate VM AOT compiler,
http://lafo.ssw.uni-linz.ac.at/papers/2014_CGO_OneVMToRuleThemAll.pdf

--
PAulo

On 9/28/14, 3:40 AM, Dmitry Olshansky wrote:
> 28-Sep-2014 00:16, Andrei Alexandrescu пишет:
> The key point is that the change is small, that's why (maybe) it's hard
> to grasp.

I think I get it now. It has a huge overlap with stuff that we already have. That doesn't bode very well!

> The whole thing is a bit of lowering and a "hint" to the
> compiler. It reuses the same mechanism that structs already have.
>
> Okay a few examples of lowering to get things going:
>
> http://dpaste.dzfl.pl/3722d9d70937
>
> (note I think there could be better lowerings and simpler set of
> primitives to bootstrap ARC-ed type)
>
> Now why would we need this trivial lowering?
>
> Typical postblit can be anything unless the compiler has full source
> code, dtor can be anything as well.
>
> With opInc/opDec compiler generates postblit/dtor on his own, in doing
> so it decorates user-defined dtor that actually clears resources.
>
> What being in control gives to the compiler:
>
> 1. Compiler always has the source of generated parts, so they can be
> inlined (and should be)

That's a given if the body of ctor/postblit/dtor is available. I don't see this as an important distinction.

> 2. Can do typical algebra optimization on opInc/opDec, no matter what's
> inside opInc and opDec (this is a contract between programmer and
> compiler).
>       e.g opInc(10) followed by OpDec(1) is opInc(9)

That typical algebra optimization is already doable post inlining without a language feature. Compilers know integer arithmetic.

> 3. Also opInc and opDec do not alter object in any capacity, nor are
> they affected by any method calls on this object (another contract)

....?

> 4. 1 + 2 = win, as by inlining postblits/dtors we expose opInc/opDecs to
> the optimizer pass which the would fold them using basic algebra
> optimizations.

I don't think this is a feature worth adding.



Andrei

28-Sep-2014 16:42, Andrei Alexandrescu пишет:
> On 9/28/14, 3:40 AM, Dmitry Olshansky wrote:
[snip]
>
> I don't think this is a feature worth adding.
>

Fair enough. Personally I suspect compilers would have to go a long way yet to optimize away ref-counts decently enough without hints.


-- 
Dmitry Olshansky