On 11 October 2015 at 17:16, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Sunday, 11 October 2015 at 06:55:50 UTC, Manu wrote:
>>
>> On 11 October 2015 at 14:35, Jonathan M Davis via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>
>>> On Sunday, 11 October 2015 at 04:16:11 UTC, deadalnix wrote:
>>>>
>>>>
>>>> If we go these DIP road, there is no coming back and this will get in the way of a principled approach.
>>>
>>>
>>>
>>> Then come up with an alternative DIP which shows a better way to solve this. As it stands, it looks likely that we'll end up with some form of DIP 74, and if you have a better proposal, then now is probably the time to do it.
>>>
>>> Personally, I obviously haven't been following this closely enough,
>>> because I don't understand why something like RefCounted can't be made to do
>>> what we need with regards to reference counting and classes. It does get a
>>> bit nasty when inheritance and whatnot get involved, but C++ was able to
>>> solve that easily enough, and we should be able to do the same.
>>
>>
>> C++ didn't solve anything(?). C++ doesn't support ref-counting at all!
>> shared_ptr is not a part of the language, or a proper ref counting
>> mechanism. It's just a hack; it's awkward, and really inefficient (the
>> compiler can't optimise it).
>> ARC requires language knowledge. I don't know what language primitives
>> can possibly allow the compiler to do proper ref fiddling optimisation
>> with a lib?
>
>
> Ref-counting with shared_ptr works just fine. It just doesn't optimize out any of the incrementing or decrementing.

I wouldn't call that 'working', I'd call that 'grossly over-working' at best.
I'd personally diagnose signals firing at improper times to be a bug,
and that's effectively what we have ;)

I think it's absolutely worthwhile that a language should know that refcounting is a thing, and what the proper handling is. I don't know any way to model it accurately without language support.

> And while having those optimized
> out would be nice, I don't think that in and of itself that makes it worth
> having ref-counting in the language rather than in the library.

For real? If you're handling ref-counted objects, ref-fiddling often
completely overwhelms your code at the interface level, and god forbid
you have some sort of adaptation layers, shims, small temporaries,
call-throughs, or iterations.
Problem is amplified big time if your ref-counting functions are
foreign library functions, which is pretty common in C api's. These
functions need to be wrapped with a primitive that properly knows when
the functions should be called.

I'm seeing objects with RC==8+ or so across one event handling call-tree.

> Maybe C++ ref-counting hasn't work well for your use cases, but I've used it for years, and it's worked great for what I've needed.

Sure. If you don't care, I'm sure it's fine. But I don't feel it's reasonable to say C++ has ref counting. You might as well say "C++ has garbage collection" (which is probably actually more true than this) ;) ... Objective-C supports ref counting.

On 11 October 2015 at 17:30, deadalnix via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Sunday, 11 October 2015 at 06:48:54 UTC, Manu wrote:
>>
>> I don't really see that DIP25 and DIP74 are particularly closely related.
>
>
> Ho they are. It is all about ownership.
>
>> DIP25 is a lame cop-out with respect to scope. I was a major backer of a proper scope implementation, and given that, I don't/didn't support DIP25 at all.
>>
>
> DIP25 being a sorry excuse for ownership is the very reason DIP74 is needed. Rust can do reference counting just fine all in library without need for core language support.
>
>> DIP74 on the other hand introduces 2 magic functions that the compiler calls as the appropriate moments to implement ref counting. The implementation is somewhat detail, theoretically changeable in the future, but I think language support for ref-counting primitives is critical for ref counting to be efficient and simple. Look at C++, and we see a disaster. C++ basically implemented rval-references to improve (not solve) the RC problem... we don't have rval-ref's in D, so we're screwed from the library angle.
>>
>
> C++ ref counting is not slow for the reasons you think it is. ARC is overly complex and also not as fast as you think it is.
>
> First, they are slow because in C++ (or ObjC), you don't know what is shared and what isn't. As a result, you got to go the pessimistic road and use thread safe ref counting. This can be a solved problem in D purely using the type qualifier.
>
> Second, exceptions. Their unwinding makes pair of inc/dec hard to recognize, plus now cleanup is need at pretty much every frames, meaning incredibly slow exceptions + various control flow optimization that aren't possible anymore (tail call being one of them).
>
> One that one, nobody has a solution. Not ARC (that's why swift does not have exception), not C++ and not us. IMO the obvious solution here is to allow ourselves to leak on unwind and rely on the GC. Yes, sometime the GC is best. That being said, it is up to the application to decide if leaking is acceptable or not, and so this can't be baked in the language (this also means that the user must be in charge the of deallocation policy, which puts a major dent into DIP74).
>
> The 3rd problem is safety. This is where ownership shines.
>
> The 4th problem is elision of refcount operations when not needed. Some thing, ownership can solve this. No need to do refcount operations when not necessary.
>
> The compiler is perfectly able to optimize pairs of inc/dec granted it has the rights infos. And a proper ownership system provide the right infos.
>
>> Yup, I was firmly behind scope (ie, borrowing), but that was
>> destroyed, and DIP25 kinda cements the death of that :(
>>
>
> It is implemented with a flag. That allowed us to try it and get real life experience. That is a good thing. We aren't committed to it at this stage. And now we have actual experimentation that shows that it is too underpowered to be really useful. The smart move here is to admit defeat and reconsider.

Nothing here explains to me how it is that the compiler can do without
RC primitives that the compiler knows it is allowed to
optimise/schedule as it likes?
How does it look in rust that it is able to optimise calls to inc/dec
primitives?

You also make the points that I expect to bank on in D; we are thread-local by default (big advantage!), nothrow is pervasive (big advantage!), but the language still doesn't know the primitives that it can use to optimise.

The ownership proposal solving the problem isn't entirely correct.
Sure, borrowed pointers imply RC elision, and it's particularly useful
for writing API's that are allocation/ownership agnostic, but it
doesn't solve the problem as you suggest. Once a pointer is borrowed,
it can't be assigned to another RC pointer anymore. If you must use
borrowing to implement RC elision across calls, then you lose the
ability to give someone else a reference down the call-tree.
Passing an RC object proper to a function also elides RC fiddling, but
the compiler can't distinguish between RC fiddling and copying logic
unless the RC primitives are explicit. I find that this is a very
common case.

On 10/11/15 10:16 AM, Jonathan M Davis wrote:
> On Sunday, 11 October 2015 at 06:55:50 UTC, Manu wrote:
>> On 11 October 2015 at 14:35, Jonathan M Davis via Digitalmars-d
>> <digitalmars-d@puremagic.com> wrote:
>>> On Sunday, 11 October 2015 at 04:16:11 UTC, deadalnix wrote:
>>>>
>>>> If we go these DIP road, there is no coming back and this will get
>>>> in the way of a principled approach.
>>>
>>>
>>> Then come up with an alternative DIP which shows a better way to
>>> solve this. As it stands, it looks likely that we'll end up with some
>>> form of DIP 74, and if you have a better proposal, then now is
>>> probably the time to do it.
>>>
>>> Personally, I obviously haven't been following this closely enough,
>>> because I don't understand why something like RefCounted can't be
>>> made to do what we need with regards to reference counting and
>>> classes. It does get a bit nasty when inheritance and whatnot get
>>> involved, but C++ was able to solve that easily enough, and we should
>>> be able to do the same.
>>
>> C++ didn't solve anything(?). C++ doesn't support ref-counting at all!
>> shared_ptr is not a part of the language, or a proper ref counting
>> mechanism. It's just a hack; it's awkward, and really inefficient (the
>> compiler can't optimise it).
>> ARC requires language knowledge. I don't know what language primitives
>> can possibly allow the compiler to do proper ref fiddling optimisation
>> with a lib?
>
> Ref-counting with shared_ptr works just fine. It just doesn't optimize
> out any of the incrementing or decrementing. And while having those
> optimized out would be nice, I don't think that in and of itself that
> makes it worth having ref-counting in the language rather than in the
> library.
>
> Maybe C++ ref-counting hasn't work well for your use cases, but I've
> used it for years, and it's worked great for what I've needed.

Yah, it's a success story and a good baseline. Two things that are suboptimal are the necessity of two allocations unless make_shared is used, and compulsive interlocked inc/dec. We ought to address both issues, but by and large we're looking at a successful experience that we want to have D users benefit from as well. -- Andrei

On 10/11/15 10:53 AM, deadalnix wrote:
>
> I'm saying that DIP 25 was implemented and showed that it was too
> limited. The various experiment with it showed that is wasn't enough
> (ref counted objects) or required to jump though a lot of hoops for
> somewhat disappointing result (ref counted arrays).
>
> DIP74 is a direct result of DIP25 limitations. I think we should start
> by acknowledging that the DIP25 experiment was not a success before
> taking it for granted and proceed.
>
> I do think a buffed up version of DIP25 would make DIP74 obsolete (or
> implementable as a library).

Walter and I are happy with DIP25, and the fact of the matter is we weren't surprised more complementary work is needed. So no, I won't acknowledge what I don't believe.

I'd say the one way to get things looked at seriously is to create a DIP. That's no guarantee it will be accepted but there is a guarantee that our chat at DConf is not sufficient even as a basis for further study.


Andrei

On Sunday, 11 October 2015 at 06:10:32 UTC, Jonathan M Davis wrote:
> alias is problematic, because it allows the class reference to escape. opDispatch doesn't have that problem, though there may be other complications that it introduces (I don't know). It does get kind of complicated though when you consider member functions which return the a reference to the object and things like that. So, while it's generally feasible, it's not that hard for it to become unsafe. How much that matters is debatable, but it could make it so that reference counting classes is infeasible in @safe code.
>
> - Jonathan M Davis

Can't we make opAssign and this(this) work with classes add @disable to them? Also rec counted classes shouldn't have members of that have their type, they should have to have members of the aliased struct containing their type.

On Sunday, 11 October 2015 at 13:51:18 UTC, Andrei Alexandrescu wrote:
> Walter and I are happy with DIP25, and the fact of the matter is we weren't surprised more complementary work is needed. So no, I won't acknowledge what I don't believe.
>

That is an empty statement. What is there to be happy about ?

Also the complementary argument pretty much destroy the best argument you and Walter made for DIP25 : it is simple. I mean, one need to look at the big picture. DIP25 + complementary addition is not simple anymore.

> I'd say the one way to get things looked at seriously is to create a DIP. That's no guarantee it will be accepted but there is a guarantee that our chat at DConf is not sufficient even as a basis for further study.
>

Yeah there are IRL discussion, there are many posts in the forum, there are by mail discussions at DIP25 creation time, there are at least one DIP already.

The only rebuttal to all of this is "Walter and I are happy with DIP25, and the fact of the matter", while everybody else is wondering what there is to be happy about.

On Sunday, 11 October 2015 at 18:52:44 UTC, deadalnix wrote:
> On Sunday, 11 October 2015 at 13:51:18 UTC, Andrei Alexandrescu wrote:
>> Walter and I are happy with DIP25, and the fact of the matter is we weren't surprised more complementary work is needed. So no, I won't acknowledge what I don't believe.
>>
>
> That is an empty statement. What is there to be happy about ?
>
> Also the complementary argument pretty much destroy the best argument you and Walter made for DIP25 : it is simple. I mean, one need to look at the big picture. DIP25 + complementary addition is not simple anymore.
>
>> I'd say the one way to get things looked at seriously is to create a DIP. That's no guarantee it will be accepted but there is a guarantee that our chat at DConf is not sufficient even as a basis for further study.
>>
>
> Yeah there are IRL discussion, there are many posts in the forum, there are by mail discussions at DIP25 creation time, there are at least one DIP already.
>
> The only rebuttal to all of this is "Walter and I are happy with DIP25, and the fact of the matter", while everybody else is wondering what there is to be happy about.

Also, I'm sorry but there is no me writing once again a document about what alternative are possible. Spending hours to write documents so that one is answered something along the line of "we are happy with the other thing, but we can't give any reason why" is something I've engaged in several time in already, and has no desire to indulge into this if I have reason to think the same will happen. Your answer here are telling me one thing: it won't be taken seriously.

Manu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Nothing here explains to me how it is that the compiler can do without
> RC primitives that the compiler knows it is allowed to
> optimise/schedule as it likes?
> How does it look in rust that it is able to optimise calls to inc/dec
> primitives?
> 
> You also make the points that I expect to bank on in D; we are thread-local by default (big advantage!), nothrow is pervasive (big advantage!), but the language still doesn't know the primitives that it can use to optimise.
> 
> The ownership proposal solving the problem isn't entirely correct.
> Sure, borrowed pointers imply RC elision, and it's particularly useful
> for writing API's that are allocation/ownership agnostic, but it
> doesn't solve the problem as you suggest. Once a pointer is borrowed,
> it can't be assigned to another RC pointer anymore. If you must use
> borrowing to implement RC elision across calls, then you lose the
> ability to give someone else a reference down the call-tree.
> Passing an RC object proper to a function also elides RC fiddling, but
> the compiler can't distinguish between RC fiddling and copying logic
> unless the RC primitives are explicit. I find that this is a very
> common case.

The key is, that RC in Rust is a move-by-default type. That means if you
are passing RCs around, you are actually passing ownership.
Only if you explicitly call 'clone', ownership is split up.
That means in the default case, no inc/dec is necessary.

Even more important, RC isn't used very often.
True shared ownership is actually quite rare. References (borrowed
pointers) and 'Box'es (unique pointers) are much more common.
shared_ptr is much overused in C++ IMO.

Tobi

On 10/11/15 9:57 PM, deadalnix wrote:
> On Sunday, 11 October 2015 at 18:52:44 UTC, deadalnix wrote:
>> On Sunday, 11 October 2015 at 13:51:18 UTC, Andrei Alexandrescu wrote:
>>> Walter and I are happy with DIP25, and the fact of the matter is we
>>> weren't surprised more complementary work is needed. So no, I won't
>>> acknowledge what I don't believe.
>>>
>>
>> That is an empty statement. What is there to be happy about ?
>>
>> Also the complementary argument pretty much destroy the best argument
>> you and Walter made for DIP25 : it is simple. I mean, one need to look
>> at the big picture. DIP25 + complementary addition is not simple anymore.
>>
>>> I'd say the one way to get things looked at seriously is to create a
>>> DIP. That's no guarantee it will be accepted but there is a guarantee
>>> that our chat at DConf is not sufficient even as a basis for further
>>> study.
>>>
>>
>> Yeah there are IRL discussion, there are many posts in the forum,
>> there are by mail discussions at DIP25 creation time, there are at
>> least one DIP already.
>>
>> The only rebuttal to all of this is "Walter and I are happy with
>> DIP25, and the fact of the matter", while everybody else is wondering
>> what there is to be happy about.
>
> Also, I'm sorry but there is no me writing once again a document about
> what alternative are possible.

Could you please point to the document you have already written?

> Spending hours to write documents so that
> one is answered something along the line of "we are happy with the other
> thing, but we can't give any reason why" is something I've engaged in
> several time in already, and has no desire to indulge into this if I
> have reason to think the same will happen. Your answer here are telling
> me one thing: it won't be taken seriously.

There's a bit of a stalemate here. So we have:

1. You say that DIP25 is a failure. More so, you demand that is admitted without evidence. What I see is a feature that solves one problem, and solves it well: annotating a function that returns a reference to its argument. The syntactic cost is low, the impact on existing code is small, and the impact on safety is positive. Walter and I think it is the simplest solution of all considered.

2. You refuse to write a DIP under the assumption it will not be taken seriously. Conversely if you do write a DIP there is an expectation it will be approved just because you put work in it. I don't think rewarding work is the right way to go. We need to reward good work. The "work" part (i.e. a DIP) is a prerequisite; you can't demand to implement a complex feature based on posts and discussions.

So I'm not sure how we can move forward from here. If you want to discuss DIP74, great, it can be discussed because it exists. My personal opinion on DIP74 is it has holes and corner cases so it seems it doesn't quite hit the spot. One option is to make it work, another is to take a different attack on the problem. But we need the appropriate DIP.


Andrei

On Sunday, 11 October 2015 at 18:52:44 UTC, deadalnix wrote:
> The only rebuttal to all of this is "Walter and I are happy with DIP25, and the fact of the matter", while everybody else is wondering what there is to be happy about.

To be fair, you haven't really said much better. You're claiming that it's clear that it's a failure, whereas Andrei is saying that he doesn't see a problem with it. No concrete arguments are being given. The closest is that you think that the fact that we need something like DIP 74 means that DIP 25's advantage of being simple is lost and that that somehow shows that DIP 25 is a failure.

I don't think that it's at all clear that DIP 25 and DIP 74 are even particularly related (and Manu has stated the same). One deals with escaping ref parameters (which doesn't involved reference counting), whereas the other has to do with reference counting. It may very well be that there's a way to cover both issues with some sort of ownership model, but that's not clear from simply looking at the DIPs or the current discussion. All we really have is your assertion that there's a better way.

I'm completely open to there being a better solution than DIP 25 or DIP 74, but I have yet to see one explained. The only alternatives to DIP 74 that I've really seen are suggestions that we should be able to do it with library types (like C++ does) and not require built-in ref-counting. If there have been any in-depth explanations or discussions on how an ownership model could solve this, I haven't seen them (though there may very well have been one in a previous thread that I missed), so I really don't know what that would even entail.

The only place that it's clear to me that an ownership model of some kind would be nice would be one related to threads so that it we could safely pass mutable data between threads, but honestly, I'm okay with it being unsafe to pass mutable data across threads. D's threading model makes it pretty easy to get that right, and most programs don't need it anyway.

But I have no idea how an ownership model for memory in general would work. I'd probably have a much better idea if I had read up on Rust (which I really haven't), but every discussion I've seen on Rust's model seems to indicate that it gets pretty complicated - complicated enough that while it might be more correct, it risks being unusable for the average programmer. I don't know how true that is, but I do think that if we're going to seriously discuss an ownership model in D, we need something concrete discuss and debate. I know that writing a DIP takes time (even a small one; it surprised me how long it took to write DIP 82 given how simple it is, and I've written DIPs before), but if we don't have something like that to discuss, then we're just going to be going in circles here. If you have a concrete proposal that you think would really benefit D, then please write up a DIP for it. We could all be way better off for it. Maybe it'll be rejected, but without it, we're definitely not going to end up with some kind of ownership model like you want. And if you're right, then we will be worse off. Concrete proposals are required to have concrete discussions and move forward.

- Jonathan M Davis