On Sunday, 11 October 2015 at 14:27:36 UTC, Freddy wrote:
> On Sunday, 11 October 2015 at 06:10:32 UTC, Jonathan M Davis wrote:
>> alias is problematic, because it allows the class reference to escape. opDispatch doesn't have that problem, though there may be other complications that it introduces (I don't know). It does get kind of complicated though when you consider member functions which return the a reference to the object and things like that. So, while it's generally feasible, it's not that hard for it to become unsafe. How much that matters is debatable, but it could make it so that reference counting classes is infeasible in @safe code.
>>
>> - Jonathan M Davis
>
> Can't we make opAssign and this(this) work with classes add @disable to them? Also rec counted classes shouldn't have members of that have their type, they should have to have members of the aliased struct containing their type.

I don't know if opAssign can be overloaded for a class or not. That gets a bit wonky IMHO given that you never assign classes - only references to them. Similarly, class objects never get postblitted. And something as simple as a parent-child relationship tends to cause serious problems restricting references to smart pointers. e.g. take this C++ code

class Parent
{
    friend class Child;
public
    Parent() _child(make_shared<Child>(this)) {}
private:
    shared_ptr<Child> _child;
}

class Child
{
public:
    Child(Parent* parent) : _parent(parent) {}
private:
    Parent* _parent;
}

auto parent = make_shared<Parent>();

In this case in C++, because the ref-counting is not built-in to the type, there is no way for the Child to have access to its parent via a shared_ptr. It has to be done via a normal pointer. D has exactly this same problem. If the ref-counting isn't built-in, then there are cases where you have to let a non-ref-counted reference escape.

The problem can be reduced by making it so that RefCounted (or whatever the wrapper struct is) doesn't allow accidental, direct access to the wrapped class object. But some stuff will simply be impossible without providing direct access.

By having the class reference contain the reference count, the problem can significantly reduced, because then it's possible to create a shared pointer from the class object without ending up with a separate reference count. But you still have the problem of that class object being passed around and not necessarily always being put in a wrapper object like it's supposed to be. So, you still have a safety issue.

The only way for this to be completely safe is for the ref-counting to be built into the language such that there is no way to have a reference being passed around which isn't ref-counted. Now, we can certainly take the approach of saying that we don't care about that being completely safe (in which case, ref-counting classes will be limited if not outright banned in @safe code), but if we want @safe ref-counting, we have no choice.

- Jonathan M Davis

On Sunday, 11 October 2015 at 07:55:08 UTC, Manu wrote:
> Sure. If you don't care, I'm sure it's fine. But I don't feel it's reasonable to say C++ has ref counting. You might as well say "C++ has garbage collection" (which is probably actually more true than this) ;) ... Objective-C supports ref counting.

Reference counting is now part of C++'s standard library. They may not have the kind of reference counting that you want, but they support reference counting as part of the C++ standard.

And certainly for my use cases, having wrapper classes like smart_ptr has worked fantastically. Any performance hit that might be incurred be extraneous increments and decrements is dwarfed by everything else that the program is doing. And I think that it's pretty clear that the same is true for a _lot_ of programs.

I can certainly believe that it's not true for the kind of programs that you write, and if we can better support those environments, great. But most programmers don't program in a world where it normally matters whether a couple of extra math operations are done or whether a function call is virtual or not. It matters in certain areas of the program but not in general. We don't want to be doing unnecessary operations, and efficiency in other areas can have a huge impact on our programs, but I think that it's quite clear that you live in a world that cares way more about every little ounce of efficiency than most programmers do.

So, I think that it's a huge stretch to say that C++ doesn't have ref-counting or that it doesn't work - or even that a similar solution in D wouldn't work. It would work for many people. If we can come up with one that better supports your use cases, then great. As long as the cost elsewhere isn't too high, why not? But that doesn't mean that a solution like smart_ptr isn't a fantastic one for many of us.

- Jonathan M Davis

On Sunday, 11 October 2015 at 20:56:28 UTC, Jonathan M Davis wrote:
> On Sunday, 11 October 2015 at 18:52:44 UTC, deadalnix wrote:
>> The only rebuttal to all of this is "Walter and I are happy with DIP25, and the fact of the matter", while everybody else is wondering what there is to be happy about.
>
> To be fair, you haven't really said much better. You're claiming that it's clear that it's a failure, whereas Andrei is saying that he doesn't see a problem with it. No concrete arguments are being given. The closest is that you think that the fact that we need something like DIP 74 means that DIP 25's advantage of being simple is lost and that that somehow shows that DIP 25 is a failure.
>

That's not true. I explained in this thread why it is too limited, most notably :

1/ It was barely able to provide a non GC managed array type.

More details here: http://forum.dlang.org/thread/mcg8qq$1mbr$1@digitalmars.com

Notably, it is impossible to reclaim the memory eagerly, but only to clean all memory at once when all reference to one of the memory location is gone. This in turn lead to the need to pretty much have a memory allocator in the array. And jump through many hoops.

2/ It is completely unable to handle reference types (hence DIP74).

Note that if you don't have indirection, there is no memory allocation problem to begin with as value type will be either on stack or involve some reference at some point.

But this whole thing is really a shift of the burden of proof. What do DIP25 enabled that is really worth it ?

On Sunday, 11 October 2015 at 21:15:38 UTC, Jonathan M Davis wrote:
> In this case in C++, because the ref-counting is not built-in to the type, there is no way for the Child to have access to its parent via a shared_ptr. It has to be done via a normal pointer. D has exactly this same problem. If the ref-counting isn't built-in, then there are cases where you have to let a non-ref-counted reference escape.

In this case one should use unique_ptr, so the D discussion about ref counting is irrelevant. If you have truely shared_ptr semantics and back pointers, then one shoud use weak_ptr for this since the back pointers don't own the resource. Keep in mind that shared_ptr only denote ownership, not resource usage.

C++ is now moving towards Rust and static analysis, but as a step seperate from compilation. And that makes sense since it is time consuming and doesn't have to be done for codegen.

On Sunday, 11 October 2015 at 22:07:44 UTC, deadalnix wrote:
> That's not true. I explained in this thread why it is too limited, most notably :
>
> 1/ It was barely able to provide a non GC managed array type.
>
> More details here: http://forum.dlang.org/thread/mcg8qq$1mbr$1@digitalmars.com
>
> Notably, it is impossible to reclaim the memory eagerly, but only to clean all memory at once when all reference to one of the memory location is gone. This in turn lead to the need to pretty much have a memory allocator in the array. And jump through many hoops.
>
> 2/ It is completely unable to handle reference types (hence DIP74).
>
> Note that if you don't have indirection, there is no memory allocation problem to begin with as value type will be either on stack or involve some reference at some point.
>
> But this whole thing is really a shift of the burden of proof. What do DIP25 enabled that is really worth it ?

DIP 25 fixes a serious flaw in ref. It makes it possible to guarantee that ref is @safe. Without it, ref is not @safe. As I understand it, that's the whole point of DIP 25, and it has nothing to do with reference counting at all. It doesn't even really have anything to do with the heap beyond the fact that a ref parameter might refer to an object on the heap - the safety problem is with the stack, not the heap.

Walter's proposal in that link does take advantage of the fact that DIP 25 restricts what can be returned by ref, but that's as close as DIP 25 gets to ref-counting, whereas all of the rest of the stuff you're talking about here has to do with ref-counting.

DIP 25 is intended to plug the holes in ref to solve the ref safety problem, whereas DIP 74 is intended to build ref-counting into the language so that it can be done safely. They both relate to @safe code, but as far as I can tell, they're pretty much orthogonal, and arguing against flaws in DIP 74 doesn't really show flaws in DIP 25. When you're arguing about them together rather than separately, it seems like you're seeing some other part of the puzzle that I'm not.

- Jonathan M Davis

On Sunday, 11 October 2015 at 22:12:50 UTC, Ola Fosheim Grøstad wrote:
> On Sunday, 11 October 2015 at 21:15:38 UTC, Jonathan M Davis wrote:
>> In this case in C++, because the ref-counting is not built-in to the type, there is no way for the Child to have access to its parent via a shared_ptr. It has to be done via a normal pointer. D has exactly this same problem. If the ref-counting isn't built-in, then there are cases where you have to let a non-ref-counted reference escape.
>
> In this case one should use unique_ptr, so the D discussion about ref counting is irrelevant. If you have truely shared_ptr semantics and back pointers, then one shoud use weak_ptr for this since the back pointers don't own the resource. Keep in mind that shared_ptr only denote ownership, not resource usage.

I don't want get into arguments about smart_ptr and unique_ptr. It's completely irrelevant to my point.

weak_ptr doesn't work when the child is constructed by the parent in the parent's constructor. The fact that the parent does not have access to the smart pointer type that it's about to be put into means that it cannot give that to the child. It has to give it a normal pointer. And that means that you can't make it so that all accesses to the parent object are via a smart pointer. And in the case of D, that means that it doesn't work to restrict all access to a class object to a wrapper struct like RefCounted, and it becomes quite easy for a reference to the class object to escape the wrapper struct. And that means that it's not @safe, because the wrapper struct might destroy the class object when it's destroyed (because its ref-count reached 0) while a reference to that class object is still floating around somewhere. Obviously, it can work (it works in C++). It just means that it can't be @safe and that the programmer has to worry about ensuring that memory doesn't get used incorrectly. But part of the whole point of DIP 74 is so that we can have @safe ref-counting in D.

- Jonathan M Davis

On Sunday, 11 October 2015 at 20:35:05 UTC, Andrei Alexandrescu wrote:
> Could you please point to the document you have already written?
>

For instance, we had a discussion with Walter and Mark that eventually yielded DIP25. In there, I made the following proposal :

http://pastebin.com/LMkuTbgN

I made several other very detailed proposal. Other did. It's not about me here. Others simply abandoned as far as I can tell. I'm just a stubborn idiot.

> There's a bit of a stalemate here. So we have:
>
> 1. You say that DIP25 is a failure. More so, you demand that is admitted without evidence. What I see is a feature that solves one problem, and solves it well: annotating a function that returns a reference to its argument. The syntactic cost is low, the impact on existing code is small, and the impact on safety is positive. Walter and I think it is the simplest solution of all considered.
>

It is indeed the simplest. However, experiences that have been made and discussed in the forum showed it was often too simple to be really useful. I cited example of this, namely the RCArray thing and the existence of DIP74.

I don't think the simplicity argument holds water in general as long as we don't have the whole thing. DIP25 + DIP74 + ... must be measured against the alternative.

> 2. You refuse to write a DIP under the assumption it will not be taken seriously. Conversely if you do write a DIP there is an expectation it will be approved just because you put work in it. I don't think rewarding work is the right way to go. We need to reward good work. The "work" part (i.e. a DIP) is a prerequisite; you can't demand to implement a complex feature based on posts and discussions.
>

No that is inaccurate. I think I have evidence that it won't be taken seriously. To start with, there are already several DIP on the subject and they are not discussed at ALL. Namely :

http://wiki.dlang.org/DIP35
http://wiki.dlang.org/DIP36
http://wiki.dlang.org/DIP69
http://wiki.dlang.org/DIP71

These do not even register as a blip on the radar. I don't see how adding my to the pile would change anything.

There are not considered because DIP25 is "simpler" and you and Walter "like it". As long as nothing changes here, there is really no point in wasting my time.

> So I'm not sure how we can move forward from here. If you want to discuss DIP74, great, it can be discussed because it exists. My personal opinion on DIP74 is it has holes and corner cases so it seems it doesn't quite hit the spot. One option is to make it work, another is to take a different attack on the problem. But we need the appropriate DIP.
>

Let's start by the beginning: what good design was enabled by DIP25 ? As long as none is presented, we can't consider it a success.

On Sunday, 11 October 2015 at 20:56:28 UTC, Jonathan M Davis wrote:
> To be fair, you haven't really said much better. You're claiming that it's clear that it's a failure, whereas Andrei is saying that he doesn't see a problem with it. No concrete arguments are being given. The closest is that you think that the fact that we need something like DIP 74 means that DIP 25's advantage of being simple is lost and that that somehow shows that DIP 25 is a failure.
>

OK first thing first, that is not how that works. DIP25 has been out for a while, so one should have something to show for it. What good thing came out of DIP25 ? That should be what is discussed here rather than having to argue why it is not good enough.

See my other messages here to see what isn't good with it. It simply isn't allowing enough to pay for itself.

> I don't think that it's at all clear that DIP 25 and DIP 74 are even particularly related (and Manu has stated the same).

They are the same thing. If you can guarantee that some reference does not escape in uncontrollable manner, you can have ref counting as library. The need for DIP74 only arise because DIP25 cannot ensure that the reference does not escape.

> But I have no idea how an ownership model for memory in general would work. I'd probably have a much better idea if I had read up on Rust (which I really haven't), but every discussion I've seen on Rust's model seems to indicate that it gets pretty complicated - complicated enough that while it might be more correct, it risks being unusable for the average programmer. I don't know how true that is, but I do think that if we're going to seriously discuss an ownership model in D, we need something concrete discuss and debate. I know that writing a DIP takes time (even a small one; it surprised me how long it took to write DIP 82 given how simple it is, and I've written DIPs before), but if we don't have something like that to discuss, then we're just going to be going in circles here. If you have a concrete proposal that you think would really benefit D, then please write up a DIP for it. We could all be way better off for it. Maybe it'll be rejected, but without it, we're definitely not going to end up with some kind of ownership model like you want. And if you're right, then we will be worse off. Concrete proposals are required to have concrete discussions and move forward.
>
> - Jonathan M Davis

Yes, you should definitively read on Rust, or read this :
https://github.com/isocpp/CppCoreGuidelines/blob/09aef9bd86d933bc1e1ffe344eb2e73d2de15685/docs/Lifetimes%20I%20and%20II%20-%20v0.9.1.pdf

Which is basically C++ adopting the idea, but no, it doesn't come from Rut, no, I tell you, C++ invented a long time ago, C++ invented everything, stop doubting us.

Things is, Rust has that one mechanism to do everything. As a result it is fairly complex, because it has to handle all common cases. We can do something fairly similar, yet simpler, if we accept to rely on the GC or unsafe constructs for things that do not fit well in the model.

On Sunday, 11 October 2015 at 22:33:44 UTC, Jonathan M Davis wrote:
> I don't want get into arguments about smart_ptr and unique_ptr. It's completely irrelevant to my point.

Ok, but keep in mind that since unique_ptr is rc which max out at 1 and therefore don't need extra support beyond having an owning pointer. So whatever is done for reasoning about rc also has to work for unique. If not thenD is doomed to fail, because nobody wants rc when unique is sufficient.

> weak_ptr doesn't work when the child is constructed by the parent in the parent's constructor. The fact that the parent does not have access to the smart pointer type that it's about to be put into means that it cannot give that to the child. It has to give it a normal pointer. And that means that you can't make it so that all accesses to the parent object are via a smart pointer. And in the case of D, that means that it doesn't work to restrict all access to a class object to a wrapper struct like RefCounted, and it becomes quite easy for a reference to the class object to escape the wrapper struct.

Well, the wrapper approach is no good (is it part of dip74?) since it messes up alignment etc, so the refcount interface should be part of the parent object. The child can be given the Refcount interface from dip74, it won't use it for a weak reference until it dereference (borrow). A weak reference just delays destruction until the borrowing is completed.

So it works(?) if you dont allow dereferencing of the weak pointer in the constructor of the child.

> get used incorrectly. But part of the whole point of DIP 74 is so that we can have @safe ref-counting in D.

Well, @safe in D is broken and should just be dropped in favour of something sane like real pointer analysis instead of piling up bad designs like dip25. You only have to run it for release... Who cares if it is slow?

On 12 Oct 2015 7:31 am, "Jonathan M Davis via Digitalmars-d" < digitalmars-d@puremagic.com> wrote:
>
> On Sunday, 11 October 2015 at 07:55:08 UTC, Manu wrote:
>>
>> Sure. If you don't care, I'm sure it's fine. But I don't feel it's
reasonable to say C++ has ref counting. You might as well say "C++ has garbage collection" (which is probably actually more true than this) ;) ... Objective-C supports ref counting.
>
>
> Reference counting is now part of C++'s standard library. They may not
have the kind of reference counting that you want, but they support reference counting as part of the C++ standard.
>
> And certainly for my use cases, having wrapper classes like smart_ptr has
worked fantastically. Any performance hit that might be incurred be extraneous increments and decrements is dwarfed by everything else that the program is doing. And I think that it's pretty clear that the same is true for a _lot_ of programs.
>
> I can certainly believe that it's not true for the kind of programs that
you write, and if we can better support those environments, great. But most programmers don't program in a world where it normally matters whether a couple of extra math operations are done or whether a function call is virtual or not. It matters in certain areas of the program but not in general. We don't want to be doing unnecessary operations, and efficiency in other areas can have a huge impact on our programs, but I think that it's quite clear that you live in a world that cares way more about every little ounce of efficiency than most programmers do.
>
> So, I think that it's a huge stretch to say that C++ doesn't have
ref-counting or that it doesn't work - or even that a similar solution in D wouldn't work. It would work for many people. If we can come up with one that better supports your use cases, then great. As long as the cost elsewhere isn't too high, why not? But that doesn't mean that a solution like smart_ptr isn't a fantastic one for many of us.
>
> - Jonathan M Davis

Incidentally, I tried to use shared_ptr initially, but it took about 20 minutes before I realised I had to pass an rc pointer from a method... Since rc is a wrapper, like you said above, you lose it as soon as you're within a method. I then had to write an invasive rc implementation and hard create a new rc instance from 'this', all over the place. Seriously, I don't understand how anyone can say shared_ptr is a success. It's a massive kludge, and it alone has me firmly convinced that rc is weak, inconvenient, and highly complicated without language support.