On Sunday, 11 October 2015 at 23:08:58 UTC, Manu wrote:
> Incidentally, I tried to use shared_ptr initially, but it took about 20 minutes before I realised I had to pass an rc pointer from a method... Since rc is a wrapper, like you said above, you lose it as soon as you're within a method.

And here you have it. You asked earlier how DIP25 and DIP74 connect, and this is where. If you want to ensure reference counting safety, you must make sure that reference do not escape in uncontrolled manner.

> I then had to write an invasive rc implementation and hard create a new rc instance from 'this', all over the place. Seriously, I don't understand how anyone can say shared_ptr is a success. It's a massive kludge, and it alone has me firmly convinced that rc is weak, inconvenient, and highly complicated without language support.

By C++'s standards, this is not that messy.

On 10/11/2015 10:35 PM, Andrei Alexandrescu wrote:
> ...
>
> 1. You say that DIP25 is a failure. More so, you demand that is admitted
> without evidence.

FWIW, DIP25 is insufficiently formal and/or incorrect.
I have been able to find those holes in the implementation pretty quickly (I'll also put them to bugzilla):


---

enum N=100;
ref int[N] foo(return ref int[N] s)@safe{ return s; }
int[N]* bar(return ref int[N] s)@safe{ return &foo(s); }
ref int[N] baz()@safe{ int[N] s; return *bar(s); }

// for illustration:
void bang(ref int[N] x)@safe{ x[]=0x25BAD; }
void main()@safe{ bang(baz()); }

---

enum N=100;
ref int[N] fun(ref int[N] x)@safe{
    ref int[N] bar(){ return x; }
    return bar();
}
ref int[N] hun()@safe{
    int[N] k;
    return fun(k);
}

// for illustration:
void bang(ref int[N] x)@safe{ x[]=0x25BAD; }
void main()@safe{ bang(hun); }

---

ref int foo()@safe{
    struct S{
        int x;
        ref int bar()return{ return x; }
    }
    return S().bar();
}

// fun fact: paste it twice into a D file to get an ICE :-)

---

ref int foo()@safe{
    int x;
    struct S{
        ref int bar(){ return x; }
    }
    return S().bar();
}
// (the DIP quite explicitly allows this,
// as x lives longer than 'this'.)

---

Basically every technique I have tried to use for escaping references to locals and non-return parameters directly after reading the DIP has been fruitful. This seems like a pretty obvious failure to achieve "with the proposed rules, it is impossible in safe code to have ref refer to a destroyed object" to me.
Had the DIP explicitly mentioned all necessary rules in more detail, there would probably be fewer cases like this. Its true complexity would then also be more apparent. (Be it low or high, but at least it would be a leveled playing ground.)

> What I see is a feature that solves one problem, and
> solves it well: annotating a function that returns a reference to its
> argument.

When you say "well", what do you mean?

My personal definition of "well" for such cases is that all the contextual information needed to type check an expression can be encoded into the type signature of a function. (Unfortunately, D violates this rule quite often.) I don't think there is a syntax-free way of fixing the holes which satisfies this, as the return attribute inherently throws away e.g. the information about which nested scope a reference returned after being accessed through a context pointer refers to.


> The syntactic cost is low,

I have noticed you tend to attach a large weight to "syntactic cost". How to evaluate syntactic cost of a feature? I think concepts that exist but have no syntax have a larger cost.

> and the impact on safety is positive.
> Walter and I think it is the simplest solution of all considered.

I'm not entirely sure. It has the lowest amount of syntax though.

On Sunday, 11 October 2015 at 23:07:18 UTC, Ola Fosheim Gr wrote:
> On Sunday, 11 October 2015 at 22:33:44 UTC, Jonathan M Davis wrote:
> Well, the wrapper approach is no good (is it part of dip74?) since it messes up alignment etc, so the refcount interface should be part of the parent object. The child can be given the Refcount interface from dip74, it won't use it for a weak reference until it dereference (borrow). A weak reference just delays destruction until the borrowing is completed.

DIP 74 proposes built-in reference counting. There is no wrapper type, and the compiler is able to elide increments and decrements of the ref count if it determines that they're unnecessary. If a type is ref-counted with DIP 74, it's always ref-counted, and it doesn't have anything for weak references (presumably leaving that up to the GC to take care of).

>> get used incorrectly. But part of the whole point of DIP 74 is so that we can have @safe ref-counting in D.
>
> Well, @safe in D is broken and should just be dropped in favour of something sane like real pointer analysis instead of piling up bad designs like dip25. You only have to run it for release... Who cares if it is slow?

@safe isn't going anywhere. And it mostly works just fine. It's primary flaw is that it's been done via blacklisting operations rather than whitelisting them, but that doesn't stop it from working. It just makes the implementation more error-prone. The only thing preventing the use of @safe in most cases is functions that haven't been updated to use it yet (in which case @trusted can be used). Walter and Andrei are completely behind @safe and are going to be in favor of fixing any holes in it that are found, not getting rid of it. And I don't see any reason why that can't work or that it's a bad idea.

- Jonathan M Davis

On Sunday, 11 October 2015 at 23:08:58 UTC, Manu wrote:
> Incidentally, I tried to use shared_ptr initially, but it took about 20 minutes before I realised I had to pass an rc pointer from a method... Since rc is a wrapper, like you said above, you lose it as soon as you're within a method. I then had to write an invasive rc implementation and hard create a new rc instance from 'this', all over the place. Seriously, I don't understand how anyone can say shared_ptr is a success. It's a massive kludge, and it alone has me firmly convinced that rc is weak, inconvenient, and highly complicated without language support.

If you use shared_ptr, what typically happens is that you use shared_ptr everywhere and naked pointers pretty much disappear from your code. The main problem is when a type needs to return a pointer to itself, but in my own code, I've found that need to be rare (if anything, I'm more likely to pass an object out of its own member function by &, not by pointer). The primary counter-example is when a class that it owns needs to have access to it, but then in most cases, that class doesn't need to pass that pointer on to anyone else, and since it lives for as long as its parent does, it doesn't have to worry about ref-counting. I suppose that using & would work just as well in those cases, though I don't tend to use & much outside of passing into a function and returning out of it (generally when having a copy is okay but I want it to be possible to avoid it), because it's too easy to make inadvertent copies.

Having ref-counting built into the type does solve some of those problems, but it can cause others. I doubt that there was ever any chance that the C++ standards committee would have ever gone with built-in ref-counting though given how invasive that is in the class hierarchy and how it would not work very well with existing code. In our case, with classes, built-in ref-counting like DIP 74 proposes is probably better.

- Jonathan M Davis

On Saturday, 10 October 2015 at 23:25:49 UTC, Manu wrote:
> So I have another upcoming opportunity to introduce D in my workplace, this time as a front-end/plugin language to our C++ infrastructure, which is promising since I already have considerable experience in this area (my work at Remedy with Quantum Break), and there is a lot of recent work to interact better with C++, which we will stress-test extensively.
>
> You only get so many shots at this; but this is a particularly promising opportunity, since the C++ code is a nightmare, and the contrast against D will allow a lot of coders to see the advantage.
>
> There is however one critical missing feature, DIP74... where is it at currently? How is it going? Is it likely to be accepted in the near-term? Some sort of approximate timeline?
>
> I think it would be a mistake for me to introduce this without DIP74, since we will rely on it VERY heavily, and the machinery to work-around it will start to look just as heavy-weight as the C++ code I'm trying to deprecate... but then waiting on it starts to look like missing the window of opportunity.
>
> Thoughts?

I recently brought this up here:
http://forum.dlang.org/thread/aetzsanbypaimljrguzb@forum.dlang.org

We discussed pros and cons, etc..

I restated my main concern here, and Walter seemed to agree:
http://forum.dlang.org/post/mv29l2$1rln$1@digitalmars.com

So it seems there is hope for DIP74(or something similar), but it may take time.

I asked Andrei when we may have another "D Vision" or similar document, and his response was here:
http://forum.dlang.org/post/mv7is9$m41$1@digitalmars.com

     Bit

Am Sun, 11 Oct 2015 07:32:26 +0000
schrieb deadalnix <deadalnix@gmail.com>:

> In C++, you need to assume things are shared, and, as such, use thread safe inc/dec . That means compiler won't be able to optimize them. D can do better as sharedness is part of the type system.

With the little nag that `shared` itself is not fleshed out.

-- 
Marco

On Monday, 12 October 2015 at 03:59:04 UTC, Marco Leise wrote:
> Am Sun, 11 Oct 2015 07:32:26 +0000
> schrieb deadalnix <deadalnix@gmail.com>:
>
>> In C++, you need to assume things are shared, and, as such, use thread safe inc/dec . That means compiler won't be able to optimize them. D can do better as sharedness is part of the type system.
>
> With the little nag that `shared` itself is not fleshed out.

Well, it really should be better fleshed out, but the reality of the matter is that it actually works pretty well as it is. The problem is primarily that it's a pain to use - and to a certain extent, that's actually a good thing, but it does make it harder to use correctly. Better support for detecting when shared can be safely cast away would be nice. e.g. being able to do something like

synchronized(mutex_for_foo)
{
    // foo is now implicitly treated as thread-local
}
// foo is now treated as shared again

would be nice. However, figuring out how to do that safely is very tricky.

Ultimately though, I think that the problem with shared generally comes down to folks not liking the fact that you pretty much can't do anything with a shared variable until you cast away shared, but the fact that it won't let you do much is actually protecting you.

We do need to take another look at shared and see what we can do to improve it, but I'm sure that it's ultimately going to be much different from how it is now.

- Jonathan M Davis

On Sunday, 11 October 2015 at 23:46:42 UTC, Jonathan M Davis wrote:
> @safe isn't going anywhere. And it mostly works just fine. It's primary flaw is that it's been done via blacklisting operations rather than whitelisting them, but that doesn't stop it from working. It just makes the implementation more error-prone.

Has nothing with whitelisting or blacklisting, that's basically the same thing for a finite set of features.

The real problem is that it cannot be assumed to work until proven correct, formally. So it does not give you anything more than a weak sanitizer would have done until such proofs have been verified. The fact that it is difficult to be convince oneself that you don't have holes reinforces this viewpoint.

If reasoning is difficult, you need proofs.

> Walter and Andrei are completely behind @safe and are going to be in favor of fixing any holes in it that are found, not getting rid of it. And I don't see any reason why that can't work or that it's a bad idea.

This has been discussed before. @trusted regions can be correct in isolation, but it will make assumptions about what goes into @safe regions, so when the @safe region changes @trusted regions cannot assumed to be correct anymore. Therefore you need a prover to prove the @trusted region to remain safe whenever you change the surrounding @safe region. That means @trusted has to be truly exceptional and rare and impose verfiied restrictions on the assumptions it makes, which is not likely to happen.

This comes in conflict with the very nature of system level programming.

On Sunday, 11 October 2015 at 23:08:58 UTC, Manu wrote:
> Incidentally, I tried to use shared_ptr initially, but it took about 20 minutes before I realised I had to pass an rc pointer from a method... Since rc is a wrapper, like you said above, you lose it as soon as you're within a method. I then had to write an invasive rc implementation and hard create a new rc instance from 'this', all over the place. Seriously, I don't understand how anyone can say shared_ptr is a success. It's a massive kludge, and it alone has me firmly convinced that rc is weak, inconvenient, and highly complicated without language support.

Shared_ptr and unique_ptr represent ownership, not references. So you should not pass them around unless you transfer ownership.

When the owner allows someone to borrow a reference, that borrowing point has to ensure that the reference does not outlive the lifespan of the object. This is not so hard to get right.

shared_ptr has several advantages:

1. It works with existing classes (from foreign libraries) without any notion of RC.

2. It allows weak references and cycles without keeping the whole object allocated.

3. It supports concurrency.

5. The reference counting mechanism can be done in a way that fits with polymorphism since it does not make any assumptions about the structure of object itself.

6. The object itself can remain fully immutable.

Disadvantages:

1. double indirection

2. two allocations

On 10/12/15 1:44 AM, deadalnix wrote:
> On Sunday, 11 October 2015 at 20:35:05 UTC, Andrei Alexandrescu wrote:
>> Could you please point to the document you have already written?
>>
>
> For instance, we had a discussion with Walter and Mark that eventually
> yielded DIP25. In there, I made the following proposal :
>
> http://pastebin.com/LMkuTbgN

This is an unstructured text. Could you please use it as a basis for a formal proposal?

> I made several other very detailed proposal.

Where are they?

> Other did. It's not about
> me here. Others simply abandoned as far as I can tell. I'm just a
> stubborn idiot.

At this point it would be great to just keep it calm and reduce inflammation. It won't achieve anything.

>> There's a bit of a stalemate here. So we have:
>>
>> 1. You say that DIP25 is a failure. More so, you demand that is
>> admitted without evidence. What I see is a feature that solves one
>> problem, and solves it well: annotating a function that returns a
>> reference to its argument. The syntactic cost is low, the impact on
>> existing code is small, and the impact on safety is positive. Walter
>> and I think it is the simplest solution of all considered.
>>
>
> It is indeed the simplest. However, experiences that have been made and
> discussed in the forum showed it was often too simple to be really
> useful. I cited example of this, namely the RCArray thing and the
> existence of DIP74.
>
> I don't think the simplicity argument holds water in general as long as
> we don't have the whole thing. DIP25 + DIP74 + ... must be measured
> against the alternative.

What is the alternative? Some handwaving asking to do ownership a la Rust cannot be analyzed.

>> 2. You refuse to write a DIP under the assumption it will not be taken
>> seriously. Conversely if you do write a DIP there is an expectation it
>> will be approved just because you put work in it. I don't think
>> rewarding work is the right way to go. We need to reward good work.
>> The "work" part (i.e. a DIP) is a prerequisite; you can't demand to
>> implement a complex feature based on posts and discussions.
>>
>
> No that is inaccurate. I think I have evidence that it won't be taken
> seriously. To start with, there are already several DIP on the subject
> and they are not discussed at ALL. Namely :
>
> http://wiki.dlang.org/DIP35
> http://wiki.dlang.org/DIP36
> http://wiki.dlang.org/DIP69
> http://wiki.dlang.org/DIP71
>
> These do not even register as a blip on the radar. I don't see how
> adding my to the pile would change anything.

Creating a DIP is no guarantee it will be approved, or discussed immediately. These in particular - I've been over most. I think DIP35 is not good. DIP36 I didn't look at yet, but was aware of it and will definitely do. DIP69 is obviously known to me because my name is on it. DIP71 is very sketchy and is not in reviewable form.

> There are not considered because DIP25 is "simpler" and you and Walter
> "like it". As long as nothing changes here, there is really no point in
> wasting my time.

That is a fair assessment. Basically I believe DIP25 is good language design, and I have evidence for it. The evidence you showed failed to convince me the design is a hack, and yelling at me is unlikely to help. Please decide as you find fit. At some point it is clear that several language designers will disagree on estimating the quality of something.

>> So I'm not sure how we can move forward from here. If you want to
>> discuss DIP74, great, it can be discussed because it exists. My
>> personal opinion on DIP74 is it has holes and corner cases so it seems
>> it doesn't quite hit the spot. One option is to make it work, another
>> is to take a different attack on the problem. But we need the
>> appropriate DIP.
>>
>
> Let's start by the beginning: what good design was enabled by DIP25 ? As
> long as none is presented, we can't consider it a success.

Probably git grep in phobos may be a good starting point.


Andrei