On 2021-11-12 14:22, Steven Schveighoffer wrote:
> On 11/12/21 1:12 PM, Andrei Alexandrescu wrote:
>> On 2021-11-12 8:03, Timon Gehr wrote:
>>> On 12.11.21 13:31, Steven Schveighoffer wrote:
>>>>
>>>> I've come to the conclusion, in order to fix this situation, so `__mutable` really means mutable and `shared` is an orthogonal piece, you need to remove the implicit sharing of `immutable`.
>>>
>>> I agree, that would be much better.
>>
>> We discussed this a couple of times. It's interesting. Sadly at this point implicit thread sharing of immutable is so baked into the language, it would take a lot of care to extricate. It would be very difficult even for Timon or Paul.
> 
> I'm not going to fight for this, I just wanted to bring it up. But I think it's pointless to try for reference counting of immutable data (the long RCSlice thread), when it's obviously not immutable data.

I'm not so sure. I mean no laws of physics or even of typing are broken.

Consider as a baseline the following: there is a global synchronized hashtable mapping pointers to objects to pointers to reference counters. The table is obviously mutable. Whenever a reference to some piece of data is copied around, a function gets called that looks up the pointer to the reference count and increments it. Depending on type, the increment is interlocked or not.

This is correct code that typechecks and everything. The challenge is to make it faster. One common solution is to group together the memory for the counter (metadata) together with the memory for the object (data). For example, metadata can be placed before the data (negative offsets) or just after the data. Then the lookup becomes as cheap as a integral to pointer addition. Sure, that all does take some gnarly code that is not automatically verifiable, but the result is safe. All languages, no matter how safe, have such manipulation in their implementation.

The real challenge is figuring out exactly the minimum amount of change to the language specification to allow defining such a scheme.

> The only solution is to make it not mean the same as immutable does now (i.e. possibly shared). The fact that `shared(immutable)` would mean the same thing as `immutable` does now is a path forward, but I can understand if nobody wants to do that. I happen to think most cases people are not sharing their immutable data, just like they mostly aren't sharing thread-local data.

Though that may be a solution, I'm not sure it's the only solution. I can tell it's an expensive enough solution (in terms of changes to the language and/or broken code) that it warrants looking for another.

On 12.11.21 23:07, Andrei Alexandrescu wrote:
> 
> The real challenge is figuring out exactly the minimum amount of change to the language specification to allow defining such a scheme.

I think one reason we haven't been making progress on this may be that your intuition for the size of the minimum required amount of change to the language specification is somewhat too small.

You have stated requirements, some of them are impossible to satisfy, most of them will require a sizeable language change. We have to go through those problems one by one and address them in order.

You have to factor in that _a lot_ of the work is actually clarifying existing semantics and fixing issues there.

My suggestion would be to prioritize satisfying the requirements that do not relate to type qualifiers.


> 
> - be as close in semantics to T[] as possible, i.e. most code should be able to simply replace T[] with RCSlice!T and work unchanged

`T[]` is magic. We first need to make it possible to define a template in druntime that works as a drop-in replacement for `T[]`, even without reference counting.

> - manage its own memory by using reference counting

Easy, except for lifetime bugs in throwing constructors. Some hassle with default init.

> - work in pure code just like T[] does

Impossible, due to strong purity. `__mutable` would have fixed this.

> - work with qualifiers like T[] does,

Impossible. `T[]` is magic and `inout` is broken.

> both RCSlice!(qual T)

Easy.

> and qual(RCSlice!T)

Impossible. Would need an escape hatch for the reference count, which you have shot down before. `__mutable` would have fixed this.

Adding appropriate subtyping relationships between them also requires a new language feature.

> - work in @nogc code just like T[] does

Very easy.

> - work in @safe code just like T[] does 

Impossible. This has spawned DIP1000 and large amounts of follow-up work. And all of this work still does not solve the problem it set out to address. I think this is the main problem. It's not actually very important to support type/function qualifiers beyond `pure @safe @nogc nothrow`. The issue is that you cannot borrow out a `ref T`. This could be solved with some explicit language support, where the compiler tracks the lifetime of the reference and lets the container know once it's dead.