On 4/12/19 8:45 PM, Andrei Alexandrescu wrote:
> Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft:
> 
> https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md
> 
> We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are:
> 
> * reference counting of immutable data structures
> * caching
> * lazy evaluation
> 
> We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both).
> 
> The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.

There seems to be a good amount of interest in the forum. Thanks! To better move things forward, we'd need to have a more fleshed-out proposal to show the community for discussion. Until then, let's move to a special interest group.

Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.

On 4/16/19 4:15 PM, Suleyman wrote:
> On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:
>> ...
>>
>> Cheers,
>> RazvanN
> 
> What it is that this DIP enables you to do that you can't already do in @system code?

A simple answer is with the DIP you get defined behavior.

On 16.04.19 11:51, RazvanN wrote:
> On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:
>> Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft:
>>
>> https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md
>>
>> We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are:
>>
>> * reference counting of immutable data structures
>> * caching
>> * lazy evaluation
>>
>> We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both).
>>
>> The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.
> 
> This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for
> the compiler with regards to `pure` functions. I feel that optimization blockers
> for `pure` are a totally different concept than __mutable/__metadata fields.
> ...

On the contrary, it's the type system information you _need_ in order to abstract over code that mutates __metadata. An assignment to a __metadata field is something you cannot ignore even though it is `pure` and does not return any interesting value. For any statement, you need to be able to assign a valid type to a function that contains this statement, and that captures what the type system knows about the statement.

> Considering that the currently the compiler does not do any optimizations based on purity, I think it would be a lot easier to just get __mutable/__metadata fields
> in, because they are a needed feature; after that, we can think on what optimizations we want to implement and how they interact with __mutable/__mutable.
> ...

You need to specify that there are optimizations that are allowed even though they change the set of mutations that the program performs. As I suggested earlier, if you want the semantics that's least restrictive for language evolution, you need to say something that implies that all optimizations based on the function signature are allowed, not "the compiler currently does not do any optimizations". That's precisely the wrong direction if you want to be conservative.

> Consider this code:
> 
> int foo() pure
> {
>       immutable(T)* x = allocate();
>       int y = bar(x);
>       deallocate(x);
>       return y;
> }
> 
> Currently, the compiler does not do any optimizations regarding purity with it. If it did, it could have swapped the invocation of `deallocate` with the one of `bar` leading to use-after-free.

What the compiler does and does not do has no bearing on the validity of the code per language semantics.

> Annotating `deallocate` with __mutable would solve the issue in the event of compiler optimizations. However, this example has nothing to do with __mutable/__metadata fields; this kind of optimizations should be discussed in a DIP of their own and not in a __mutable/__metadata DIP. 

That's simply nonsense. You define the language semantics, the compiler developer figures out optimizations that are consistent with that semantics. You don't need any DIPs for optimizations, you need DIPs to specify how programs may or may not behave. You can't really first come and say programs may not behave a certain way and then later relax that and say that actually those programs can also behave in this other way and at the same time claim that this is a conservative approach. It is not; it breaks code.


> Moreover,
> is this the direction we want to head in?
Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.

> Require the user to mentally trace the optimizations that the compiler might do?

The user of __metadata. _Some_ kinds of optimizations.

> This is just too complicated.
> ...

Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.

> That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform.
> 
> As for __metadata fields:
> 
> 1. If we decide that __metadata fields are not conceptually part of the
> object, why would accesses to them be unsafe? We could  still make them `private`,
> but we can view them as normal accesses from a @safety perspective.
> ...

Then those accesses can't be pure because they are like accesses to global variables.

> 2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified.
> ...

1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.


> If we take out optimizations out of the picture, things become a lot more clearer.

You need to define the semantics!

> For me, the DIP is more of "how can we mutate fields in a non-mutable object", not

You clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly.

> "how do we implement optimizations regarding purity".
> 
> I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler,
> 
> ...

It shouldn't.

On Wednesday, 17 April 2019 at 01:00:35 UTC, Timon Gehr wrote:
> On the contrary, it's the type system information you _need_ in order to abstract over code that mutates __metadata. An assignment to a __metadata field is something you cannot ignore even though it is `pure` and does not return any interesting value. For any statement, you need to be able to assign a valid type to a function that contains this statement, and that captures what the type system knows about the statement.
>
> You need to specify that there are optimizations that are allowed even though they change the set of mutations that the program performs. As I suggested earlier, if you want the semantics that's least restrictive for language evolution, you need to say something that implies that all optimizations based on the function signature are allowed, not "the compiler currently does not do any optimizations". That's precisely the wrong direction if you want to be conservative.
>
I see, now I understand.
>
> What the compiler does and does not do has no bearing on the validity of the code per language semantics.
>
> That's simply nonsense. You define the language semantics, the compiler developer figures out optimizations that are consistent with that semantics. You don't need any DIPs for optimizations, you need DIPs to specify how programs may or may not behave. You can't really first come and say programs may not behave a certain way and then later relax that and say that actually those programs can also behave in this other way and at the same time claim that this is a conservative approach. It is not; it breaks code.
>
I understand now. It was very confusing for me that `pure` was
simply implemented as a set of compile time checks not backed by
any optimizations. I figured that if none of them were implemented
by now, it will certainly break a lot of code when we will have them,
so there is a great chance that `pure` will simply remain in the
current stage. If that would be the case, adding the cognitive load
for __metadata functions to the user is not worth it.
>
>> Moreover,
>> is this the direction we want to head in?
> Absolutely. If you want to use __metadata you need to be aware that pure functions can be elided, and if the mutation of __metadata within a function is not self-contained such that it can be elided, you need to annotate that function.
>
>> Require the user to mentally trace the optimizations that the compiler might do?
>
> The user of __metadata. _Some_ kinds of optimizations.
>

But it's not just the user of __metadata. If you implement in a library a refcounted object then __metadata is going to be propagated to user code.

>> This is just too complicated.
>> ...
>
> Then give up on __metadata and just implement your persistent data structures without the `immutable` qualifier slapped on top! `immutable` has no value if it cannot be used to justify equational reasoning on pure functions.
>
>> That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform.
>> 
>> As for __metadata fields:
>> 
>> 1. If we decide that __metadata fields are not conceptually part of the
>> object, why would accesses to them be unsafe? We could  still make them `private`,
>> but we can view them as normal accesses from a @safety perspective.
>> ...
>
> Then those accesses can't be pure because they are like accesses to global variables.

But they actually aren't because they are "physically" part of the object.

>
>> 2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified.
>> ...
>
> 1 and 2 are fundamentally at odds with each other if you want __mutable fields to be accessible in pure functions, because you can't check 2 safely.
>
>
>> If we take out optimizations out of the picture, things become a lot more clearer.
>
> You need to define the semantics!

What I was trying to say is that it would be easier to define semantics
if we wouldn't pretend that the compiler treats `pure` in a way that it
actually doesn't. I see your point, however, even if we have __metadata/
__mutable functions, once the purity optimizations will be implemented,
all the users will have to check their code to add __metadata/__mutable
to their functions to be sure their calls to strongly pure functions are
not elided.

>
>> For me, the DIP is more of "how can we mutate fields in a non-mutable object", not
>
> You clearly can't mutate fields in a non-mutable object except non-mutable is some higher-level abstraction. That abstraction has to be defined. One way to define semantics for __metadata is to explicitly give the set of allowed rewrites, but you can also define it implicitly.
>
>> "how do we implement optimizations regarding purity".
>> 
>> I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler,
>> 
>> ...
>
> It shouldn't.

From an academical perspective, I agree with everything you said and I think
I finally understand your point of view and your DIP draft. I think that a
small chat would have made it easier for both parties to reach an understanding.

At this point, I think it all comes down to what we want to do with `pure`. It really doesn't make any sense to take all these optimizations into account and
burden the user with __mutable functions if those optimizations will never be
implemented.

Cheers,
RazvanN

It looks to me that this proposal breaks everything that purity stands for and I don't have a real solution to it. I'm not against breaking the rules when necessary I just don't see why we need a DIP for it.

On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu wrote:
> Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.

If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.

On 4/17/19 2:04 PM, Suleyman wrote:
> It looks to me that this proposal breaks everything that purity stands for and I don't have a real solution to it. I'm not against breaking the rules when necessary I just don't see why we need a DIP for it.
> 
> On Tuesday, 16 April 2019 at 20:40:58 UTC, Andrei Alexandrescu wrote:
>> Those interested who can commit one hour a week to a meeting, please email me. We'll exchange information via email (mailing list if necessary) and meet every week to discuss progress.
> 
> If you take it somewhere else CC me I want to monitor this. My email is sahmi.soulaimane gmail.com.

Thanks. You are the only who answered. So it's Timon (who is active but did not answer my request about weekly meetings), my student Razvan Nitu, Walter, and myself, with you as an observer. It seems to me that between the lot of us we don't have a passable solution to define __metadata in such a way that'd make it possible to implement the simplest case of reference counting.

Of interest: https://github.com/dlang/dlang.org/pull/2627