April 15, 2019
On 4/15/19 9:23 AM, Timon Gehr wrote:
> On 15.04.19 14:56, Steven Schveighoffer wrote:
>> On 4/15/19 8:49 AM, Timon Gehr wrote:
>>> On 15.04.19 14:32, Steven Schveighoffer wrote:
>>>> ...
>>>>
>>> No, this is clearly not the use case of reference counting. Where do you see references that are being counted?
>>> Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
>>
>> I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed.
>>
> 
> I know, and this is why my original DIP draft had the concept of a __mutable function.

Ah, ok. So a __mutable function is not strong-pure. That would work.

-Steve
April 15, 2019
On 4/15/19 9:14 AM, Radu wrote:

> I also have the impression that `shared` should be used here instead of `__mutable`. Even in the current incarnation shared offers some limited guarantees that atomicity is required for writing shared values. I think people are working on making stronger guarantees for shared.
> 

Only immutable structs would have shared metadata. Normal mutable structs could have thread-local metadata. So that doesn't work.

-Steve
April 15, 2019
On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:
> On 14.04.19 22:55, Suleyman wrote:
>> ...
>
> No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`.
>
> ...

Just making it @system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments.

April 15, 2019
On 15.04.19 17:08, Suleyman wrote:
> On Monday, 15 April 2019 at 11:17:55 UTC, Timon Gehr wrote:
>> On 14.04.19 22:55, Suleyman wrote:
>>> ...
>>
>> No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`.
>>
>> ...
> 
> Just making it @system is not helping much instead of evading reality you rather treat as it really is an immutable ref with mutable fields hence the compiler must not elide function calls with these weakly immutable arguments.
> 

At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?
April 15, 2019
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
> At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?

I don't know but immutability has been breached for sure and hiding that with @system doesn't help I would say disallowing __mutable inside immutable objects makes more sense than just concealing the crime with @system unless someone has a safe solution.
April 15, 2019
On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
> At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?

Otherwise the sole protection from being stored in readonly memory is not enough incentive for adding a keyword feature if this is the only contribution beyond an unsafe cast then a compiler directive should suffice such as pragma(noROM) or similar.
April 15, 2019
On 15.04.19 19:24, Suleyman wrote:
> On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
>> At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?
> 
> Otherwise the sole protection from being stored in readonly memory

This is not true and it is not even the right way to think about it. The language semantics is distinct from how it is implemented in a compiler. The reason why data with __mutable fields cannot be put in readonly memory is because crash on write is not acceptable behavior. It does not even need to be explicitly specified.

> is not enough incentive for adding a keyword feature

The cost of adding a __keyword is pretty close to zero. And anyway, that's a superficial syntactic concern, not even really worth debating over at this point, as a) the current version of the DIP gets the semantics wrong and b) the necessary changes to the language specification are of similar size whether you add a __keyword or a pragma. (And whether or not you add a tiny bit more type checking is independent of the annotation syntax.)

> if this is the only contribution beyond an unsafe cast

This is not what's being proposed. There are unsafe operations and there are invalid operations. The DIP's aim is to make some cases unsafe that were previously invalid. (And if you have to implement an annotation for fields anyway, adding the handful lines of frontend code that avoid additional casts, which are a blunt instrument that can hide what would otherwise be type errors, e.g. during a refactoring, does not seem bad enough to me to not even be counted as a contribution.)

> then a compiler directive should suffice such as pragma(noROM) or similar.
A pragma is sufficient (but not required) anyway, but noROM is a terrible name.
April 15, 2019
On 15.04.19 18:37, Suleyman wrote:
> On Monday, 15 April 2019 at 15:59:35 UTC, Timon Gehr wrote:
>> At that point, why do you need to slap `immutable` on your type in the first place? Why does your proposed semantics make any sense for reference counting or caching/lazy evaluation?
> 
> I don't know but immutability has been breached for sure

No. It's like saying Haskell does not support immutable data because the only way to implement lazy evaluation on the machine is using mutation. (Or even worse, to claim that D does not support immutability because the garbage collector can reclaim memory that was previously typed immutable.)

I think D needs to be able to implement its own runtime library and manual memory allocation schemes without compiler magic unavailable to user programs.

> and hiding that with @system doesn't help

Nothing is being hidden.

> I would say disallowing __mutable inside immutable objects  makes more sense

Regarding sense: I have yet to find any in your "concealing" argument. I have tried and wasted too much time now.

> than just concealing the crime with @system
@system means you are not protected against writing programs that do not have defined semantics. Seems like a good enough fit.

> unless someone has a safe solution.

No, that makes no sense at all. The only simple enough safe solution is to turn all __metadata use cases into compiler magic one-by-one. And you win nothing. The simplest way to implement that is to just add __metadata and simply add a compiler check that ensures user code cannot access it.
April 16, 2019
On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:
> Razvan Nitu is working on the DIP initiated by Timon Gehr, known colloquially as the one that introduces __mutable - i.e. a mechanism for allowing controlled changes to immutable data. Here's a draft:
>
> https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/DIP1xxx-rn.md
>
> We figured that we're dealing a misnomer - we don't want __mutable, but instead __metadata - information that is nominally part of the object but needs certain leeway from the type system. Typical use cases are:
>
> * reference counting of immutable data structures
> * caching
> * lazy evaluation
>
> We got stuck at the interaction of __mutable with const parent objects (unclear whether the parent object originated as immutable or unqualified), and how pure functions should deal with __mutable. The few solutions we are toying with are either incomplete or too complicated (or both).
>
> The help of a few PL and compiler specialists would be very valuable here. I'm cc'ing a few, if anyone wants to help please let us know.

This is a draft proposal and it is not finished. The initial draft that was done by Timon [1] considers __mutable functions that act as optimization blockers for
the compiler with regards to `pure` functions. I feel that optimization blockers
for `pure` are a totally different concept than __mutable/__metadata fields.

Considering that the currently the compiler does not do any optimizations based on purity, I think it would be a lot easier to just get __mutable/__metadata fields
in, because they are a needed feature; after that, we can think on what optimizations we want to implement and how they interact with __mutable/__mutable.

Consider this code:

int foo() pure
{
     immutable(T)* x = allocate();
     int y = bar(x);
     deallocate(x);
     return y;
}

Currently, the compiler does not do any optimizations regarding purity with it. If it did, it could have swapped the invocation of `deallocate` with the one of `bar` leading to use-after-free. Annotating `deallocate` with __mutable would solve the issue in the event of compiler optimizations. However, this example has nothing to do with __mutable/__metadata fields; this kind of optimizations should be discussed in a DIP of their own and not in a __mutable/__metadata DIP. Moreover,
is this the direction we want to head in? Require the user to mentally trace the optimizations that the compiler might do? This is just too complicated.

That is why, I think that we should focus on implementing __metadata with regards to fields and later on think about the optimizations that we can perform.

As for __metadata fields:

1. If we decide that __metadata fields are not conceptually part of the
object, why would accesses to them be unsafe? We could  still make them `private`,
but we can view them as normal accesses from a @safety perspective.

2. I agree that `purity` should not be affected by __mutable/__metadata fields; the object passed as argument will not be conceptually modified.

If we take out optimizations out of the picture, things become a lot more clearer.
For me, the DIP is more of "how can we mutate fields in a non-mutable object", not
"how do we implement optimizations regarding purity".

I don't see why the addition of __metadata should be delayed by how optimizations based on purity are applied in the compiler,

Cheers,
RazvanN

[1] https://github.com/RazvanN7/DIPs/blob/Mutable_Dip/DIPs/timon_dip.md
April 16, 2019
On Tuesday, 16 April 2019 at 09:51:33 UTC, RazvanN wrote:
> ...
>
> Cheers,
> RazvanN

What it is that this DIP enables you to do that you can't already do in @system code?