April 14, 2019
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
> I don't think it should be @safe. Rather, `pure` and `immutable` should retain their meanings,

It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here.

> which implies that there are wrong ways to use `__mutable` (hence unsafe)

can you elaborate?

April 15, 2019
On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
> On 14.04.19 02:19, Suleyman wrote:
>> On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
>>> Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)
>> 
>> If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in @safe code.
>
> I don't think it should be @safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.

A few questions/thoughts (or at least food for thought for others more informed than me):

1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly?

1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion).

Later on, if the "mutable" qualifier is desired, we aren't carrying around both mutable and __mutable, or __mutable and mut or whatever ends up being the flavor of the day.

1c. Having said that, I think the "mutable" discussion and the "metadata" discussion are two separate things - which I think Andrei alluded to in the keyword change - this is potentially much bigger than just short-circuiting Immutability for a good cause.

2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute?

struct something
{
    metadata        // maybe type qualifiers disallowed in here
    {
        int a;
    }
    private
    {
        int b;
    }
    public
    {
        int c;
    }
}

And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits.

-Doc
April 15, 2019
On 15.04.19 06:05, Doc Andrew wrote:
> On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
>> On 14.04.19 02:19, Suleyman wrote:
>>> On Saturday, 13 April 2019 at 08:58:06 UTC, Timon Gehr wrote:
>>>> Won't happen, because data with __metadata annotations inside will not be put in read-only memory. (This can be known statically.)
>>>
>>> If it's adding these kinds of protections then I think it would be worth it and a step forward in making D more safe. you might also consider the 'shared' protection I mentioned earlier then maybe you can lift the restriction on modification in @safe code.
>>
>> I don't think it should be @safe. Rather, `pure` and `immutable` should retain their meanings, which implies that there are wrong ways to use `__mutable` (hence unsafe), and there are still non-cosmetic reasons to use `immutable` even if there are `__mutable` fields.
> 
> A few questions/thoughts (or at least food for thought for others more informed than me):
> 
> 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly?
> ...

Yes. Also, it means we don't reserve a previously unreserved identifier.

> 1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion).
> ...

It's not the same.
> 
> 2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute?
> 
> struct something
> {
>      metadata        // maybe type qualifiers disallowed in here
>      {
>          int a;
>      }
>      private
>      {
>          int b;
>      }
>      public
>      {
>          int c;
>      }
> }
> ...

Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?

> And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits.
> 
> -Doc

Storing the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.
April 15, 2019
On 14.04.19 22:55, Suleyman wrote:
> On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
>> I don't think it should be @safe. Rather, `pure` and `immutable` should retain their meanings,
> 
> It cannot be strongly pure. Same as we have weakly pure this is weakly immutable we're dealing with here.
> ...

No, the point is that whatever high-level rewrites you can deduce from `immutable` and `pure` shouldn't be restricted by code that modifies `__metadata`.

>> which implies that there are wrong ways to use `__mutable` (hence unsafe)
> 
> can you elaborate?
> 

struct S{
    private __metadata x;
}

void foo(immutable ref S s)pure{
    s.x += 1;
}

void main(){
    immutable S s;
    foo(s); // there is no reason for this call to happen
    assert(s.x==1); // can't rely on this, it might also be 0
}

struct S{
    private __mutable x;
}

int foo(immutable ref S s)pure{
    s.x += 1;
    return s.x;
}

void main(){
    immutable S s;
    int a=foo(s);
    int b=foo(s); // could just reuse the previous result
    assert(a!=b); // can't rely on this, might be the same
}
April 15, 2019
On 4/15/19 7:17 AM, Timon Gehr wrote:
> On 14.04.19 22:55, Suleyman wrote:
>> On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
>>> which implies that there are wrong ways to use `__mutable` (hence unsafe)
>>
>> can you elaborate?
>>
> 
> struct S{
>      private __metadata x;
> }
> 
> void foo(immutable ref S s)pure{
>      s.x += 1;
> }
> 
> void main(){
>      immutable S s;
>      foo(s); // there is no reason for this call to happen
>      assert(s.x==1); // can't rely on this, it might also be 0
> }
> 
> struct S{
>      private __mutable x;
> }
> 
> int foo(immutable ref S s)pure{
>      s.x += 1;
>      return s.x;
> }
> 
> void main(){
>      immutable S s;
>      int a=foo(s);
>      int b=foo(s); // could just reuse the previous result
>      assert(a!=b); // can't rely on this, might be the same
> }

Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA.

-Steve
April 15, 2019
On 15.04.19 14:32, Steven Schveighoffer wrote:
> On 4/15/19 7:17 AM, Timon Gehr wrote:
>> On 14.04.19 22:55, Suleyman wrote:
>>> On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
>>>> which implies that there are wrong ways to use `__mutable` (hence unsafe)
>>>
>>> can you elaborate?
>>>
>>
>> struct S{
>>      private __metadata x;
>> }
>>
>> void foo(immutable ref S s)pure{
>>      s.x += 1;
>> }
>>
>> void main(){
>>      immutable S s;
>>      foo(s); // there is no reason for this call to happen
>>      assert(s.x==1); // can't rely on this, it might also be 0
>> }
>>
>> struct S{
>>      private __mutable x;
>> }
>>
>> int foo(immutable ref S s)pure{
>>      s.x += 1;
>>      return s.x;
>> }
>>
>> void main(){
>>      immutable S s;
>>      int a=foo(s);
>>      int b=foo(s); // could just reuse the previous result
>>      assert(a!=b); // can't rely on this, might be the same
>> }
> 
> Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA.
> 
> -Steve

No, this is clearly not the use case of reference counting. Where do you see references that are being counted?
Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
April 15, 2019
On 4/15/19 8:49 AM, Timon Gehr wrote:
> On 15.04.19 14:32, Steven Schveighoffer wrote:
>> On 4/15/19 7:17 AM, Timon Gehr wrote:
>>> On 14.04.19 22:55, Suleyman wrote:
>>>> On Sunday, 14 April 2019 at 00:38:16 UTC, Timon Gehr wrote:
>>>>> which implies that there are wrong ways to use `__mutable` (hence unsafe)
>>>>
>>>> can you elaborate?
>>>>
>>>
>>> struct S{
>>>      private __metadata x;
>>> }
>>>
>>> void foo(immutable ref S s)pure{
>>>      s.x += 1;
>>> }
>>>
>>> void main(){
>>>      immutable S s;
>>>      foo(s); // there is no reason for this call to happen
>>>      assert(s.x==1); // can't rely on this, it might also be 0
>>> }
>>>
>>> struct S{
>>>      private __mutable x;
>>> }
>>>
>>> int foo(immutable ref S s)pure{
>>>      s.x += 1;
>>>      return s.x;
>>> }
>>>
>>> void main(){
>>>      immutable S s;
>>>      int a=foo(s);
>>>      int b=foo(s); // could just reuse the previous result
>>>      assert(a!=b); // can't rely on this, might be the same
>>> }
>>
>> Note that this is exactly the use case of reference counting. Which is the main draw of having a mutable portion of an immutable. I would hazard to guess that if the above has to be the semantics, this is DOA.
>>
> No, this is clearly not the use case of reference counting. Where do you see references that are being counted?
> Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?

I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed.

-Steve
April 15, 2019
On Saturday, 13 April 2019 at 03:45:35 UTC, Suleyman wrote:
> On Saturday, 13 April 2019 at 00:45:07 UTC, Andrei Alexandrescu wrote:
>> ...
>
> The problems you encountered are normal but the question is whether this feature is useful at all if the field is not explicitly shared? If so the a solution to the const case is to introduce an '@unshared' attribute instead which would protect against the worst cases of both being shared and being thread local. this attribute would disallow unsychronized read/write and at the same time prevent implicit conversion to 'shared' since it may just be thread local hence mixing it with normal 'shared' may not bring good results.
>
> example:
> ```
> struct S
> {
>     private shared int* p; // or @unshared
>
>     void inc() inout pure @system
>     {
>         import core.atomic;
>         atomicOp!"+="(*cast(shared(int*))p, 1); // or @unshared
>     }
> }
>
> void foo(ref immutable S a) pure
> {
>     a.inc();
> }
>
> void foo(ref const S a) pure
> {
>     a.inc();
> }
>
> int x; // thread local
> shared int y;
>
> void main()
> {
>     auto i = immutable S(cast(immutable)&x);
>     auto c = const S(&y);
>     //auto d = const S(cast(@unshared)&x);
>     foo(i);
>     foo(c);
> }
> ```

I also have the impression that `shared` should be used here instead of `__mutable`. Even in the current incarnation shared offers some limited guarantees that atomicity is required for writing shared values. I think people are working on making stronger guarantees for shared.

ATM uses like the one below are a no-op, I think the reuse of shared makes sense as it works nicely with the strong thread safety guarantees `immutable` has, and also offers the safety net for atomic changes. Not to mention that it is not ugly!

immutable struct Foo
{
    shared(int) x;
}



April 15, 2019
On 15.04.19 14:56, Steven Schveighoffer wrote:
> On 4/15/19 8:49 AM, Timon Gehr wrote:
>> On 15.04.19 14:32, Steven Schveighoffer wrote:
>>> ...
>>>
>> No, this is clearly not the use case of reference counting. Where do you see references that are being counted?
>> Why should the fact that the data structure is reference counted block optimizations such as eliding reference copies?
> 
> I mean when you add/remove reference for an immutable, it's passing an immutable to what needs to be a pure function, which then increments/decrements a __metadata field (just like your examples above). If one of those 2 gets elided, the reference count is hosed.
> 
> -Steve

I know, and this is why my original DIP draft had the concept of a __mutable function.
April 15, 2019
On Monday, 15 April 2019 at 11:09:23 UTC, Timon Gehr wrote:
>> 
>> A few questions/thoughts (or at least food for thought for others more informed than me):
>> 
>> 1. Maybe it's just me, but why the leading underscores? I'm wondering if this is because it's a scary/ugly-on-purpose feature like __gshared that is intended only sparingly?
>> ...
>
> Yes. Also, it means we don't reserve a previously unreserved identifier.
>
>> 1b. A lot of people have been clamoring for immutable by default, might keeping the DIP's original annotation "mutable" (with no underscores) make this a step in that direction? (with all the baggage and argument that carries with it, which is out of scope for this discussion).
>> ...
>
> It's not the same.

Oh no, I'm just suggesting that, under the original proposed DIP, the __mutable keyword might as well be made "mutable", which may be useful later on for other purposes. The metadata concept could be a lot more powerful, though, and I think it's basically orthogonal to the mutability concern:

>> 
>> 2. If the __metadata qualifier can only be applied to private fields, and carries special semantics w.r.t. escaping the type system, is there any benefit to treating it more like a visibility attribute?
>> 
>> struct something
>> {
>>      metadata        // maybe type qualifiers disallowed in here
>>      {
>>          int a;
>>      }
>>      private
>>      {
>>          int b;
>>      }
>>      public
>>      {
>>          int c;
>>      }
>> }
>> ...
>
> Is your suggestion just to make __metadata imply private, or is there anything else you are trying to say?

Not just private, but it would be a way to annotate data that lives "outside" of the POD of the class. For instance, serializing an object with variables in the metadata{} block wouldn't (or couldn't?) include them.

Another idea I had this morning is that the metadata{} block might be a nice place to put class/function attributes to avoid cluttering the signatures:

class Foo
{
    metadata
    {
        synchronized;     // can move class attributes in here
        scope;

        @(someUDA);       // UDA's live here too

        int secretSauce;  // Can always be accessed or modified by Foo, regardless of
                          // mutability
    }
}

The concept fits nicely with contracts and functions, too. Just as metadata{} blocks in a immutable object allow modification of those member variables, maybe member variables in a function metadata{} block can be modified and persisted across calls for pure functions, too. It would act just like a static variable, available at both compile-time and run-time, maybe useful for things like debugging, profiling or benchmarking. Maybe code in the in & out contract blocks can access it, but we'd make it illegal for the function body to do it. I'd need to think through the side-effects a little more.

int Bar(int a, int b)
metadata
{
    pure:                       // Not 100% sure what syntax would make sense
    @nogc:                      // maybe just force @ for everything here
    @safe:

    long startTicks;            // Stuff for benchmarking
    long endTicks;              // Not pertinent to the function itself
    static int numTimesCalled;  // This is a pure function...
}
in
{
    numTimesCalled++;           // ...but it's OK, we're just modifying it's metadata
    ...
}
do
{
    if(numTimesCalled > 4)      // Illegal to do this here, breaks purity
    ...
}

>
>> And... maybe metadata shouldn't be stored with the object at all? This way, it doesn't affect the layout of the object in memory, and doesn't require the need to short-circuit the type system if a later use of the class is marked Immutable. The object's metadata lives, not sorta-outside the type system, but outside the object itself? There are some challenges I can think of with this approach that I can expand on, but also some benefits.
>> 
>> -Doc
>
> Storing the data within the object is the point. As the original post states, the motivation for the DIP is to allow memory allocation schemes such as reference counting, as well as lazy initialization to be implemented for `immutable`-qualified data structures. The mutability is supposed to be an implementation detail -- from the outside, the data will still appear to be `immutable`. (The current version of the DIP completely misses this point though, leaking __metadata into the type system.) Compiler optimizations can change the semantics of code that operates on __metadata. (e.g., if it is able to elide a read to a lazily-initialized field, that field will not be initialized, if it elides a reference copy, the reference count does not need to be updated, etc.) Therefore, modifications of __metadata need to be consistent with rewrites that are valid for strongly pure functions.

Sure, I'm just suggesting that keeping that reference count or other metadata separately is a possible workaround - as long as there's an easy way to get the metadata for a given object (whether in memory alongside the rest of the object's data or not).

The idea of storing metadata member variables separately is an implementation detail. To the class, it appears that the metadata members are no different than regular variables, but casting to void* or serializing it leaves out the metadata, so it's a safe place to keep things like reference counts. In this scheme, metadata can be added to classes or structs without changing their memory layout, and metadata vars have some other special powers as far as being treated differently by the type system, strictly because they are stored "outside" the object memory, not because of a special-case in the type system.

-Doc