Tracing garbage collection can afford the luxury of e.g. mutating data that was immutable during its lifetime.

Reference counting needs to make minute mutations to data while references to that data are created. In fact, it's not mutation of the "useful" data, the payload of a data structure; it's mutation of metadata, additional information about the data (i.e. a reference count integral).

The RCOs described in DIP74 and also RCSlice discussed in this forum need to work properly with const and immutable. Therefore, they need a way to reliably define and access metadata for a data structure.

One possible solution is to add a "@mutable" or "@metadata" attribute similar to C++'s keyword "mutable". Walter and I both dislike that solution because it's hamfisted and leaves too much opportunity for abuse - people can essentially create unbounded amounts of mutable payload for an object claimed to be immutable. That makes it impossible (or unsafe) to optimize code based on algebraic assumptions.

We have a few candidates for solutions, but wanted to open with a good discussion first. So, how do you envision a way to define and access mutable metadata for objects (including immutable ones)?


Andrei

On 1 March 2015 at 11:40, Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Tracing garbage collection can afford the luxury of e.g. mutating data that was immutable during its lifetime.
>
> Reference counting needs to make minute mutations to data while references to that data are created. In fact, it's not mutation of the "useful" data, the payload of a data structure; it's mutation of metadata, additional information about the data (i.e. a reference count integral).
>
> The RCOs described in DIP74 and also RCSlice discussed in this forum need to work properly with const and immutable. Therefore, they need a way to reliably define and access metadata for a data structure.
>
> One possible solution is to add a "@mutable" or "@metadata" attribute similar to C++'s keyword "mutable". Walter and I both dislike that solution because it's hamfisted and leaves too much opportunity for abuse - people can essentially create unbounded amounts of mutable payload for an object claimed to be immutable. That makes it impossible (or unsafe) to optimize code based on algebraic assumptions.
>
> We have a few candidates for solutions, but wanted to open with a good discussion first. So, how do you envision a way to define and access mutable metadata for objects (including immutable ones)?
>
>
> Andrei

Perhaps an operator that may be implemented to return a mutable metadata pointer for objects? From an algebraic point of view, it should be defined that mutating metadata will have no effect on the actual object.

Interestingly, I wonder if such an approach may also be used to
satisfy a common problem that people have with const objects, where
it's not possible to implement caching systems for repeated work
optimisation?
The result cache is another thing that may be stored in the object's
metadata...?

On 2/28/15 5:57 PM, Manu via Digitalmars-d wrote:
> Perhaps an operator that may be implemented to return a mutable
> metadata pointer for objects? From an algebraic point of view, it
> should be defined that mutating metadata will have no effect on the
> actual object.

But how would custom data be defined?

> Interestingly, I wonder if such an approach may also be used to
> satisfy a common problem that people have with const objects, where
> it's not possible to implement caching systems for repeated work
> optimisation?
> The result cache is another thing that may be stored in the object's
> metadata...?

That's already abuse. No.


Andrei

On 1 March 2015 at 12:22, Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 2/28/15 5:57 PM, Manu via Digitalmars-d wrote:
>>
>> Perhaps an operator that may be implemented to return a mutable metadata pointer for objects? From an algebraic point of view, it should be defined that mutating metadata will have no effect on the actual object.
>
>
> But how would custom data be defined?

However the user likes? It would just be a mutable pointer. For RC, it may just be an int*.


>> Interestingly, I wonder if such an approach may also be used to
>> satisfy a common problem that people have with const objects, where
>> it's not possible to implement caching systems for repeated work
>> optimisation?
>> The result cache is another thing that may be stored in the object's
>> metadata...?
>
>
> That's already abuse. No.

It's a very long-standing problem, one of the biggest recurring
complaints relating to the D type system, and the problem is of a very
similar nature; can't write to object because it's not mutable - but
we need this for RC, however may also be fine in caching cases because
cached results do not change the logical state of the object.
I see the same problem in both cases, and a good solution could
conceivably solve both problems.

Andrei Alexandrescu wrote:

> 
> We have a few candidates for solutions, but wanted to open with a good discussion first. So, how do you envision a way to define and access mutable metadata for objects (including immutable ones)?
> 
> 
> Andrei

It seems to me that (in the particular case of _this_ RC metadata - not
attempting to cover any more general case (beyond my ken)), the compiler
could internally mark class members that are modified inside the
opAddRef/opRelease methods as mutable. This would not require any special
source code markups.
--ted

On 2/28/15 6:33 PM, Manu via Digitalmars-d wrote:
> one of the biggest recurring
> complaints relating to the D type system

That didn't get talked about in I don't remember. -- Andrei

On 1 March 2015 at 12:48, Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 2/28/15 6:33 PM, Manu via Digitalmars-d wrote:
>>
>> one of the biggest recurring
>> complaints relating to the D type system
>
>
> That didn't get talked about in I don't remember. -- Andrei

So, what's the solution?
I've never complained about it, but I still have this problem
regularly. The solution in my experience is; 'everything is always
mutable'.

On Sun, Mar 01, 2015 at 12:53:24PM +1000, Manu via Digitalmars-d wrote:
> On 1 March 2015 at 12:48, Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> > On 2/28/15 6:33 PM, Manu via Digitalmars-d wrote:
> >>
> >> one of the biggest recurring
> >> complaints relating to the D type system
> >
> >
> > That didn't get talked about in I don't remember. -- Andrei
> 
> So, what's the solution?
> I've never complained about it, but I still have this problem
> regularly. The solution in my experience is; 'everything is always
> mutable'.

I ran into the same problem, and as a result I hardly ever make use of D's const system.

One possible solution that occurred to me, though, is to introduce a kind of "logical const" template that constructs a partially-const type, with the "logical data" parts const/immutable/etc., but the "metadata" parts mutable. User-defined attributes could be used for this purpose:

	struct Metadata {}

	class MyClass {
		int field; // regular data field
		@Metadata int fieldCache; // metadata
		...
	}

	template Const(T) {
		class Const {
			const {
				// insert stuff not marked with
				// @Metadata here
			}
			// insert stuff marked with @Metadata here
		}
	}

So then Const!MyClass is a modified version of MyClass where the data fields are const (similarly, we can define Immutable for the analogous purpose) but the fields marked as metadata will remain mutable.

Of course, this is just a crude first stab at the problem; I'm sure there's plenty of room for refinement to make it more usable, and to address some obvious roadblocks, like how to make MyClass implicitly convertible to Const!MyClass, etc.. But it seems likely that D's template machinery can actually express this in a way that does not violate the guarantee of physical const.


T

-- 
Doubt is a self-fulfilling prophecy.

The RC wrapper allocates _mutable_ memory for the reference count on the heap (it has to do that anyway, because it needs to be shared by all instances). As far as I understand, mutating a mutable variable is safe if all users of that variable are aware that it's mutable, _even if it's only reachable through a const pointer_.

This condition is easy to enforce in an RC wrapper, it just has to keep the pointer to the refcount private. To make it easier to prove safety, the pointer needs to be declared as const; all refcount manipulation needs to happen through two small (inlinable) helper methods that do the appropriate cast() magic. The refcount can also be stored next to the payload (better for cache locality anyway), in which case we don't even need to store a const pointer at all, which also means that nobody can accidentally access it in the wrong way.

On Sunday, 1 March 2015 at 06:42:02 UTC, H. S. Teoh wrote:
> On Sun, Mar 01, 2015 at 12:53:24PM +1000, Manu via Digitalmars-d wrote:
>> On 1 March 2015 at 12:48, Andrei Alexandrescu via Digitalmars-d
>> <digitalmars-d@puremagic.com> wrote:
>> > On 2/28/15 6:33 PM, Manu via Digitalmars-d wrote:
>> >>
>> >> one of the biggest recurring
>> >> complaints relating to the D type system
>> >
>> >
>> > That didn't get talked about in I don't remember. -- Andrei
>> 
>> So, what's the solution?
>> I've never complained about it, but I still have this problem
>> regularly. The solution in my experience is; 'everything is always
>> mutable'.
>
> I ran into the same problem, and as a result I hardly ever make use of
> D's const system.
>
> One possible solution that occurred to me, though, is to introduce a
> kind of "logical const" template that constructs a partially-const type,
> with the "logical data" parts const/immutable/etc., but the "metadata"
> parts mutable. User-defined attributes could be used for this purpose:
>
> 	struct Metadata {}
>
> 	class MyClass {
> 		int field; // regular data field
> 		@Metadata int fieldCache; // metadata
> 		...
> 	}
>
> 	template Const(T) {
> 		class Const {
> 			const {
> 				// insert stuff not marked with
> 				// @Metadata here
> 			}
> 			// insert stuff marked with @Metadata here
> 		}
> 	}
>
> So then Const!MyClass is a modified version of MyClass where the data
> fields are const (similarly, we can define Immutable for the analogous
> purpose) but the fields marked as metadata will remain mutable.
>
> Of course, this is just a crude first stab at the problem; I'm sure
> there's plenty of room for refinement to make it more usable, and to
> address some obvious roadblocks, like how to make MyClass implicitly
> convertible to Const!MyClass, etc.. But it seems likely that D's
> template machinery can actually express this in a way that does not
> violate the guarantee of physical const.

You still cannot access it through a const reference, though.