I'm reviewing Ali's insightful presentation from 2013 DConf. I wonder has he or anyone else followed up on the concepts or formalized some guidelines that could achieve consensus. I definitely agree it would be helpful to have a 50 Ways To Improve Your D. The first thing I'd like to see is a set of guidelines on mutability along the lines he discussed in his talk. But as it stands, I don't know if there was any finalization/consensus. For instance, from the first two guidelines at the end (after several iterative reconstructions of those guidelines):

1. If a variable is never mutated, make it const, not immutable.
2. Make the parameter reference to immutable if that is how you will use it anyway. It is fine to ask a favor from the caller.
...

If you follow (1) exclusively, why the need for immutable in the language at all?

Maybe it is a philosophical question, but where does immutability really come from? Is it an aspect of some piece of data or is it a promise that function will not change it? Or is it a requirement by a function that data passed not be changed by anyone else?

The two keywords cover all in some sense.

I found the end of the video amusing, when one gentleman looking at a rather sophisticated "canonical" struct with three overloads for 'this(...)' and two overloads for opAssign, asked if all those methods were required. I think there was back and forth and head-scratching. Another asked if the language designers were happy with the resultant complexity. Naturally the answer was yes - it is a good mix. If that is the case I wonder if the reason is they don't write software in D like Ali was looking to develop. I imagine standard library code is much more functional than OO by its very nature. My money says you will not find a struct S constructed like Ali's in the wild - it is just too much boilerplate. But surely they have their own guidelines/approaches.

By posting this I am not looking for a single answer to the simple question above as it is just one of many questions in the set of "how do you choose among the options in general". If you have such guidelines - please post them.

Thanks
Dan

On 10/02/2013 06:09 AM, Daniel Davidson wrote:

> I'm reviewing Ali's insightful presentation from 2013 DConf. I
> wonder has he or anyone else followed up on the concepts or
> formalized some guidelines that could achieve consensus.

I have not followed up on those concepts.

> 1. If a variable is never mutated, make it const, not immutable.
> 2. Make the parameter reference to immutable if that is how you will use
> it anyway. It is fine to ask a favor from the caller.
> ...
>
> If you follow (1) exclusively, why the need for immutable in the
> language at all?

For one, immutable is thread-safe. Also, an object can safely hold on to a piece of data and trust that it will never change. A file name is a good example: An object need not copy a string that is given to it as a constructor parameter. That string will be immutable as long as that reference is valid.

> Maybe it is a philosophical question, but where does
> immutability really come from? Is it an aspect of some piece of
> data or is it a promise that function will not change it? Or is
> it a requirement by a function that data passed not be changed
> by anyone else?

The two concepts are necessarily conflated in languages like C++ because for those languages there is only the 'const' keyword.

D's immutable is the last point you made: a requirement by a function that data passed not be changed by anyone else?

> I found the end of the video amusing, when one gentleman looking at a
> rather sophisticated "canonical" struct with three overloads for
> 'this(...)' and two overloads for opAssign, asked if all those methods
> were required.

I think you are referring to Ben Gertzfield's question. He later told me that he was sorry that he asked the wrong question at the wrong time. :D

To be honest, that slide was added more as an after thought. I suspect that that canonical struct is simpler today after changes made to dmd since the conference. (I will look at it later.) For example, there are no 'pure' keywords around in that code. I think the (ability to) use of that keyword will simplify matters.

However, I should have answered Ben's question by the following:

* If the struct is simply a value type or has no mutable indirections, no member function is really necessary. D takes care of it automatically.

* Sometimes post-blit will be necessary for correctness. For example, we may not want two objects share the same internal buffer, File, etc.

* As noted in the presentation, the default behavior of struct assignment in D is exception-safe: first copy then swap. In some cases that automatic behavior is less than optimal e.g. when an object's existing buffer can be reused; no need to "copy then swap (which implicitly destroys)" in that case. (The spec or Andrei's book has that exact case as an example.) So, opAssign should be for optimization reasons only.

(I will look at these again later.)

> By posting this I am not looking for a single answer to the simple
> question above as it is just one of many questions in the set of "how do
> you choose among the options in general". If you have such guidelines -
> please post them.
>
> Thanks
> Dan
>

Thanks for asking these questions.

Ali

On Wednesday, 2 October 2013 at 17:07:55 UTC, Ali Çehreli wrote:
> On 10/02/2013 06:09 AM, Daniel Davidson wrote:
>
> > 1. If a variable is never mutated, make it const, not
> immutable.
> > 2. Make the parameter reference to immutable if that is how
> you will use
> > it anyway. It is fine to ask a favor from the caller.
> > ...
> >
> > If you follow (1) exclusively, why the need for immutable in
> the
> > language at all?
>
> For one, immutable is thread-safe.

Ok - then new guidelines should be added to the effect that, "if you want a guarantee of read-only that can be counted on across threads, use immutable". But, I'm not sure that is a sensible guideline or when you know up-front to deal with it. Often threading is an afterthought. My point is not that immutable is not useful, but that if you never make an lvalue immutable in the guidelines (via guideline 1), it is hard to see them ever being used at all. The case in the slides where it was used was to when a method required it on a parameter type with mutable aliasing. In this case the client was required to copy beforehand. In other words, I think even the summary simple guidelines are not so simple, maybe because they are incomplete. I would love to help complete them.

> Also, an object can safely hold on to a piece of data and trust that it will never change.

I think for this to be true in general, the piece of data must be immutable, not just tail-immutable. If the class holds onto a string (i.e. only tail-immutable), the class can add to the string - just not change the existing characters.

> A file name is a good example: An object need not copy a string that is given to it as a constructor parameter. That string will be immutable as long as that reference is valid.
>

IMHO file name and ultimately string are the *worst* example :-) The reason is they are a special case because they provide *safe sharing*. String encapsulates its contiguous data and does not allow mutation of elements. But it does allow concatenation that by design detail can not be seen by other references to the same data because of copy-on-write. Granted, if you have a handle to  immutable(T)[] you know *it* (i.e. the complete string) will not change (unless you change it). The problem is it is just a byproduct of the implementation details of immutable(T)[]. From the referrer's point of view it is immutable.

For example, the following comparable associative array does not have the same benefit as string since two references to the same underlying data *do see changes*:

    import std.stdio;

    struct V {
      string s = "I'm a value";
    }

    alias immutable(V)[int] Map;

    void main() {
      Map m = [ 1:V(), 2:V("bar") ];
      Map m2 = m;
      m2[3] = V("moo");
      writeln(m);
      writeln(m2);
    }

The reason I think string is a bad example is one might incorrectly assume tail-immutable is good enough for true read/only thread-safe data. And it is for string or any immutable(T)[], but not in general.


> > Maybe it is a philosophical question, but where does
> > immutability really come from? Is it an aspect of some piece
> of
> > data or is it a promise that function will not change it? Or
> is
> > it a requirement by a function that data passed not be changed
> > by anyone else?
>
> The two concepts are necessarily conflated in languages like C++ because for those languages there is only the 'const' keyword.
>
> D's immutable is the last point you made: a requirement by a function that data passed not be changed by anyone else?
>
> > I found the end of the video amusing, when one gentleman
> looking at a
> > rather sophisticated "canonical" struct with three overloads
> for
> > 'this(...)' and two overloads for opAssign, asked if all
> those methods
> > were required.
>
> I think you are referring to Ben Gertzfield's question. He later told me that he was sorry that he asked the wrong question at the wrong time. :D
>

I can understand - I would not want to put you on the spot on stage either. But, I have no qualms doing it in the news groups :-)

> To be honest, that slide was added more as an after thought. I suspect that that canonical struct is simpler today after changes made to dmd since the conference. (I will look at it later.) For example, there are no 'pure' keywords around in that code. I think the (ability to) use of that keyword will simplify matters.
>
> However, I should have answered Ben's question by the following:
>
> * If the struct is simply a value type or has no mutable indirections, no member function is really necessary. D takes care of it automatically.
>

Which I wonder if is a good idea. Immediately from that comes the problem that "D takes care of it" breaks when you go from no mutable aliasing to some mutable aliasing. That mutable aliasing could be deep down in the composition chain. I know you can feel this issue since your slides point out cases where code may break when well encapsulated structs make changes that should not impact client code but will break them.

> * Sometimes post-blit will be necessary for correctness. For example, we may not want two objects share the same internal buffer, File, etc.
>

Agreed - but post-blits are easier said than done with nested data. For instance, if you have a member that is a struct that does not provide a post-blit (i.e. the original author did not concern himself with sharing) then you have to effectively write his post-blit for him in your post-blit. This is why I lobby for language supported generalized dup.

> * As noted in the presentation, the default behavior of struct assignment in D is exception-safe: first copy then swap. In some cases that automatic behavior is less than optimal e.g. when an object's existing buffer can be reused; no need to "copy then swap (which implicitly destroys)" in that case. (The spec or Andrei's book has that exact case as an example.) So, opAssign should be for optimization reasons only.
>
> (I will look at these again later.)
>

I look forward to it and would be glad to help get to a good set of guidelines. For instance filling out this, which only covers variable declarations - not parameter passing, with a robust "when to use".

| context         | T | C(T) | I(T) | I(T)[] |
|-----------------+---+------+------+--------|
| local stack     |   |      |      |        |
| local heap      |   |      |      |        |
| global          |   |      |      |        |
| instance member |   |      |      |        |
| static member   |   |      |      |        |

You could actually have two tables - one for T1 with no mutable aliasing and one for T2 with mutable aliasing. But relying on that in the decision matrix may lead to issues.

Thanks,
Dan

On 10/02/2013 10:07 AM, Ali Çehreli wrote:

> On 10/02/2013 06:09 AM, Daniel Davidson wrote:
>
>  > I'm reviewing Ali's insightful presentation from 2013 DConf. I
>  > wonder has he or anyone else followed up on the concepts or
>  > formalized some guidelines that could achieve consensus.
>
> I have not followed up on those concepts.
>
>  > 1. If a variable is never mutated, make it const, not immutable.
>  > 2. Make the parameter reference to immutable if that is how you will use
>  > it anyway. It is fine to ask a favor from the caller.
>  > ...

I have to report that I am not finding much time to go deeper in these issues. However, I spent some time this weekend, gaining a hint of enlightenment.

immutable is a requirement on two concepts, second of which is new to me:

1) As it is commonly known, it is a requirement that the data does not mutate.

2) It is also a requirement on the type that the variables of that type will always be usable as immutable. I have not formalized "be usable" yet but what I mean is that the type cannot be modified in the future in a way that inhibits its use in existing code.

To look at just one usage example, the following line carries two requirements:

    auto a = T();
    immutable b = a;

1) b will be an immutable copy of a.

2) T will always be usable as in that fashion.

If T appears on an API, it is the responibility of the user to ensure whether they are allowed to treat T in that way. Otherwise, they risk maintainability if the module decides to change T in any way that fits the module's needs. If they have not yet advertised that T can be used as immutable, it should not be.

So, the question is how can a module expose a type that can always be used as immutable? Can the module guarantee that its future needs will never disallow T's use as immutable (e.g. by inserting mutable references into the type)? Can the library always guarantee that T can be used as immutable by improving its definition, e.g. by adding necessary opCast, opEquals, 'alias this', constructors, etc?

I don't know these answers... :/ (yet! :p)

Ali

On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson wrote:
> 1. If a variable is never mutated, make it const, not immutable.
> 2. Make the parameter reference to immutable if that is how you will use it anyway. It is fine to ask a favor from the caller.
> ...

I think guideline 1 should be about "arguments" not "variables". Functions should take const arguments, so they can be called with mutable and immutable values.

I would rephrase the second guideline as: "Never dup or idup an argument to make it mutable or immutable, but require the caller to do this (might be able to avoid it)".

For more guidelines:

3. Return value, which are freshly created, should never be const. They should be mutable or immutable.

Basically, I cannot come up with a case, where const would be preferable to mutable and immutable. Maybe someone is able to find a good example? Of course, this does not hold for returned values, which are retrieved from somewhere (array,container,etc). In this case, I have no general advice.

4. Data structures should not restrict themselves to be mutable, const, or immutable.

This is a noble goal, which in reality is probably either trivial or impossible. So I am not sure, if it is worthy to be called a guideline.

On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:
> To look at just one usage example, the following line carries two requirements:
>
>     auto a = T();
>     immutable b = a;
>
> 1) b will be an immutable copy of a.
>
> 2) T will always be usable as in that fashion.
>
> If T appears on an API, it is the responibility of the user to ensure whether they are allowed to treat T in that way. Otherwise, they risk maintainability if the module decides to change T in any way that fits the module's needs. If they have not yet advertised that T can be used as immutable, it should not be.

I do not agree with you, that the user has the responsibility. Rather I think the provider of T has the responsibility to maintain backwards compatibility. My principle is "anything is allowed, unless explicitly forbidden". This includes immutable type constructing. It is great that we get a type error, if backwards compatibility is broken.

Nevertheless, the question of responsibility seems to be subjective. Are there any clear technical reasons for either way?

Unfortunately, there is no way for the provider to allow or disallow immutable(T). Maybe there should be a UDA or something for this?

On 10/08/2013 03:12 PM, qznc wrote:

> On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:
>> To look at just one usage example, the following line carries two
>> requirements:
>>
>>     auto a = T();
>>     immutable b = a;
>>
>> 1) b will be an immutable copy of a.
>>
>> 2) T will always be usable as in that fashion.
>>
>> If T appears on an API, it is the responibility of the user to ensure
>> whether they are allowed to treat T in that way. Otherwise, they risk
>> maintainability if the module decides to change T in any way that fits
>> the module's needs. If they have not yet advertised that T can be used
>> as immutable, it should not be.
>
> I do not agree with you, that the user has the responsibility.

I have difficulty agreeing with myself as well. :) However, the power of immutable makes me think so. Interestingly, once a user creates an immutable variable of a type, that type must support that use.

> Rather I
> think the provider of T has the responsibility to maintain backwards
> compatibility.

Agreed but I don't know how. Here is challenge: Let's start with the following program:

// Library type
struct MyInt
{
    int i;
}

void main()
{
    // User code
    auto a = MyInt(1);
    immutable b = a;
}

Let's assume that the library adds a private dynamic array of ints to that type:

// Library type
struct MyInt
{
    int i;
    private int[] history;    // <-- Added
}

void main()
{
    // User code
    auto a = MyInt(1);
    immutable b = a;          // <-- Existing code breaks
}

Error: cannot implicitly convert expression (a) of type MyInt to immutable(MyInt)

Apparently, the library made a change that made its type non-immutable-able. :p What is missing in MyInt? How should it be defined instead of simply adding 'history'?

> My principle is "anything is allowed, unless explicitly
> forbidden". This includes immutable type constructing. It is great that
> we get a type error, if backwards compatibility is broken.

Agreed. What is the solution?

> Nevertheless, the question of responsibility seems to be subjective. Are
> there any clear technical reasons for either way?

Technically, MyInt is in a breaking state because the user used it as immutable. One way to look at this situation is to observe that the user took some freedom of the library just by using its type as immutable.

> Unfortunately, there is no way for the provider to allow or disallow
> immutable(T). Maybe there should be a UDA or something for this?

I think the language is lacking tools to support the use case above.

Ali

On 10/08/2013 03:03 PM, qznc wrote:

> On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson wrote:
>> 1. If a variable is never mutated, make it const, not immutable.
>> 2. Make the parameter reference to immutable if that is how you will
>> use it anyway. It is fine to ask a favor from the caller.
>> ...
>
> I think guideline 1 should be about "arguments" not "variables".
> Functions should take const arguments, so they can be called with
> mutable and immutable values.

As demonstrated in the presentation, const erases the actual type. So, the function must take a copy if the argument is to be used in an immutable context.

Instead, if the immutable appears on the interface, that copy will be avoided.

> I would rephrase the second guideline as: "Never dup or idup an argument
> to make it mutable or immutable, but require the caller to do this
> (might be able to avoid it)".

Agreed. But it means you agree with me that immutable should indeed appear on function signatures as needed.

> For more guidelines:
>
> 3. Return value, which are freshly created, should never be const. They
> should be mutable or immutable.

Agreed. I say that it should be mutable if the function is pure, as such a return values is automatically convertible to immutable.

> Basically, I cannot come up with a case, where const would be preferable
> to mutable and immutable. Maybe someone is able to find a good example?

You mean on the return type? Perhaps if the freshly created object carries references to existing data that you don't want mutated. Making the head const makes tail const; so the members would all be protected.

However, it is always possible to return a mutable object of a type that has const members:

struct Result
{
    const(int)[] reference_to_precious_existing_data;
}

Result freshly_created_value()
{
    // ...
}

> Of course, this does not hold for returned values, which are retrieved
> from somewhere (array,container,etc). In this case, I have no general
> advice.
>
> 4. Data structures should not restrict themselves to be mutable, const,
> or immutable.

What is the template of a struct that can be used as such? Providing simple values seems to be insufficient:

struct MyInt
{
    int i;
    private int[] history;
}

What else should the struct above have to make it usable as mutable, const, immutable?

> This is a noble goal, which in reality is probably either trivial or
> impossible. So I am not sure, if it is worthy to be called a guideline.

We have to nail this down already. We have this great addition of immutable in the language but we either don't know how to use it or the language is lacking hopefully trivial things to make it work.

Ali

On Wednesday, 9 October 2013 at 04:41:35 UTC, Ali Çehreli wrote:
> > 4. Data structures should not restrict themselves to be
> mutable, const,
> > or immutable.
>
> What is the template of a struct that can be used as such? Providing simple values seems to be insufficient:
>
> struct MyInt
> {
>     int i;
>     private int[] history;
> }
>
> What else should the struct above have to make it usable as mutable, const, immutable?
>
> > This is a noble goal, which in reality is probably either
> trivial or
> > impossible. So I am not sure, if it is worthy to be called a
> guideline.
>
> We have to nail this down already. We have this great addition of immutable in the language but we either don't know how to use it or the language is lacking hopefully trivial things to make it work.

For this case, there are basically two approaches: a) prevent the use as immutable in the first place or b) make it work as immutable by force.

For version a) we could have a UDA like this:

  @NeverImmutable struct MyInt { int i; }

  auto x = immutable(MyInt)(42);  // compile error
  immutable y = MyInt(42);        // compile error

However, the library writer would have to anticipate the private dynamic array, which probably does not happen. Defensive coding like putting @NeverImmutable on everything would defeat the purpose as well.

For version b) we would relax the type system, which opens an ugly can of worms of unknown size. Basically, immutable data would be allowed to have mutable private data, if the author sufficiently asserts everything is correct. Since there are lots of interesting compiler optimizations with immutable things, I can imagine there will be lots of subtle errors until one can make something like this work.

At this point, I cannot imagine that there is a solution which makes this case work, so I think we have to accept this breakage.

On Wednesday, 9 October 2013 at 04:31:55 UTC, Ali Çehreli wrote:
> On 10/08/2013 03:12 PM, qznc wrote:
>
> > On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:
> >> To look at just one usage example, the following line
> carries two
> >> requirements:
> >>
> >>     auto a = T();
> >>     immutable b = a;
> >>
> >> 1) b will be an immutable copy of a.
> >>
> >> 2) T will always be usable as in that fashion.
> >>
> >> If T appears on an API, it is the responibility of the user
> to ensure
> >> whether they are allowed to treat T in that way. Otherwise,
> they risk
> >> maintainability if the module decides to change T in any way
> that fits
> >> the module's needs. If they have not yet advertised that T
> can be used
> >> as immutable, it should not be.
> >
> > I do not agree with you, that the user has the responsibility.
>
> I have difficulty agreeing with myself as well. :) However, the power of immutable makes me think so. Interestingly, once a user creates an immutable variable of a type, that type must support that use.
>
> > Rather I
> > think the provider of T has the responsibility to maintain
> backwards
> > compatibility.
>
> Agreed but I don't know how. Here is challenge: Let's start with the following program:
>
> // Library type
> struct MyInt
> {
>     int i;
> }
>
> void main()
> {
>     // User code
>     auto a = MyInt(1);
>     immutable b = a;
> }
>
> Let's assume that the library adds a private dynamic array of ints to that type:
>
> // Library type
> struct MyInt
> {
>     int i;
>     private int[] history;    // <-- Added
> }
>
> void main()
> {
>     // User code
>     auto a = MyInt(1);
>     immutable b = a;          // <-- Existing code breaks
> }
>
> Error: cannot implicitly convert expression (a) of type MyInt to immutable(MyInt)
>
> Apparently, the library made a change that made its type non-immutable-able. :p What is missing in MyInt? How should it be defined instead of simply adding 'history'?
>

The example is great. I would say it like "Apparently the library made a change that added mutable aliasing and therefore broke the special logic the compiler applies to types without aliasing". I think the root cause is the language special casing structs with no mutable aliasing by allowing the "immutable b = a" to work in the first place. One step toward a solution would be the existence of a generalized dup. Then calls to "immutable b = a" with mutable aliasing could lower to a call to the generalized dup, causing a deep copy and therefore still allowing assignment from non-immutable to immutable. This has its own set of problems - like what if the library designer wants sharing and not deep copies (perhaps they employ copy on write semantics on their own).

What are other solutions or ways to prevent this jump from no aliasing to mutable aliasing breaking code? Maybe assume mutable aliasing on all your structs from the start, then those statements like "immutable b = a" won't appear in the first place.

struct T {
  ...
  version(unittest) int[] _justToAddMutableAliasing;
}

Now you know that assignment of "immutable b = a" will fail in unittests. But this is crazy and it would only help your users if they run unittests.

>
> Technically, MyInt is in a breaking state because the user used it as immutable. One way to look at this situation is to observe that the user took some freedom of the library just by using its type as immutable.
>

I don't quite understand the terminology "using it's type as immutable". I think the breaking state comes from copying in general having two flavors - shallow and deep. Transitive deep copy always offers the option of immutable source and target. Shallow copy also offers the option of immutable source and target, as long as the type has no mutable aliasing, simply because a shallow copy is also a deep copy if no aliasing. And, shallow copy does not offer the option of immutable in the face of types with mutable aliasing. It is when types go from having no mutable aliasing to having mutable aliasing (and the reverse) that the rules of the compiler change the game.

> > Unfortunately, there is no way for the provider to allow or
> disallow
> > immutable(T). Maybe there should be a UDA or something for
> this?
>

One way might be to disallow copy and assignment (is this a D capability?) and provide pure factory that only returns immutable(T).

> I think the language is lacking tools to support the use case above.
>

I think if the default assignment and copy constructor for structs with aliasing were automatic transitive deep copy this issue would disappear. Other issues would come up. Also, I'm pretty sure Walter is not a fan of this concept because he does not advocate copying any data in postblits.

> Ali