On Wednesday, 16 October 2013 at 19:01:59 UTC, H. S. Teoh wrote:
> On Wed, Oct 16, 2013 at 08:49:51PM +0200, Daniel Davidson wrote:
>> On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:
>> >On Wed, Oct 16, 2013 at 07:23:24PM +0200, Daniel Davidson wrote:
> [...]
>> >>If you have a type that has now or may ever have in the future any
>> >>mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts are
>> >>mutable) do not ever use the immutable keyword in any context as
>> >>things just break down.
>> >
>> >Yes, because immutable means nothing, no one, can change the data
>> >after it's constructed, ever. If you want mutable aliasing, what you
>> >want is const, not immutable.
>> >
>> 
>> I think the term "mutable aliasing" and "what you want" don't work
>> together. "mutable aliasing" is not about context of usage or what
>> you want, it is a compile time attribute of a struct. I want to use
>> associative arrays in composition because I think they are the way I
>> view and use my data. `struct T { string[string] i; }` has mutable
>> aliasing, like it or not. So, it is not so much that I want mutable
>> aliasing - in fact I fear it. But once you use AAs it is a fact of
>> life.
> [...]
>> Is the suggestion here: use immutable in any composition context
>> where you have slices and you want to not "worry about mutable
>> aliasing". So, do like string does: any case where you have T[] as a
>> member, prefer immutable(T)[] since then you don't have to worry
>> about sharing. Well, not sure that works in general because the T in
>> string is char and has no mutable aliasing itself. Suppose T itself
>> has mutable aliasing, then what? Or, suppose it is a struct with no
>> mutable aliasing *now*. Who's to say it won't change.
>> 
>> So, where does all this leave us w.r.t. a good set of guidelines?
>
> Sorry, I kinda barged into this conversation, so I'm not sure what
> exactly you're trying to achieve here. What kind of composition contexts
> do you have in mind? Maybe you could help me understand what you're
> trying to do?
>

Thanks. I appreciate any help I can get. Here is a small sample: http://pastebin.com/TeiQ9DYa

Other samples have at least 5 levels of composition. I'm generating the structure from json schema so I'm looking for something that scales and is immune to changes in deeply nested members.

> Keeping in mind that AA's, as they are currently implemented, leaves a
> lot of room for improvement, to say the least. So you might be running
> into some AA-related issues, not the type system proper.
>

I understand. It is a shame. If the data source is rich json data - I think AAs are a necessity.

On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:
> On Wednesday, 16 October 2013 at 19:06:06 UTC, Daniel Davidson wrote:
>> I don't understand how it could be fine. As code grows it would lead to people not adding useful members like history just because of the huge repercussions.
>>
>> struct User {
>>   immutable(Foo) foos;
>> }
>>
>> How can I as a user adapt to that change? Before the change assignment worked equally well among all of Mutable, Immutable, Const. After that change any `foos ~= createFoo(...)` would require change. And it is not clear what the change would be.
>
> I think any usage of immutable with types/entities not initially designed for immutability is an potential mistake and in that sense it is good that change has broken the user code. Same goes for operating on immutable entity in generic code as if it is a value type without actually checking it via introspection.

I don't disagree. Now, what does it mean to "initially design for immutability" and what are the guidelines. I thought that was kind of what we were talking about here. How to effectively use const and immutable. If the rule is - don't use immutable unless you have immutability in mind for your design, ok. Maybe take it a step further...

How about this - show a good use of immutable in any context where mutable aliasing (e.g. AAs) are in the mix. If the response is there are none, it is just too hairy of a proposition then something is wrong.

One general idea that was presented by Ali is that an immutable parameter means that not only are you guaranteeing that you will not change your data, but also that no one else, even in another thread will. That sounds appealing. After all, who doesn't want the data they are using in a function to not change from underneath them? Suppose you have highly structured, deeply nested reference data you read from a nosql DB. Surely there is opportunity and some benefit to use immutable? The result of the query is just a read only data source. But then that data will be consumed - presumably by functions with either immutable or const parameters. If all signatures use const then when can you benefit from the fact that the data is really immutable (by the time it gets to a function with const parm the fact that it really was/is immutable is lost.

On Wed, Oct 16, 2013 at 09:06:05PM +0200, Daniel Davidson wrote:
> On Wednesday, 16 October 2013 at 18:52:23 UTC, qznc wrote:
[...]
> >Library code:
> >
> >struct Foo { int x; }
> >
> >User code:
> >
> >Foo f;
> >immutable f2 = f;
> >
> >This works, even though the library writer might not have anticipated that someone makes Foo immutable. However, now the library writer obliviously releases a new version of the library, which extends it like this:
> >
> >struct Foo {
> >  int x;
> >  private int[] history;
> >}
> >
> >Unfortunately, now the user code is broken due to the freshly introduced mutable aliasing. Personally, I think is fine. Upon compilation the user code gives a  error message and user developer can adapt to the code to the new library version. Some think the library writer should have a possibility to make this work.
> 
> I don't understand how it could be fine. As code grows it would lead to people not adding useful members like history just because of the huge repercussions.
> 
> struct User {
>    immutable(Foo) foos;
> }
> 
> How can I as a user adapt to that change? Before the change assignment worked equally well among all of Mutable, Immutable, Const. After that change any `foos ~= createFoo(...)` would require change. And it is not clear what the change would be.

The root of the problem is reliance on assignment between mutable / immutable / const. This reliance breaks encapsulation because you're making an assumption about the assignability of a presumedly opaque library type to immutable / const. In D, immutable is *physical* immutability, not logical immutability; by writing immutable(Foo) you're saying that you wish to have physically-immutable instances of Foo. However, whether this is possible depends on the implementation details of Foo, which, if Foo is supposed to be an opaque type, breaks encapsulation. Without knowing how Foo is implemented (and user code shouldn't know that), you can't reliably go around and claim Foo can be made immutable from a mutable instance. The fact that you're relying on Foo being implicitly convertible to immutable(Foo) means you're already depending on implementation details of Foo, and should be prepared to change code when Foo's implementation changes.

If you want to say that User cannot modify the Foo's it contains, you should use const rather than immutable. It is safe to use const because anything is implicitly convertible to const, so it doesn't introduce any reliance upon implementational details of Foo.

If you insist on being able to append to immutable(Foo)[], then you'll need a createFoo method that returns immutable instances of Foo:

	struct User {
		immutable(Foo)[] foos;
	}
	immutable(Foo) createFoo(...) { ... }

	User u;
	u.foos ~= createFoo(...); // now this works

The problem with this, of course, is that it unnecessarily restricts createFoo(): if you want *mutable* instances of Foo, then you can't use this version of createFoo(), but have to create another function that probably does exactly the same thing. So an alternative solution is to use Phobos' assumeUnique template:

	struct User {
		immutable(Foo)[] foos;
	}
	Foo createFoo(...) { ... }

	User u;
	u.foos ~= assumeUnique(createFoo(...));

The assumeUnique template basically does a cast from mutable to immutable, but explicitly documents the purpose of this cast in the code. It places the onus on the user to ensure that the Foo returned by createFoo is actually unique. If not, you break the type system and the immutability guarantee may no longer hold.

To illustrate why adding mutable aliases to Foo *should* break code, consider this:

	/* This is what Foo looked like before:
	struct OriginalFoo {
		int x;
	}
	*/

	/* This is what Foo looks like now */
	struct Foo {
		int x;
		private int[] history;
		void changeHistory() { history[0]++; }
	}

	Foo createFoo(int x) {
		Foo f;
		f.x = x;
		f.history = [1];
	}

	Foo f = createFoo();

	immutable(Foo) g = f; // doesn't compile, but suppose it does
	f.changeHistory();    // oops, g.history has mutated, so it's
	                      // *not* immutable after all

That's why assigning f to g must be made illegal, since it breaks immutability guarantees. OTOH, if you absolutely have to do it, you can document your intent thus:

	Foo f = createFoo();
	immutable(Foo) g = assumeUnique(f);

	// Now if you use f to mutate g, it's your own problem: you
	// claimed that g was unique but actually it isn't. So it's your
	// own fault when your supposedly-immutable Foo mutates.
	// If you *don't* do stupid things, OTOH, this lets your code
	// continue to work when the library writer decides to change
	// Foo's implementation to contain mutable aliases.


T

-- 
Do not reason with the unreasonable; you lose by definition.

On Wed, Oct 16, 2013 at 09:45:09PM +0200, Daniel Davidson wrote:
> On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:
[...]
> >I think any usage of immutable with types/entities not initially designed for immutability is an potential mistake and in that sense it is good that change has broken the user code. Same goes for operating on immutable entity in generic code as if it is a value type without actually checking it via introspection.
> 
> I don't disagree. Now, what does it mean to "initially design for immutability" and what are the guidelines. I thought that was kind of what we were talking about here. How to effectively use const and immutable.

I'd say that user code should use const, not immutable (unless the library provides a way of constructing immutable instances of the type, of course), because immutable makes assumptions about implementation details, which should not be known unless you break encapsulation. As a user of a properly encapsulated type, I can't make any guarantees about its immutability; the best I can do is to promise I don't change it myself -- i.e., const.

Immutable should be used in library code to provide strong guarantees to the user: since you're the one responsible for implementing the type, you're in the position to make guarantees about its uniqueness (and hence, immutability).


[...]
> One general idea that was presented by Ali is that an immutable parameter means that not only are you guaranteeing that you will not change your data, but also that no one else, even in another thread will. That sounds appealing. After all, who doesn't want the data they are using in a function to not change from underneath them?

I'm actually wary of this view, to be honest. An immutable parameter means you expect your *caller* to provide you with a value that cannot be changed, not by you, nor by anybody else, ever. Sure, it's nice to have, but that imposes a rather high bar on your callers. They have to be responsible to guarantee that whatever they hand to you cannot be changed by anything or anyone else, at any time. If they can do this, then great; if not, they won't be able to call your function.


> Suppose you have highly structured, deeply nested reference data you read from a nosql DB. Surely there is opportunity and some benefit to use immutable? The result of the query is just a read only data source. But then that data will be consumed - presumably by functions with either immutable or const parameters. If all signatures use const then when can you benefit from the fact that the data is really immutable (by the time it gets to a function with const parm the fact that it really was/is immutable is lost.

I'm of the view that code should only require the minimum of assumptions it needs to actually work. If your code can work with mutable types, then let it take a mutable (unqualified) type. If your code works without modifying input data, then let it take const. Only if your code absolutely will not work correctly unless the data is guaranteed to never change, ever, by anyone, should it take immutable.

I'm not sure what "benefits" you get from requiring immutable when the code doesn't really need to depend on immutability. It just makes the code harder to use (you have to make sure whatever you pass to it is immutable, and sometimes that's not easy to guarantee, and would require a lot of copying). Immutable only benefits you when the code *can't* work correctly unless the data is guaranteed never to change, ever. Hash table keys come to mind -- if you compute the hash value of the key and use that to determine which slot to put the data into, it would be very bad if somebody else mutated that key via a mutable reference after the fact -- now your AA is broken because the hash value no longer matches the key.  By requiring an immutable key, you ensure that this never happens.

Note that in D, everything is thread-local by default unless explicitly made shared, so using immutable to guarantee other threads won't mutate the data isn't really necessary.


T

-- 
What do you mean the Internet isn't filled with subliminal messages? What about all those buttons marked "submit"??

On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:
> I'm of the view that code should only require the minimum of assumptions
> it needs to actually work. If your code can work with mutable types,
> then let it take a mutable (unqualified) type. If your code works
> without modifying input data, then let it take const. Only if your code
> absolutely will not work correctly unless the data is guaranteed to
> never change, ever, by anyone, should it take immutable.

Sounds reasonable. Which one of the following would you prefer?

foo1(const T t) {
  takes_immutable(t.idup);
}

foo2(immutable(T) t) {
  takes_immutable(t);
}

I'd say foo2, which puts the burden of the copy on the caller. However, if the caller already has an immutable value an unecessary copy is avoided.

On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:
> I'm of the view that code should only require the minimum of assumptions
> it needs to actually work. If your code can work with mutable types,
> then let it take a mutable (unqualified) type. If your code works
> without modifying input data, then let it take const. Only if your code
> absolutely will not work correctly unless the data is guaranteed to
> never change, ever, by anyone, should it take immutable.

What about functions, which are not thread-safe?

auto percentageDistribution(const int[] percentages) {
  [...]
  assert (sum == 100);
  [...]
}

Not a realistic example, but it should do. This function checks that the sum of all percentages equals 100, but a concurrent modification might break this. You could require an immutable array to avoid that. Alternatively, the absence of "shared" could be considered enough to express this.

On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:
> On Wed, Oct 16, 2013 at 09:45:09PM +0200, Daniel Davidson wrote:
>> On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:
> [...]
>> >I think any usage of immutable with types/entities not initially
>> >designed for immutability is an potential mistake and in that
>> >sense it is good that change has broken the user code. Same goes
>> >for operating on immutable entity in generic code as if it is a
>> >value type without actually checking it via introspection.
>> 
>> I don't disagree. Now, what does it mean to "initially design for
>> immutability" and what are the guidelines. I thought that was kind
>> of what we were talking about here. How to effectively use const and
>> immutable.
>

Thanks - this is most helpful.

> I'd say that user code should use const, not immutable (unless the
> library provides a way of constructing immutable instances of the type,
> of course), because immutable makes assumptions about implementation
> details, which should not be known unless you break encapsulation. As a
> user of a properly encapsulated type, I can't make any guarantees about
> its immutability; the best I can do is to promise I don't change it
> myself -- i.e., const.
>

The distinction between user code and the rest (non-user code?) is not enough for me to get clear guidance. I am writing user code, the vast majority will be in libraries because I find that helpful. I would hate to have a separate set of rules for my code and the code I use. We should strive for guidelines that work in general - as in this is a good way to do D.

But leaving "user code" aside, I like the argument and the vote for const over immutable.


> Immutable should be used in library code to provide strong guarantees to
> the user: since you're the one responsible for implementing the type,
> you're in the position to make guarantees about its uniqueness (and
> hence, immutability).
>

I don't see the benefit of separating usage of types, usually via functions where mutability guarantees adorn the types, and implementation guarantees in a world with turtles all the way down. Maybe if those guarantees are totally hidden from user code via encapsulation then the impact of immutable and the difficulties of dealing with it are lessened. But then is the immutable guarantee for the user or just the developer of the encapsulated non-user code?

How about this... here is a plea for ideas on a good use of immutable for a type with potentially mutable aliasing. String I think is a great use of immutable but char will likely never have aliasing introduced to it. At this stage I want the information for educational purposes, because based on this thread and my experimentation - Ali's presentation guidelines aside - I am about to abandon immutable altogether. The fight is too hard and my skills too weakened.

>
> [...]
>> One general idea that was presented by Ali is that an immutable
>> parameter means that not only are you guaranteeing that you will not
>> change your data, but also that no one else, even in another thread
>> will. That sounds appealing. After all, who doesn't want the data
>> they are using in a function to not change from underneath them?
>
> I'm actually wary of this view, to be honest. An immutable parameter
> means you expect your *caller* to provide you with a value that cannot
> be changed, not by you, nor by anybody else, ever. Sure, it's nice to
> have, but that imposes a rather high bar on your callers. They have to
> be responsible to guarantee that whatever they hand to you cannot be
> changed by anything or anyone else, at any time. If they can do this,
> then great; if not, they won't be able to call your function.
>

I now agree with you on this, especially since it goes with my new guideline of don't use immutable.

>
>> Suppose you have highly structured, deeply nested reference data you
>> read from a nosql DB. Surely there is opportunity and some benefit
>> to use immutable? The result of the query is just a read only data
>> source. But then that data will be consumed - presumably by
>> functions with either immutable or const parameters. If all
>> signatures use const then when can you benefit from the fact that
>> the data is really immutable (by the time it gets to a function with
>> const parm the fact that it really was/is immutable is lost.
>
> I'm of the view that code should only require the minimum of assumptions
> it needs to actually work. If your code can work with mutable types,
> then let it take a mutable (unqualified) type. If your code works
> without modifying input data, then let it take const. Only if your code
> absolutely will not work correctly unless the data is guaranteed to
> never change, ever, by anyone, should it take immutable.
>

I don't have the instincts yet to really buy this. Perhaps a specific example would be helpful.

I think most of the time `foo(ref const(T) t)` is written such that it is assumed t is never changed during the span of foo and the compiler helps. While it is possible to, in a single thread call out to code that also has a handle to shared state so t could be accidentally or purposely modified elsewhere, it is probably a rare design goal.

Take the query example. A big mongo db query result comes back and is deserialized into a large web of json like data - lists, dictionaries at many levels. Is that a good reason then for `foo(ref immutable(QueryResult) qr)`. Surely while you are working on query result you don't want it to change. I am ambivalent here because immutable sounds appealing. Any iteration over a hash in the data set must be immutable. I think this is the expectation most have with all const or immutable types taken in a function. Sticking with const just makes life easier.


> I'm not sure what "benefits" you get from requiring immutable when the
> code doesn't really need to depend on immutability. It just makes the
> code harder to use (you have to make sure whatever you pass to it is
> immutable, and sometimes that's not easy to guarantee, and would require
> a lot of copying). Immutable only benefits you when the code *can't*
> work correctly unless the data is guaranteed never to change, ever. Hash
> table keys come to mind -- if you compute the hash value of the key and
> use that to determine which slot to put the data into, it would be very
> bad if somebody else mutated that key via a mutable reference after the
> fact -- now your AA is broken because the hash value no longer matches
> the key.  By requiring an immutable key, you ensure that this never
> happens.
>

I like the AA example - since it sounds like a good use for immutable.

> Note that in D, everything is thread-local by default unless explicitly
> made shared, so using immutable to guarantee other threads won't mutate
> the data isn't really necessary.
>
>
> T

On Thu, Oct 17, 2013 at 08:56:08AM +0200, qznc wrote:
> On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:
> >I'm of the view that code should only require the minimum of assumptions it needs to actually work. If your code can work with mutable types, then let it take a mutable (unqualified) type. If your code works without modifying input data, then let it take const. Only if your code absolutely will not work correctly unless the data is guaranteed to never change, ever, by anyone, should it take immutable.
> 
> Sounds reasonable. Which one of the following would you prefer?
> 
> foo1(const T t) {
>   takes_immutable(t.idup);
> }
> 
> foo2(immutable(T) t) {
>   takes_immutable(t);
> }
> 
> I'd say foo2, which puts the burden of the copy on the caller. However, if the caller already has an immutable value an unecessary copy is avoided.

Well, that's a bit too generic to really decide one way or the other. What do foo1 and foo2 do? I don't think it's fair to say one or the other is superior without more context. It depends on what you're trying to achieve.


T

-- 
Too many people have open minds but closed eyes.

On Thu, Oct 17, 2013 at 09:08:16AM +0200, qznc wrote:
> On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:
> >I'm of the view that code should only require the minimum of assumptions it needs to actually work. If your code can work with mutable types, then let it take a mutable (unqualified) type. If your code works without modifying input data, then let it take const. Only if your code absolutely will not work correctly unless the data is guaranteed to never change, ever, by anyone, should it take immutable.
> 
> What about functions, which are not thread-safe?
> 
> auto percentageDistribution(const int[] percentages) {
>   [...]
>   assert (sum == 100);
>   [...]
> }
> 
> Not a realistic example, but it should do. This function checks that the sum of all percentages equals 100, but a concurrent modification might break this. You could require an immutable array to avoid that. Alternatively, the absence of "shared" could be considered enough to express this.

In D, variables are thread-local by default, and only shared if explicitly stated so. If this function took shared(const(int)[]), or you pass it something that's __gshared, then you might have a point. But in normal D code I'd recommend against doing that.

Currently, D has some issues in the area of shared, requiring a cast to strip shared from the type. So basically the onus is on the caller to ensure that when shared is stripped from the type it's actually safe from concurrent modification. In which case the above code should be OK as-is.


T

-- 
Guns don't kill people. Bullets do.