What if D2 were to introduce a keyword that allows you to decouple a class member from the class state for the purposes of const?  This keyword basically means that the member is stored with the class data but is unaffected by the constancy of an instance.  In other words, a non-state member variable is not cast to const when the class instance is cast to const.  This is an implementation of the 'is associated with' OO relationship.  This is similar to the 'mutable' keyword in C++, or the way mutexes work in D.

If this were to happen, would you consider this to be a positive, negative, or inconsequential change?

Would you stop using D because of it?

Would you switch to D2 because of it?

Would you consider this concept hard to explain or understand?

On 2008-05-29 18:41:21 +0200, "Steven Schveighoffer" <schveiguy@yahoo.com> said:

> What if D2 were to introduce a keyword that allows you to decouple a class
> member from the class state for the purposes of const?  This keyword
> basically means that the member is stored with the class data but is
> unaffected by the constancy of an instance.  In other words, a non-state
> member variable is not cast to const when the class instance is cast to
> const.  This is an implementation of the 'is associated with' OO
> relationship.  This is similar to the 'mutable' keyword in C++, or the way
> mutexes work in D.
> 
> If this were to happen, would you consider this to be a positive, negative,
> or inconsequential change?

I already expressed my view about it in
	http://www.wikiservice.at/d/wiki.cgi?TransitiveConst

I think that it is a good idea, but I think that its name should be much uglier unsafe_mutable, and should be seen by everybody in the community as a bad, and not to be used.

Basically it is an unsafe hole that allows to implement nice an useful stuff (suspended values(lazy eval & cache), dataflow variables, memoization, performace/statistical information), but can break invariant and introduce subtle bugs.

> Would you stop using D because of it?
no
> Would you switch to D2 because of it?
no
> Would you consider this concept hard to explain or understand?
no, but hard to use correctly if it really breaks the invariance, and not much useful if it does not break invariance.

Fawzi

"Steven Schveighoffer" <schveiguy@yahoo.com> wrote in message news:g1mmbh$ftq$1@digitalmars.com...
> What if D2 were to introduce a keyword that allows you to decouple a class member from the class state for the purposes of const?  This keyword basically means that the member is stored with the class data but is unaffected by the constancy of an instance.  In other words, a non-state member variable is not cast to const when the class instance is cast to const.  This is an implementation of the 'is associated with' OO relationship.  This is similar to the 'mutable' keyword in C++, or the way mutexes work in D.
>
> If this were to happen, would you consider this to be a positive, negative, or inconsequential change?

This is certainly a feature I've felt the need for in the past, and caching to me is the obvious use-case.

Having said that, I recall arguments to the effect that there are better ways to implement such things that don't require the use of such a keyword and prevent the need to "break" the transitive-const model.

Given this, my current stance is that it should be left out unless it can be demonstrated that there is a useful design that cannot be implemented without it, and I will be interested to see if I continue to feel the need for it when optimising code in future.

On 29/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
> but can break invariant

That part is not correct. A couple of months back, I submitted a paper to Walter and the gang, based on these ideas, in which I showed that the transitive closure of const/invariant remained fully intact under this model. Andrei confirmed that I was correct, and subsequently tried to help explain it to Walter.

> and introduce subtle bugs.

That part is probably true, but it depends on whether or not people "get it", which I think is what Steven was asking. Consider the following (legal) code

    struct A
    {
        static int n;
    }

    invariant A a;
    a.n = 42;

I don't think anyone would argue that "static breaks invariant", but could you say that code like the above "introduces subtle bugs"? Maybe. Maybe not. It depends on understanding.

"nonstate" has been known by other names in previous discussions, including "unpaintable". (There was quite a long thread with that one). The general feeling over at Phobos is that the number of use-cases is small enough that a handful of library solutions could eliminate the need for the keyword. The problem, as I see it, is that those library solutions have been talked about, but not yet implemented. Consequently, there is still a perceived need for "nonstate" - a perceived need which would go away as soon as those alternative solutions were made available.

On 2008-05-30 08:08:35 +0200, "Janice Caron" <caron800@googlemail.com> said:

> On 29/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
>> but can break invariant
> 
> That part is not correct. A couple of months back, I submitted a paper
> to Walter and the gang, based on these ideas, in which I showed that
> the transitive closure of const/invariant remained fully intact under
> this model. Andrei confirmed that I was correct, and subsequently
> tried to help explain it to Walter.

Indeed there are two ways to introduce mutable space.
The first one that gives access to the mutable space also to pure functions (but potentially breaks everything).
The catch is that by default updates to this mutable state might get lost due to compiler optimizations (as the compiler is free to float pure functions on them out of a loop, or to copy invariant objects), and this should be considered ok (which means that the result of the program cannot depend on this mutable state).
This can be used to implement efficiently memoization and lazy caching also for pure functions (which I want).

The other possibility (that Steven and you propose, if I understood correctly) is to disallow pure functions to access mutable state. This diminished the usefulness of pure functions or mutable state depending from how you look at the problem.
Indeed this solution does not break  invariance.

If implicit automatic copies (without calling a copy method) of object or part of it are allowed then the picture becomes more complex, but such a thing has a questionable value.

>> and introduce subtle bugs.
> 
> That part is probably true, but it depends on whether or not people
> "get it", which I think is what Steven was asking. Consider the
> following (legal) code
> 
>     struct A
>     {
>         static int n;
>     }
> 
>     invariant A a;
>     a.n = 42;
> 
> I don't think anyone would argue that "static breaks invariant", but
> could you say that code like the above "introduces subtle bugs"?
> Maybe. Maybe not. It depends on understanding.
> 
> "nonstate" has been known by other names in previous discussions,
> including "unpaintable". (There was quite a long thread with that
> one). The general feeling over at Phobos is that the number of
> use-cases is small enough that a handful of library solutions could
> eliminate the need for the keyword. The problem, as I see it, is that
> those library solutions have been talked about, but not yet
> implemented. Consequently, there is still a perceived need for
> "nonstate" - a perceived need which would go away as soon as those
> alternative solutions were made available.

I agree that a good library implementation is what one should strive to-
The question is how to implement them.
Without keyword one could achieve this using fully unsafe casts, I think, but is something like that allowed in a pure function?
If the answer is no, then to be able to implement them in plain D in a library the keyword would still be needed, or a way to declare an unsafe function pure (basically haskell unsafePerformIO).

On 30/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
>  The other possibility (that Steven and you propose, if I understood
> correctly) is to disallow pure functions to access mutable state.

As currently planned, inputs and outputs to pure functions must be invariant (or implicitly castable to invariant, e.g. int). You can use mutable data inside the function, but the lifetime of that data is the scope of the function - it will not persist beyond it.

> Without keyword one could achieve [a good library implementation] using fully unsafe casts, I think, but is something like that allowed in a pure function?

I think not. Also, caching could be achieved by a library implementation without any unsafe casts at all - but it still wouldn't be allowed in a pure function because it modifies global state.

If you think about it logically, Steven's proposed keyword "nostate" is a good way of looking at it. A "nostate" variable is not part of the class's state. It is well known that pure functions cannot access global state, but more generally, pure functions cannot access anything which is not part of the state of its inputs. "nostate" variables would be, by definition, not part of the objects' state, and therefore inaccessible to pure functions, same as static member variables.


> If the answer is no, then to be able to implement them in plain D in a library the keyword would still be needed, or a way to declare an unsafe function pure (basically haskell unsafePerformIO).

I don't think I've completely understood what you're trying to get at. Saying "basically haskell unsafePerformIO" tells me nothing, because I don't speak Haskell and have no idea what that means.

One of the advantages of pure functions, however, is that (like inlining) memoization can be done /by the compiler/, so you don't have to do it yourself. How good a job the compiler will do remains to be seen. My guess would be, poor at first, getting increasingly better over time.

On 2008-05-31 07:40:56 +0200, "Janice Caron" <caron800@googlemail.com> said:

> On 30/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
[...]
>> If the answer is no, then to be able to implement them in plain D in a library
>> the keyword would still be needed, or a way to declare an unsafe function pure
>> (basically haskell unsafePerformIO).
> 
> I don't think I've completely understood what you're trying to get at.
> Saying "basically haskell unsafePerformIO" tells me nothing, because I
> don't speak Haskell and have no idea what that means.

Sorry , basically I want a hack to convince the compiler that even if a function looks not pure it actually is.
Maybe a cast, like this (or with an uglier name)
pure int function(T) x=cast(pure int function(T))&y;
basically it says, to the compiler "yes I was careful and you can treat this function as pure, just do it, an if there are bug they are my fault because I am doing something potentially unsafe".

If there is something like this then no_state becomes useful, otherwise t is much less useful.

> One of the advantages of pure functions, however, is that (like
> inlining) memoization can be done /by the compiler/, so you don't have
> to do it yourself. How good a job the compiler will do remains to be
> seen. My guess would be, poor at first, getting increasingly better
> over time.

Sorry that is not good enough for me I want functions lifted out of inner loops, and a compiler that says to me either you use the slow pure function, and I will lift it out of the loop, or the fast one, but it will stay in the loop, is not good enough for me.
I want a hole to be able to trick the compiler into floating out the fast function.
In haskell this hole is called unsafePerformIO, and I think that it is a good name because when you use it it makes it very clear that you are on your own on slippery soil... :)

Fawzi

On 31/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
>  Sorry , basically I want a hack to convince the compiler that even if a
> function looks not pure it actually is.

The one example of this which I can recall, which has been demonstrated in this forum, is calling a "helper function" from within a pure function. Is that what you're talking about? If so, it's been mentioned before. Here's an example:

    void helper(char[] s) // not pure
    {
        s[0] = 'J';
    }

    string foo(string s) pure
    {
        char[] tmp = s.dup;
        helper(tmp);
        return assumeUnique(tmp);
    }

Currently, it seems that the above might not be legal code, since pure functions cannot call non-pure functions. However, what makes this case (and others like it) special is that, although helper itself is not pure, if it were inlined, it wouldn't stop foo from being pure.

This has been pointed out before, and I am /sure/ that Walter will come up with a solution which makes it possible (though not necessarily in the first attempt).

I have absolutely no idea, however, what this has to do with Steven's question. Is it relevant, or are we heading off-topic?



>  Maybe a cast, like this (or with an uglier name)
>  pure int function(T) x=cast(pure int function(T))&y;

The way I read it, the following will work, as is already legal syntax

    auto x = cast(function int(T) pure)&y;


> > One of the advantages of pure functions, however, is that (like inlining) memoization can be done /by the compiler/, so you don't have to do it yourself. How good a job the compiler will do remains to be seen. My guess would be, poor at first, getting increasingly better over time.
>
>  Sorry that is not good enough for me I want functions lifted out of inner
> loops, and a compiler that says to me either you use the slow pure function,
> and I will lift it out of the loop, or the fast one, but it will stay in the
> loop, is not good enough for me.
>  I want a hole to be able to trick the compiler into floating out the fast
> function.

I still don't get it. Can you show me what you mean with an example?

On 2008-05-31 11:37:44 +0200, "Janice Caron" <caron800@googlemail.com> said:

> On 31/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
>> Sorry , basically I want a hack to convince the compiler that even if a
>> function looks not pure it actually is.
> 
> The one example of this which I can recall, which has been
> demonstrated in this forum, is calling a "helper function" from within
> a pure function. Is that what you're talking about? If so, it's been
> mentioned before. Here's an example:
> 
>     void helper(char[] s) // not pure
>     {
>         s[0] = 'J';
>     }
> 
>     string foo(string s) pure
>     {
>         char[] tmp = s.dup;
>         helper(tmp);
>         return assumeUnique(tmp);
>     }
> 
> Currently, it seems that the above might not be legal code, since pure
> functions cannot call non-pure functions. However, what makes this
> case (and others like it) special is that, although helper itself is
> not pure, if it were inlined, it wouldn't stop foo from being pure.
> 
> This has been pointed out before, and I am /sure/ that Walter will
> come up with a solution which makes it possible (though not
> necessarily in the first attempt).
> 
> I have absolutely no idea, however, what this has to do with Steven's
> question. Is it relevant, or are we heading off-topic?

but yes with this we are heading off-topic, what I meant is
1) I want a loophole to be able to access this no state mutable state also in pure functions (even if I know that in general this should not be allowed as it potentially breaks everything)

2) if there is a way to convince the compiler that a non pure function should be treated as a pure function (and this is what I was after before) then the "safe" no_state can be used to achieve 1).

If the following works:
>> Maybe a cast, like this (or with an uglier name)
>> pure int function(T) x=cast(pure int function(T))&y;
> 
> The way I read it, the following will work, as is already legal syntax
> 
>     auto x = cast(function int(T) pure)&y;

then basically we have 2)
int potentiallyUnsafe(T x){ ...}

pure int ITellYouItIsSafe(T x){
 return (cast(function int(T) pure)&potentiallyUnsafe)(x);
}

(I am surprised at how harmlessly it looks)
and adding no_state is equivalent with the more powerful (and dangerous) version that I want.

>>> One of the advantages of pure functions, however, is that (like
>>> inlining) memoization can be done /by the compiler/, so you don't have
>>> to do it yourself. How good a job the compiler will do remains to be
>>> seen. My guess would be, poor at first, getting increasingly better
>>> over time.
>> 
>> Sorry that is not good enough for me I want functions lifted out of inner
>> loops, and a compiler that says to me either you use the slow pure function,
>> and I will lift it out of the loop, or the fast one, but it will stay in the
>> loop, is not good enough for me.
>> I want a hole to be able to trick the compiler into floating out the fast
>> function.
> 
> I still don't get it. Can you show me what you mean with an example?

This is like a generalization of the array indexing example

starting with a code like this:
 void g(int i, const Pippo x, Pippo y){
	auto t=pureF(x);
	y.doSomething(t,i);
 }

then this loop
 for (int i=0;i<100;i++)
   g(i,x,y);
should be transformed as follow:
inline g:
 for (int i=0;i<100;i++){
	auto t=pureF(x);
	y.doSomething(t,i);
}

move pureF(x) outside the loop:
 auto t=pureF(x);
 for (int i=0;i<100;i++){
	y.doSomething(t,i);
}

This last operation is valid *only* if pureF is pure.
Now imagine that the previous fragment is called several times, and that memoization in pureF can improve much its performance.
Then I either have to choose to use memoization (but then the function is not pure anymore for D, and it stays in the loop) or avoid memoization, and the function is pure but then pay the penalty later when the previous code snipped is calculated several times.
I want it all, I want it to be possible to have memoization a purity related optimizations.

It seems that with a safe no_state and a cast this is possible, so yes I very much want no_state.

Fawzi

On 31/05/2008, Fawzi Mohamed <fmohamed@mac.com> wrote:
>  It seems that with a safe no_state and a cast this is possible, so yes I
> very much want no_state.

But it's likely also to be possible /without/ nostate.

As I noted in my first post on this thread, only a small handful of use-cases for nostate (aka mutable, aka unpaintable, etc) have been identified. One of these uses is caching (aka memoization). The current plan is to make caching available as a library solution, and so *therefore*, you will not need "nostate" in order to do this.

I have no idea whether or not the library solution will be considered pure. I imagine that if it can be proven safe, then it will count as pure. If not, well, there's always explicit casting.

So, given that, do you /still/ want nostate? Remember that (like explicit casts and unions) it opens the door to /serious/ abuse of the const system, just like in C++. Inexperienced programmers may well think to themselves: "I can't get this program to compile, unless I make member x mutable, so I'll do that". It seems to me that instead of opening that door, a better solution is (1) identify all reasonable use-cases (and we've only managed to come up with three or four so far, so even if we've missed a few, it's clearly not a big number), and then (2) provide alternative solutions for each of those use-cases.

What do you think?