So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`:
---
string v;
debug string sure;

void f(string s) { v = s; debug sure = s.idup; }
void g() { assert(v == sure); }
---
I also store a copy of `s` into `sure` for my friend to ensure immutable date hasn't been mutated.
Can my friend's assertion ever fail without breaking a type-system?
Sure. Just consider this:
---
void main() {
    auto s = "abba".idup;
    f(s);
    delete s;
    g();
}
---
Is it by-design? Looks like deleting immutable (and const because of implicit conversion) data should be prohibited.
OK. Let `delete` be fixed. Can we still fail?
---
void h() {
    immutable(char)[4] s = "abba";
    f(s);
}
void main() {
    h();
    g();
}
---
Damn! So, what can we do with it? Not sure, but I have a proposal.

Fix it in language:
    * disallow `delete` of const/immutable data
    * disallow immutable data on the stack

This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).

On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
> So, I'm a function `f`, I have an `immutable(type)[]` argument and I
> want to store it for my friend `g` in an TLS variable `v`:
> ---
> string v;
> debug string sure;
>
> void f(string s) { v = s; debug sure = s.idup; }
> void g() { assert(v == sure); }
> ---
> I also store a copy of `s` into `sure` for my friend to ensure immutable
> date hasn't been mutated.
> Can my friend's assertion ever fail without breaking a type-system?
> Sure. Just consider this:
> ---
> void main() {
> auto s = "abba".idup;
> f(s);
> delete s;
> g();
> }
> ---
> Is it by-design? Looks like deleting immutable (and const because of
> implicit conversion) data should be prohibited.
> OK. Let `delete` be fixed. Can we still fail?
> ---
> void h() {
> immutable(char)[4] s = "abba";
> f(s);
> }
> void main() {
> h();
> g();
> }
> ---
> Damn! So, what can we do with it? Not sure, but I have a proposal.
>
> Fix it in language:
> * disallow `delete` of const/immutable data
> * disallow immutable data on the stack
>
> This makes data really immutable if I don't miss something. Anyway, I
> want `immutable` qualified data to be immutable without breaking a
> type-system (if one do it, its his own responsibility), so some changes
> should be made (IMHO).

You are using unsafe language features to break the type system. That is not the fault of the type system.

'@safe:' at the top of the program should stop both examples from working, it is a bug that it does not.

On 01/01/12 13:50, Timon Gehr wrote:
> On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
>> So, I'm a function `f`, I have an `immutable(type)[]` argument and I
>> want to store it for my friend `g` in an TLS variable `v`:
>> ---
>> string v;
>> debug string sure;
>>
>> void f(string s) { v = s; debug sure = s.idup; }
>> void g() { assert(v == sure); }
>> ---
>> I also store a copy of `s` into `sure` for my friend to ensure immutable
>> date hasn't been mutated.
>> Can my friend's assertion ever fail without breaking a type-system?
>> Sure. Just consider this:
>> ---
>> void main() {
>> auto s = "abba".idup;
>> f(s);
>> delete s;
>> g();
>> }
>> ---
>> Is it by-design? Looks like deleting immutable (and const because of
>> implicit conversion) data should be prohibited.
>> OK. Let `delete` be fixed. Can we still fail?
>> ---
>> void h() {
>> immutable(char)[4] s = "abba";
>> f(s);
>> }
>> void main() {
>> h();
>> g();
>> }
>> ---
>> Damn! So, what can we do with it? Not sure, but I have a proposal.
>>
>> Fix it in language:
>> * disallow `delete` of const/immutable data
>> * disallow immutable data on the stack
>>
>> This makes data really immutable if I don't miss something. Anyway, I
>> want `immutable` qualified data to be immutable without breaking a
>> type-system (if one do it, its his own responsibility), so some changes
>> should be made (IMHO).
>
> You are using unsafe language features to break the type system. That is
> not the fault of the type system.
>
> '@safe:' at the top of the program should stop both examples from
> working, it is a bug that it does not.

That's the point -- *which* checks are missing from @safe?
But I'm not sure that you're right, this looks broken to me, even without @safe.

What does it mean to create immutable data on the stack? The stack is intrinsically mutable!
What does it mean to delete immutable data?
I think it's reasonable for both of them to require a cast, even in @system code.

On 10/17/2012 01:49 PM, Don Clugston wrote:
> On 01/01/12 13:50, Timon Gehr wrote:
>> On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
>>> So, I'm a function `f`, I have an `immutable(type)[]` argument and I
>>> want to store it for my friend `g` in an TLS variable `v`:
>>> ---
>>> string v;
>>> debug string sure;
>>>
>>> void f(string s) { v = s; debug sure = s.idup; }
>>> void g() { assert(v == sure); }
>>> ---
>>> I also store a copy of `s` into `sure` for my friend to ensure immutable
>>> date hasn't been mutated.
>>> Can my friend's assertion ever fail without breaking a type-system?
>>> Sure. Just consider this:
>>> ---
>>> void main() {
>>> auto s = "abba".idup;
>>>     f(s);
>>>     delete s;
>>>     g();
>>> }
>>> ---
>>> Is it by-design? Looks like deleting immutable (and const because of
>>> implicit conversion) data should be prohibited.
>>> OK. Let `delete` be fixed. Can we still fail?
>>> ---
>>> void h() {
>>>     immutable(char)[4] s = "abba";
>>>     f(s);
>>> }
>>> void main() {
>>>     h();
>>>     g();
>>> }
>>> ---
>>> Damn! So, what can we do with it? Not sure, but I have a proposal.
>>>
>>> Fix it in language:
>>> * disallow `delete` of const/immutable data
>>> * disallow immutable data on the stack
>>>
>>> This makes data really immutable if I don't miss something. Anyway, I
>>> want `immutable` qualified data to be immutable without breaking a
>>> type-system (if one do it, its his own responsibility), so some changes
>>> should be made (IMHO).
>>
>> You are using unsafe language features to break the type system. That is
>> not the fault of the type system.
>>
>> '@safe:' at the top of the program should stop both examples from
>> working, it is a bug that it does not.
>
> That's the point -- *which* checks are missing from @safe?

Escaping stack data and arbitrarily freeing memory are not operations
found in memory safe languages.

> But I'm not sure that you're right, this looks broken to me, even
> without @safe.
>
> What does it mean to create immutable data on the stack? The stack is
> intrinsically mutable!

So is the heap.

What does it mean to garbage collect immutable data?

What does it mean to allocate an 'int' on the stack?

> What does it mean to delete immutable data?

Deallocate the storage for it and make it available for reuse.
Accessing it afterwards leads to arbitrary behaviour. This is the same
with mutable data. As the program may behave arbitrarily in this case,
it is valid behaviour to act as if immutable data changed.

> I think it's reasonable for both of them to require a cast, even in
> @system code.
>

The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping. Rust's borrowed pointers may give some hints on how
to extend 'scope' to fields of structs.

As to delete, delete is as unsafe when the involved data is immutable
as when it is mutable. Why require an additional cast in one case?

The issue is that you're thinking as you would in Java.

I guess the rule in D for immutable is this: Immutable data won't change as long as it exists.

The last part of that sentence would be a stupid thing to say in Java because things don't just cease to exist while you're still doing something with them. That is not the case in D.

That being said it's very unlikely that you will ever run into this situation. You have to end the lifetime of the object manually to run into these issues. And in that case it'll be very easy to figure out what's wrong.

On 17/10/12 18:02, Timon Gehr wrote:
> On 10/17/2012 01:49 PM, Don Clugston wrote:
>> On 01/01/12 13:50, Timon Gehr wrote:
>>> On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
>>>> So, I'm a function `f`, I have an `immutable(type)[]` argument and I
>>>> want to store it for my friend `g` in an TLS variable `v`:
>>>> ---
>>>> string v;
>>>> debug string sure;
>>>>
>>>> void f(string s) { v = s; debug sure = s.idup; }
>>>> void g() { assert(v == sure); }
>>>> ---
>>>> I also store a copy of `s` into `sure` for my friend to ensure
>>>> immutable
>>>> date hasn't been mutated.
>>>> Can my friend's assertion ever fail without breaking a type-system?
>>>> Sure. Just consider this:
>>>> ---
>>>> void main() {
>>>> auto s = "abba".idup;
>>>>     f(s);
>>>>     delete s;
>>>>     g();
>>>> }
>>>> ---
>>>> Is it by-design? Looks like deleting immutable (and const because of
>>>> implicit conversion) data should be prohibited.
>>>> OK. Let `delete` be fixed. Can we still fail?
>>>> ---
>>>> void h() {
>>>>     immutable(char)[4] s = "abba";
>>>>     f(s);
>>>> }
>>>> void main() {
>>>>     h();
>>>>     g();
>>>> }
>>>> ---
>>>> Damn! So, what can we do with it? Not sure, but I have a proposal.
>>>>
>>>> Fix it in language:
>>>> * disallow `delete` of const/immutable data
>>>> * disallow immutable data on the stack
>>>>
>>>> This makes data really immutable if I don't miss something. Anyway, I
>>>> want `immutable` qualified data to be immutable without breaking a
>>>> type-system (if one do it, its his own responsibility), so some changes
>>>> should be made (IMHO).
>>>
>>> You are using unsafe language features to break the type system. That is
>>> not the fault of the type system.
>>>
>>> '@safe:' at the top of the program should stop both examples from
>>> working, it is a bug that it does not.
>>
>> That's the point -- *which* checks are missing from @safe?
>
> Escaping stack data and arbitrarily freeing memory are not operations
> found in memory safe languages.

HOW do you propose to check for escaping stack data?

>> But I'm not sure that you're right, this looks broken to me, even
>> without @safe.
>>
>> What does it mean to create immutable data on the stack? The stack is
>> intrinsically mutable!
>
> So is the heap.

No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.

> What does it mean to garbage collect immutable data?

From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.

> What does it mean to allocate an 'int' on the stack?
>
>> What does it mean to delete immutable data?
>
> Deallocate the storage for it and make it available for reuse.
> Accessing it afterwards leads to arbitrary behaviour. This is the same
> with mutable data. As the program may behave arbitrarily in this case,
> it is valid behaviour to act as if immutable data changed.

No, you've broken the type system if you've deleted immutable data.
If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee.

With a mutable variable, I have no such guarantee. (It's not safe to allocate something different in the deleted location, but it's OK to run the finalizer and then wipe all the memory).

>> I think it's reasonable for both of them to require a cast, even in
>> @system code.
>>
>
> The implementation of the 'scope' storage class should be fixed. We
> could then require an unsafe cast(scope) to disable prevention of stack
> address escaping.

No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?

void foo(bool b, string y)
{
  immutable (char)[4] x = "abba";
  string s = b ? x : y;
  f(s);
}

Make it safe.


> Rust's borrowed pointers may give some hints on how
> to extend 'scope' to fields of structs.

I think it is more fundamental than that.

> As to delete, delete is as unsafe when the involved data is immutable
> as when it is mutable. Why require an additional cast in one case?

This is not about safety.
Modifying immutable data breaks the type system. Deleting mutable data does not. AFAIK it is safe to implement delete as a call to the finalizer, followed by setting the memory to T.init. Only the GC can determine if it is safe to reuse the memory.

Deleting immutable data just doesn't make sense.

On 10/18/12 10:08, Don Clugston wrote:
> On 17/10/12 18:02, Timon Gehr wrote:
>> On 10/17/2012 01:49 PM, Don Clugston wrote:
>>> On 01/01/12 13:50, Timon Gehr wrote:
>>>> On 01/01/2012 10:40 AM, Denis Shelomovskij wrote:
>>>>> So, I'm a function `f`, I have an `immutable(type)[]` argument and I want to store it for my friend `g` in an TLS variable `v`:
>>>>> ---
>>>>> string v;
>>>>> debug string sure;
>>>>>
>>>>> void f(string s) { v = s; debug sure = s.idup; }
>>>>> void g() { assert(v == sure); }
>>>>> ---
>>>>> I also store a copy of `s` into `sure` for my friend to ensure
>>>>> immutable
>>>>> date hasn't been mutated.
>>>>> Can my friend's assertion ever fail without breaking a type-system?
>>>>> Sure. Just consider this:
>>>>> ---
>>>>> void main() {
>>>>> auto s = "abba".idup;
>>>>>     f(s);
>>>>>     delete s;
>>>>>     g();
>>>>> }
>>>>> ---
>>>>> Is it by-design? Looks like deleting immutable (and const because of
>>>>> implicit conversion) data should be prohibited.
>>>>> OK. Let `delete` be fixed. Can we still fail?
>>>>> ---
>>>>> void h() {
>>>>>     immutable(char)[4] s = "abba";
>>>>>     f(s);
>>>>> }
>>>>> void main() {
>>>>>     h();
>>>>>     g();
>>>>> }
>>>>> ---
>>>>> Damn! So, what can we do with it? Not sure, but I have a proposal.
>>>>>
>>>>> Fix it in language:
>>>>> * disallow `delete` of const/immutable data
>>>>> * disallow immutable data on the stack
>>>>>
>>>>> This makes data really immutable if I don't miss something. Anyway, I want `immutable` qualified data to be immutable without breaking a type-system (if one do it, its his own responsibility), so some changes should be made (IMHO).
>>>>
>>>> You are using unsafe language features to break the type system. That is not the fault of the type system.
>>>>
>>>> '@safe:' at the top of the program should stop both examples from working, it is a bug that it does not.
>>>
>>> That's the point -- *which* checks are missing from @safe?
>>
>> Escaping stack data and arbitrarily freeing memory are not operations found in memory safe languages.
> 
> HOW do you propose to check for escaping stack data?

/How/ is not a problem (ignoring implementation costs), the /language definition/
part is trickier - you need a very precise definition of what is allowed and what
isn't; otherwise different compilers will make different decisions and every
compiler will support only a vendor-specific non-std dialect...
(eg storing a scoped-ref into some kind of container, passing that down to other
functions could work, but what if you then need to let the container escape and
want to do that by removing the scoped-ref? It might be possible for the compiler
to prove that it's safe, but it's unlikely that every compiler will act the same)

>>> But I'm not sure that you're right, this looks broken to me, even without @safe.
>>>
>>> What does it mean to create immutable data on the stack? The stack is intrinsically mutable!
>>
>> So is the heap.
> 
> No it is not. Data on the stack *cannot* survive past the end of the function call. Data on the heap can last forever.

Lifetime and mutability are different things.

>> What does it mean to garbage collect immutable data?
> 
> From the point of view of the application, it doesn't happen. There are no observable semantics. It's merely an implementation detail.
> 
>> What does it mean to allocate an 'int' on the stack?
>>
>>> What does it mean to delete immutable data?
>>
>> Deallocate the storage for it and make it available for reuse. Accessing it afterwards leads to arbitrary behaviour. This is the same with mutable data. As the program may behave arbitrarily in this case, it is valid behaviour to act as if immutable data changed.
> 
> No, you've broken the type system if you've deleted immutable data.
> If I have a reference to an immutable variable, I have a guarantee that it will never change. delete will break that guarantee.

Yes. The alternative (to allow explicit delete on immutable data) would likely be too complicated to be worth implementing in the near future - you need to ensure the data is unique, there are no other refs to it, and forbid accessing it after the 'delete' op. I guess an easy way out would be to ask the GC to run a collect cycle and return back whether an object was successfully collected. But i can't really see a useful application for it, and you'd need a special convention, as the GC would have to given the last ref to the object.


> With a mutable variable, I have no such guarantee. (It's not safe to allocate something different in the deleted location, but it's OK to run the finalizer and then wipe all the memory).
> 
>>> I think it's reasonable for both of them to require a cast, even in @system code.
>>>
>>
>> The implementation of the 'scope' storage class should be fixed. We could then require an unsafe cast(scope) to disable prevention of stack address escaping.
> 
> No we can't. f cannot know that the string it has been given is on the stack. So main() must prevent it from being given to f() in the first place. How can it do that?
> 
> void foo(bool b, string y)
> {
>   immutable (char)[4] x = "abba";
>   string s = b ? x : y;
>   f(s);
> }
> 
> Make it safe.

Trivial.

   void f(scope string s) { /*v = s; # now illegal */ debug sure = s.idup; }

Obviously, it would need a properly working 'scope'. /Then/ making
it illegal to pass refs to local (or otherwise) scoped object in a
way that would allow them to escape would work.
Right now, when 'scope' is just decoration, trying to enforce such
restrictions would only create more bugs and confusion, I'm afraid.

The other reason why a working 'scope' is important is so that

   void g(string s) { f(s~"xyz"); }

does not cause heap allocation unless necessary. Even when the allocation is necessary the new object can be reused and the given back to the allocator when it's really dead. Reducing the amount of garbage left behind is the best way to optimize the GC...

[of course 'scope' should have been the default etc etc, but that's not really D2 material]

artur

On 10/18/2012 10:08 AM, Don Clugston wrote:
> On 17/10/12 18:02, Timon Gehr wrote:
>> On 10/17/2012 01:49 PM, Don Clugston wrote:
>>> ...
>>>
>>> That's the point -- *which* checks are missing from @safe?
>>
>> Escaping stack data and arbitrarily freeing memory are not operations
>> found in memory safe languages.
>
> HOW do you propose to check for escaping stack data?
>

Static escape analysis. Use the 'scope' qualifier to designate
data that is not allowed to be escaped in order to make it modular.

> ...
>>
>> The implementation of the 'scope' storage class should be fixed. We
>> could then require an unsafe cast(scope) to disable prevention of stack
>> address escaping.
>
> No we can't. f cannot know that the string it has been given is on the
> stack. So main() must prevent it from being given to f() in the first
> place. How can it do that?
>

f can know that it mustn't escape it, which is enough.

> void foo(bool b, string y)
> {
>    immutable (char)[4] x = "abba";
>    string s = b ? x : y;
>    f(s);
> }
>
> Make it safe.
>

It is safe if the parameter to f is marked with 'scope'. (and this in
turn obliges f not to escape it.)

Analyze scope on the expression level.

The analysis would determine that x[] is 'scope'. It would
conservatively propagate this fact to (b ? x[] : y). Then the local
variable 's' will get the 'scope' storage class.

In general, use a fixed-point iteration to determine all local
variables that might refer to scope'd data and prevent that they get
escaped.

>
>> Rust's borrowed pointers may give some hints on how
>> to extend 'scope' to fields of structs.
>
> I think it is more fundamental than that.
>
>> As to delete, delete is as unsafe when the involved data is immutable
>> as when it is mutable. Why require an additional cast in one case?
>
> This is not about safety.
> Modifying immutable data breaks the type system. Deleting mutable data
> does not.
> AFAIK it is safe to implement delete as a call to the
> finalizer, followed by setting the memory to T.init.
> ...


Now I see where you are coming from. This is indeed a safe approach for
references to/arrays of fully mutable value types, but not for delete
in general.

Make sure to treat void* specially though.

struct S{ immutable int x; this(int x){this.x=x;}}

void main()@safe{
    void* s = new S(2);
    delete s;
}

Class instance memory does not have a T.init, because it is not
assigned a T. And even if it was, how would you know at compile time if
the bound instance has any immutable fields?
Should that be a runtime exception?

On 18/10/12 19:43, Timon Gehr wrote:
> On 10/18/2012 10:08 AM, Don Clugston wrote:
>> On 17/10/12 18:02, Timon Gehr wrote:
>>> On 10/17/2012 01:49 PM, Don Clugston wrote:
>>>> ...
>>>>
>>>> That's the point -- *which* checks are missing from @safe?
>>>
>>> Escaping stack data and arbitrarily freeing memory are not operations
>>> found in memory safe languages.
>>
>> HOW do you propose to check for escaping stack data?
>>
>
> Static escape analysis. Use the 'scope' qualifier to designate
> data that is not allowed to be escaped in order to make it modular.
>
>> ...
>>>
>>> The implementation of the 'scope' storage class should be fixed. We
>>> could then require an unsafe cast(scope) to disable prevention of stack
>>> address escaping.
>>
>> No we can't. f cannot know that the string it has been given is on the
>> stack. So main() must prevent it from being given to f() in the first
>> place. How can it do that?
>>
>
> f can know that it mustn't escape it, which is enough.
>
>> void foo(bool b, string y)
>> {
>>    immutable (char)[4] x = "abba";
>>    string s = b ? x : y;
>>    f(s);
>> }
>>
>> Make it safe.
>>
>
> It is safe if the parameter to f is marked with 'scope'. (and this in
> turn obliges f not to escape it.)

Well, OK, but that involves changing the semantics of immutable. You could not pass this kind of "local immutable" to _any_ existing code.
It would render almost all existing code that uses immutable obsolete.

And what do you get in exchange? Practically nothing!

>
> Analyze scope on the expression level.
>
> The analysis would determine that x[] is 'scope'. It would
> conservatively propagate this fact to (b ? x[] : y). Then the local
> variable 's' will get the 'scope' storage class.
>
> In general, use a fixed-point iteration to determine all local
> variables that might refer to scope'd data and prevent that they get
> escaped.
>
>>
>>> Rust's borrowed pointers may give some hints on how
>>> to extend 'scope' to fields of structs.
>>
>> I think it is more fundamental than that.
>>
>>> As to delete, delete is as unsafe when the involved data is immutable
>>> as when it is mutable. Why require an additional cast in one case?
>>
>> This is not about safety.
>> Modifying immutable data breaks the type system. Deleting mutable data
>> does not.
>> AFAIK it is safe to implement delete as a call to the
>> finalizer, followed by setting the memory to T.init.
>> ...
>
>
> Now I see where you are coming from. This is indeed a safe approach for
> references to/arrays of fully mutable value types, but not for delete
> in general.
>
> Make sure to treat void* specially though.
>
> struct S{ immutable int x; this(int x){this.x=x;}}
>
> void main()@safe{
>      void* s = new S(2);
>      delete s;
> }
>
> Class instance memory does not have a T.init, because it is not
> assigned a T. And even if it was, how would you know at compile time if
> the bound instance has any immutable fields?
> Should that be a runtime exception?

Probably. Yeah, it's a bit hard if you have a base class, you can't statically check if it has immutable members in a derived class.
Or you could be conservative and disallow delete of anything where you don't know the exact type at compile time.