Jonathan M Davis:

> There will be a library solution to do it, but again, it's unsafe.

It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.

Bye,
bearophile

On Sunday 19 December 2010 14:26:19 bearophile wrote:
> Jonathan M Davis:
> > There will be a library solution to do it, but again, it's unsafe.
> 
> It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.

Whereas, I would argue that it's completely unnecessary. structs and classes serve different purposes. There is no need for scoped classes. They may perodically be useful, but on the whole, they're completely unnecessary.

The compiler can help, but it can't fix the problem any more that it can guarantee that a pointer to a local variable doesn't escape once you've passed it to another function. In _some_ circumstances, it can catch escaping pointers and references, but in the general case, it can't.

If we have library solutions for people who want to play with fire, that's fine. But scoped classes is just not one of those things that the language really needs. They complicate things unnecessarily for minimal benefit.

- Jonathan M Davis

On Sun, 19 Dec 2010 14:38:17 -0800
Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Sunday 19 December 2010 14:26:19 bearophile wrote:
> > Jonathan M Davis:
> > > There will be a library solution to do it, but again, it's unsafe.
> > 
> > It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.
> 
> Whereas, I would argue that it's completely unnecessary. structs and classes serve different purposes. There is no need for scoped classes. They may perodically be useful, but on the whole, they're completely unnecessary.

How do you define a need for scoped classes?

I see two aspects there: first is having destructor called at known point rather than arbitrarily, second is performance.

It looks perfectly reasonable for me to want to have first. Do you know why things which makes program act more deterministic are shunned from D? Does it really add so much to safety that it is worth reducing programmer's control?

Second is irrelevant to types, if I want to speed up some code and can do it by placing things on stack rather than in heap -- what does it matter if it is class or struct or integer? We have alloca() anyway, so removing modifier won't save me from myself, just push me to more C-like code. Which kind of defeats the purpose of using D.

> The compiler can help, but it can't fix the problem any more that it can guarantee that a pointer to a local variable doesn't escape once you've passed it to another function.

Absolutely. Yet we won't have library solution for pointers instead of language support (hopefully)? :) I think it all goes against "being practical" as an objective of the language. Safety is important but you don't achieve safety by means of making unsafe thing unconvenient and inefficient. If there is emplace() then there is no reason not to have scope storage class. At least looking from user's POV. I don't know how hard it is on the compiler.

> In _some_ circumstances, it can catch escaping pointers and references, but in the general case, it can't.

In _general_ case there is no safety in D. With all low-level capabilities one can always defeat compiler. Removing intermediate-level safer (yet unsafe) capabilities arguabily gains nothing but frustration. I'm all for encouraging good practices, but this is different.

-- 
Nick Voronin <elfy.nv@gmail.com>

On 12/20/10, Nick Voronin <elfy.nv@gmail.com> wrote:
> I see two aspects there: first is having destructor called at known point rather than arbitrarily, second is performance.
>

There's still an alternative for the first part, scope(exit):

import std.stdio;

class A
{
    ~this()
    {
        writeln("A.dtor");
    }
    void test()
    {
        writeln("test");
    }
}

void main()
{
    A a = new A();
    scope(exit)
    {
        clear(a);
    }
    a.test();
}

> test
> A.dtor
> A.dtor

On Sunday 19 December 2010 16:50:34 Nick Voronin wrote:
> On Sun, 19 Dec 2010 14:38:17 -0800
> 
> Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > On Sunday 19 December 2010 14:26:19 bearophile wrote:
> > > Jonathan M Davis:
> > > > There will be a library solution to do it, but again, it's unsafe.
> > > 
> > > It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.
> > 
> > Whereas, I would argue that it's completely unnecessary. structs and classes serve different purposes. There is no need for scoped classes. They may perodically be useful, but on the whole, they're completely unnecessary.
> 
> How do you define a need for scoped classes?
> 
> I see two aspects there: first is having destructor called at known point rather than arbitrarily, second is performance.
> 
> It looks perfectly reasonable for me to want to have first. Do you know why things which makes program act more deterministic are shunned from D? Does it really add so much to safety that it is worth reducing programmer's control?

Then use a struct. That's one of the reasons that they exist. And if you _really_ want to make sure that a class' destructor gets run, you can always use clear() (which is arguably an abomination in its own right). And if you use that in combination with scope(exit), then you get a class which is guaranteed to be destroyed when the function exits, and (assuming that clear() is fully implemented and actually zeroes out the vtable - which I don't think that it does yet), you won't be using garbage memory (though obviously it will still go boom - likely due to a segfault).

Part of the whole point of separating classes and structs is to separate the issue of where they go - on the stack or on the heap. So, putting classes on the stack kind of negates the whole point of having both structs and classes in the first place.

> In _general_ case there is no safety in D. With all low-level capabilities one can always defeat compiler. Removing intermediate-level safer (yet unsafe) capabilities arguabily gains nothing but frustration. I'm all for encouraging good practices, but this is different.

In the general case when using SafeD, there D _is_ safe. scoped classes are definitely not in SafeD.

structs already are on the stack. If you want something on the stack, then use structs. And if you _really_ want a class on the stack for whatever reason (which I _really_ question), then there will be a library solution to the problem. By putting classes on the stack, you're pretty much ignoring the differences between structs and classes.

- Jonathan M Davis

On Sun, 19 Dec 2010 17:26:20 -0800
Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Sunday 19 December 2010 16:50:34 Nick Voronin wrote:
> > On Sun, 19 Dec 2010 14:38:17 -0800
> > 
> > Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > > On Sunday 19 December 2010 14:26:19 bearophile wrote:
> > > > Jonathan M Davis:
> > > > > There will be a library solution to do it, but again, it's unsafe.
> > > > 
> > > > It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.
> > > 
> > > Whereas, I would argue that it's completely unnecessary. structs and classes serve different purposes. There is no need for scoped classes. They may perodically be useful, but on the whole, they're completely unnecessary.
> > 
> > How do you define a need for scoped classes?
> > 
> > I see two aspects there: first is having destructor called at known point rather than arbitrarily, second is performance.
> > 
> > It looks perfectly reasonable for me to want to have first. Do you know why things which makes program act more deterministic are shunned from D? Does it really add so much to safety that it is worth reducing programmer's control?
> 
> Then use a struct.

Here is where we diverge. Choosing struct vs class on criteria of their placement makes no sense to me. Difference in default placement hardly matters at all, you can perfectly put structs in heap or in static segment yet still maintain same properties. It's those properties which matter when I choose one or another, not where it will reside in particular part of program.

> That's one of the reasons that they exist. And if you _really_ want to make sure that a class' destructor gets run, you can always use clear() (which is arguably an abomination in its own right).

This is what I don't understand. Fine. structs are meant for stack, class for heap. Conceptually, anyway. Still there will be ways to put class on stack and struct in heap. And there is(will be) clear(). Now I do see benefit of scope storage. It looks clean, it is supported by compiler meaning reasonable level of protection against misuse, it is extremely effective. What are benefits of _not having_ it?

-- 
Nick Voronin <elfy.nv@gmail.com>

On Sunday 19 December 2010 18:13:54 Nick Voronin wrote:
> On Sun, 19 Dec 2010 17:26:20 -0800
> 
> Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > On Sunday 19 December 2010 16:50:34 Nick Voronin wrote:
> > > On Sun, 19 Dec 2010 14:38:17 -0800
> > > 
> > > Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > > > On Sunday 19 December 2010 14:26:19 bearophile wrote:
> > > > > Jonathan M Davis:
> > > > > > There will be a library solution to do it, but again, it's unsafe.
> > > > > 
> > > > > It can be safer if the compiler gives some help. For me it's one of the important unfinished parts of D.
> > > > 
> > > > Whereas, I would argue that it's completely unnecessary. structs and classes serve different purposes. There is no need for scoped classes. They may perodically be useful, but on the whole, they're completely unnecessary.
> > > 
> > > How do you define a need for scoped classes?
> > > 
> > > I see two aspects there: first is having destructor called at known point rather than arbitrarily, second is performance.
> > > 
> > > It looks perfectly reasonable for me to want to have first. Do you know why things which makes program act more deterministic are shunned from D? Does it really add so much to safety that it is worth reducing programmer's control?
> > 
> > Then use a struct.
> 
> Here is where we diverge. Choosing struct vs class on criteria of their placement makes no sense to me. Difference in default placement hardly matters at all, you can perfectly put structs in heap or in static segment yet still maintain same properties. It's those properties which matter when I choose one or another, not where it will reside in particular part of program.
> 
> > That's one of the reasons that they exist. And if you
> > _really_ want to make sure that a class' destructor gets run, you can
> > always use clear() (which is arguably an abomination in its own right).
> 
> This is what I don't understand. Fine. structs are meant for stack, class for heap. Conceptually, anyway. Still there will be ways to put class on stack and struct in heap. And there is(will be) clear(). Now I do see benefit of scope storage. It looks clean, it is supported by compiler meaning reasonable level of protection against misuse, it is extremely effective. What are benefits of _not having_ it?

Except that the compiler _can't_ protect against misuse. That's one of the main reasons that it's going away as part of the language. Structs are intended to their lifetime be the same as their scope. The lifetime for classes, on the other hand, is essentially as long as the program is running (though the garbage collector can obviously collect them in cases where they aren't used anymore). By trying to force a reference type to have a scoped lifetime, you're going against how they work, and it's just asking for trouble. Yes, it _can_ be done, but no, it _can't_ be done safely.

As I understand it, Andrei isn't even entirely happy that clear() exists. I believe that pretty much the only reason that it exists is because there are cases where classes hold resources and you want to make sure that they get released rather than waiting for the object to get destroyed at some indeterminate point in the future. But really, the garbage collected heap is supposed to be managed by the garbage collector. And with scope, you're trying to take something that's designed to live on the garbage collected heap and put it on the stack. You're taking something that is designed to be used in one type of storage and using it in another. It's going to cause problems.

C++ has a number of issues pricely because it allows classes to live on both the stack and the heap. By separating types that live on the stack from those that live on the heap, you avoid those problems.

And actually, the fact that structs can live in the heap in D can be problematic, as evidenced by the bugs relating to destructors for structs on the heap. Not only do their destructors not get run, it's questionable whether the language will _ever_ be such that the destructors of structs on the heap will be run. The result is that in many cases, structs should either be designed to be on the stack or on the heap.

D was designed so that classes live on the heap, and that's the way it works. Putting them on the stack is going against how classes are intended to work, and it's asking for trouble.

- Jonathan M Davis

Jonathan M Davis:

> Whereas, I would argue that it's completely unnecessary.

Recently even the Oracle Java VM allocates some class instances on the stack, when Escape Analysis finds it's safe to do so. And D lacks the very efficient GC of the JVM, so for D it's even more important to use the stack for some classes.

Think about the "pure" tag of D2.

D1 lacks the "pure" annotation, but an advanced D1 compiler may add some logic able to tell if a function is pure, and use this info for performance purposes. The problem here is that if you change the function a little your function may stop being pure and the optimization becomes impossible. The "pure" annotation of D2 is a contract between the programmer and the compiler. In the same way, even if a D compiler gains an automatic escape analysis like Java and a much more efficient GC, a way to annotate a class as scoped is useful still, because like "pure" it's a contract between the compiler and the programmer, it enforces that a class instance doesn't escape a scope (if you try to escape it, the good compiler in most cases gives an error. While in this case the JVM just doesn't allocate the object on the stack), this also means guaranteed performance and deallocation determinism.

Another thing to keep in account is that we aren't talking just about the stack, but generally about in-place allocation. So if you have a class instance A, one of its fields may be another class instance B, that's scoped. Here B is allocated on the heap, because B is contained inside A, that is (generally) on the heap. Yet B is allocated in-place. Having a single heap allocation instead of two when you create an instance A is a significant performance gain (and I think Java doesn't perform this optimization yet). A well designed type system is then able to make all this safe.


> The compiler can help, but it can't fix the problem any more that it can guarantee that a pointer to a local variable doesn't escape once you've passed it to another function. In _some_ circumstances, it can catch escaping pointers and references, but in the general case, it can't.

See also: http://en.wikipedia.org/wiki/Linear_types
Stronger and more powerful type systems are possible (D has means to break any type system, but the programmer does this knowing what she or he is doing. And there are ways similar to @safe to statically disallow the means you may use to break this part of the type system).


>So, putting classes on the stack kind of negates the whole point of having both structs and classes in the first place.<

This is false, the definition of D class instance doesn't specify where the instance memory is allocated.


>In the general case when using SafeD, there D _is_ safe. scoped classes are definitely not in SafeD.<

SafeD is not safe. It allows integer overflows, memory overflows, etc. SafeD just disallows a certain (important) kind of memory bugs. This is why some people have asked to call it "memory safe".

If you take a look at coding standards that specify how to write high integrity software (MISRA-C, SPARK, Joint Strike Fighter Air Vehicle C++ Coding Standards, etc), you will see that they forbid ALL heap allocations, because stack allocations are more deterministic and more predictable, so they are safer. If you disallow recursion the max required size of the stack is even computable statically, avoiding stack overflows.

---------------------

Nick Voronin:

>Safety is important but you don't achieve safety by means of making unsafe thing unconvenient and inefficient.<

I know that sounds a little weird, but in practice this is how some parts of D are designed :-)
Yet, Walter has said many things that he doesn't like "safety by convention". So I presume there is some contradiction here :-)

A language like Cyclone has tried to make low-level operations in a very C-language safe, and it succeeds in this, but Cyclone is both slow, fussy, not handy, a failure. So by design D doesn't try to make low-level operation safe, it just add safe(r) hi-level operations. You are allowed to use the low-level means still, but in this case you are fully on your own, like in C. For a lot of time I have not appreciated much this basic design decision, but so far it has worked well enough, far better than Cyclone.

Bye,
bearophile

On Sunday 19 December 2010 18:33:56 bearophile wrote:
> Jonathan M Davis:
> > Whereas, I would argue that it's completely unnecessary.
> 
> Recently even the Oracle Java VM allocates some class instances on the stack, when Escape Analysis finds it's safe to do so. And D lacks the very efficient GC of the JVM, so for D it's even more important to use the stack for some classes.
> 
> Think about the "pure" tag of D2.
> 
> D1 lacks the "pure" annotation, but an advanced D1 compiler may add some logic able to tell if a function is pure, and use this info for performance purposes. The problem here is that if you change the function a little your function may stop being pure and the optimization becomes impossible. The "pure" annotation of D2 is a contract between the programmer and the compiler. In the same way, even if a D compiler gains an automatic escape analysis like Java and a much more efficient GC, a way to annotate a class as scoped is useful still, because like "pure" it's a contract between the compiler and the programmer, it enforces that a class instance doesn't escape a scope (if you try to escape it, the good compiler in most cases gives an error. While in this case the JVM just doesn't allocate the object on the stack), this also means guaranteed performance and deallocation determinism.
> 
> Another thing to keep in account is that we aren't talking just about the stack, but generally about in-place allocation. So if you have a class instance A, one of its fields may be another class instance B, that's scoped. Here B is allocated on the heap, because B is contained inside A, that is (generally) on the heap. Yet B is allocated in-place. Having a single heap allocation instead of two when you create an instance A is a significant performance gain (and I think Java doesn't perform this optimization yet). A well designed type system is then able to make all this safe.
> 
> > The compiler can help, but it can't fix the problem any more that it can guarantee that a pointer to a local variable doesn't escape once you've passed it to another function. In _some_ circumstances, it can catch escaping pointers and references, but in the general case, it can't.
> 
> See also: http://en.wikipedia.org/wiki/Linear_types
> Stronger and more powerful type systems are possible (D has means to break
> any type system, but the programmer does this knowing what she or he is
> doing. And there are ways similar to @safe to statically disallow the
> means you may use to break this part of the type system).
> 
> >So, putting classes on the stack kind of negates the whole point of having both structs and classes in the first place.<
> 
> This is false, the definition of D class instance doesn't specify where the instance memory is allocated.
> 
> >In the general case when using SafeD, there D _is_ safe. scoped classes are definitely not in SafeD.<
> 
> SafeD is not safe. It allows integer overflows, memory overflows, etc. SafeD just disallows a certain (important) kind of memory bugs. This is why some people have asked to call it "memory safe".

The whole point of "safe" when talking about safe in D is memory saftey. And scoped classes are an issue of memory safety. They are _not_ safe with regards to memory. If the compiler can determine that a particular class object can be put on the stack and optimize it that way. Fine, but it's pretty rare that it can do that - essentially only in cases where you don't pass it to _anything_ except for pure functions (including calls to member functions). And if the compiler can do that, then it there's no need for the programmer to use scope explicitly.

And no, a compiler _can't_ do pure optimizations on its own, generally-speaking, because that would require looking not only at the body of the function that's being called but at the function bodies of any functions that it calls. D is not designed in a way that the compiler even necessarily has _access_ to a function's body when compiling, and you can't generally look at a function's body when doing optimizations when calling that function. So, _some_ pure optimizations could be done, but most couldn't. This is not the case with scoped classes, because purity already gives you the information that you need.

> If you take a look at coding standards that specify how to write high integrity software (MISRA-C, SPARK, Joint Strike Fighter Air Vehicle C++ Coding Standards, etc), you will see that they forbid ALL heap allocations, because stack allocations are more deterministic and more predictable, so they are safer. If you disallow recursion the max required size of the stack is even computable statically, avoiding stack overflows.
> 
> ---------------------
> 
> Nick Voronin:
> >Safety is important but you don't achieve safety by means of making unsafe thing unconvenient and inefficient.<
> 
> I know that sounds a little weird, but in practice this is how some parts of D are designed :-) Yet, Walter has said many things that he doesn't like "safety by convention". So I presume there is some contradiction here :-)

Safety by convention means that the language and the compiler do not enforce it in any way. When the language is designed to include it, then it's _not_ by convention. So, for instance, Go doesn't use its type system to protect you from threads stomping on each other. Rather, the safety is done by convention - the programmer does it by programming in a particular way. In D, on the other hand, the type system enforces it. There's nothing contradictory about Walter's stance. He's for having safety built into the language as much as reasonably possible and against having it thrust upon the programmer to program in a particular way to avoid unsafe stuff. @safe is a prime example of this (assuming that's implemented correctly, which isn't the case at the moment). It causes the compiler to enforce that only safe operations are used rather than relying on the programmer not to use unsafe operations.

- Jonathan M Davis

On Sun, 19 Dec 2010 21:33:56 -0500
bearophile <bearophileHUGS@lycos.com> wrote:

> >So, putting classes on the stack kind of negates the whole point of having both structs and classes in the first place.<
> 
> This is false, the definition of D class instance doesn't specify where the instance memory is allocated.

For me, the important difference is that classes are referenced, while structs are plain values. This is a semantic distinction of highest importance. I would like structs to be subtype-able and to implement (runtime-type-based) polymorphism.

Denis
-- -- -- -- -- -- --
vit esse estrany ☣

spir.wikidot.com