grauzone wrote:
> Andrei Alexandrescu wrote:
>> That being said, I like stack-allocated and I asked Walter for a long time to introduce them in the language. They have their problems though.
> They can just be disabled in safe mode, if that's the problem.

One of the problems is they can easily lead to stack exhaustion. Stack sizes for a thread must all be preallocated.

== Quote from Chad J (chadjoan@__spam.is.bad__gmail.com)'s article
> http://d.puremagic.com/issues/show_bug.cgi?id=3696
> I'm realizing now that returning an object allocated on this stack is a
> tricky task unless there is help from the caller.  The lifetime of the
> object has to be determined somehow.  The Tango convention makes it
> fairly explicit that the parent's scope will be the beginning and the
> end for the memory.  If it needs to live longer then the caller just
> passes a buffer that isn't stack/scoped memory.  Only the caller really
> knows how long the memory needs to hang around.  hmmmmm.

The optional buffer idiom is great for stuff that's returned from a function.  I use it all the time.  On the other hand, for temporary buffers that are used internally, whose mere existence is an implementation detail, having the caller maintain and pass in a buffer is a horrible violation of encapsulation and good API design, even by the standards of performance-oriented APIs.  Even if you know it's never going to change, it's still annoying for the caller to even have to think about these kinds of details.

> Still that SuperStack thing would be useful.  It's like alloca but
> better (look ma, no cast) and slightly more general (ex: it can be freed
> in parent's scope, if you are willing to play it risky).

Yeah, for about the past year I've been playing around with TempAlloc, wich was basically me taking Andrei's SuperStack idea when it was first proposed and running with it because I needed it sooner rather than later.  It's basically encapsulated in dstats.alloc (http://svn.dsource.org/projects/dstats/docs/alloc.html).

Its one **huge** bug that I've been trying to fix is that it's not scanned by the GC because scanning the whole stack would be too inefficient and I can't think of a way to do it more efficiently.  However, it's still useful in cases where either you're just rearranging data and there's already a copy somewhere that is scanned by the GC (sorting for non-parametric statistical calculations is my killer use) or your data is pure value types like floats.  It's got the following advantages over alloca:

1.  Can't overflow unless you're completely out of address space.  As a last resort, it allocates another chunk of memory from the heap.

2.  Nicer syntax.  For example, to allocate an array of 1,000 uints:

auto foo = newStack!uint(1_000);

3.  Since lifetimes are not bound to any function scope unless you want them to be, you can implement complex abstract data structures (I've got hash tables, hash sets and AVL trees so far) on this stack.  This is useful when you need to build a lookup table as part of some algorithm.

dsimcha wrote:
> 
> The optional buffer idiom is great for stuff that's returned from a function.  I use it all the time.  On the other hand, for temporary buffers that are used internally, whose mere existence is an implementation detail, having the caller maintain and pass in a buffer is a horrible violation of encapsulation and good API design, even by the standards of performance-oriented APIs.  Even if you know it's never going to change, it's still annoying for the caller to even have to think about these kinds of details.
> 

Yes, quite.

Chad J wrote:
> dsimcha wrote:
>> The optional buffer idiom is great for stuff that's returned from a function.  I
>> use it all the time.  On the other hand, for temporary buffers that are used
>> internally, whose mere existence is an implementation detail, having the caller
>> maintain and pass in a buffer is a horrible violation of encapsulation and good
>> API design, even by the standards of performance-oriented APIs.  Even if you know
>> it's never going to change, it's still annoying for the caller to even have to
>> think about these kinds of details.
>>
> 
> Yes, quite.

I think an API that is memory-safe relies on one global array like this:

template SuperStack(T) {
    private T[] buffer;
    private enum slackfactor = 2.5;

    T[] getBuffer(size_t n) {
        buffer.length = buffer.length + n;
        return buffer[$ - n .. $];
    }

    void releaseBuffer(T[] b) {
        enforce(b is buffer[$ - b.length .. $]);
        buffer.length = buffer.length - b.length;
        if (gc.capacity(buffer) > buffer.length * slackfactor) {
            // Reallocate buffer to allow collection of slack
            buffer = buffer.dup;
        }
    }
}

Further encapsulation is possible e.g. by defining a struct that pairs calls to getBuffer and releaseBuffer.

The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.


Andrei

Andrei Alexandrescu wrote:
> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.

That's an interesting point. You can have two things:
1. Correct memory managment (one can never access an object that's supposed to be free'd)
2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)

In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.

> 
> Andrei

grauzone wrote:
> Andrei Alexandrescu wrote:
>> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.
> 
> That's an interesting point. You can have two things:
> 1. Correct memory managment (one can never access an object that's supposed to be free'd)
> 2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)
> 
> In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.

Normal D must be debuggable D.

Andrei

Andrei Alexandrescu wrote:
> grauzone wrote:
>> Andrei Alexandrescu wrote:
>>> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.
>>
>> That's an interesting point. You can have two things:
>> 1. Correct memory managment (one can never access an object that's supposed to be free'd)
>> 2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)
>>
>> In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.
> 
> Normal D must be debuggable D.

That isn't the case right now and never will be.

That said, I'd prefer solutions that follow 1. instead of 2. I'm sure it can work somehow; it also worked with closures (at least partial) and normal stack variables + ref.

> Andrei

grauzone wrote:
> Andrei Alexandrescu wrote:
>> grauzone wrote:
>>> Andrei Alexandrescu wrote:
>>>> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.
>>>
>>> That's an interesting point. You can have two things:
>>> 1. Correct memory managment (one can never access an object that's supposed to be free'd)
>>> 2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)
>>>
>>> In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.
>>
>> Normal D must be debuggable D.
> 
> That isn't the case right now and never will be.

Then if I were you and I really believed that, I wouldn't waste one more minute hanging out in this group.

Andrei

Andrei Alexandrescu wrote:
> grauzone wrote:
>> Andrei Alexandrescu wrote:
>>> grauzone wrote:
>>>> Andrei Alexandrescu wrote:
>>>>> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.
>>>>
>>>> That's an interesting point. You can have two things:
>>>> 1. Correct memory managment (one can never access an object that's supposed to be free'd)
>>>> 2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)
>>>>
>>>> In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.
>>>
>>> Normal D must be debuggable D.
>>
>> That isn't the case right now and never will be.
> 
> Then if I were you and I really believed that, I wouldn't waste one more minute hanging out in this group.

Don't worry, I won't leave so easily.

That said, if you use "delete" (or GC.free) incorrectly, you may end up with an undebuggable program. Plus there's no way manual free will ever be removed from D. I'm sure you know that, but it sounds like you're denying that. Whatever.

> Andrei

grauzone wrote:
> Andrei Alexandrescu wrote:
>> grauzone wrote:
>>> Andrei Alexandrescu wrote:
>>>> grauzone wrote:
>>>>> Andrei Alexandrescu wrote:
>>>>>> The idea is that the API offers a means to define and use temporary buffers without compromising memory safety. Even if you escape data allocated via getBuffer that persists after releaseBuffer, that will not cause undefined behavior. (It may, however, cause data to be overwritten because another call to getBuffer will reuse memory.) Leaks are also possible if you don't release the buffer. That can be solved by not offering getBuffer and releaseBuffer as they are, but instead only encapsulated in a struct with the appropriate constructor and destructor.
>>>>>
>>>>> That's an interesting point. You can have two things:
>>>>> 1. Correct memory managment (one can never access an object that's supposed to be free'd)
>>>>> 2. Safe memory managment (event if you access an object that's supposed to be free'd, it's memory safe)
>>>>>
>>>>> In safe mode, 1. can't be allowed, and 2. is better than nothing. In normal D, I'd say 2. is quite useless, except as an debugging option.
>>>>
>>>> Normal D must be debuggable D.
>>>
>>> That isn't the case right now and never will be.
>>
>> Then if I were you and I really believed that, I wouldn't waste one more minute hanging out in this group.
> 
> Don't worry, I won't leave so easily.

I know. Clearly there is something in D that you find irresistibly attractive; I wonder what that is :o).

> That said, if you use "delete" (or GC.free) incorrectly, you may end up with an undebuggable program. Plus there's no way manual free will ever be removed from D. I'm sure you know that, but it sounds like you're denying that. Whatever.

Of course I'm denying it because it's false. SafeD will not allow programs containing manual free. And SafeD is quickly becoming a reality, your pessimism notwithstanding.


Andrei