On Thursday, 6 February 2014 at 13:03:43 UTC, Paulo Pinto wrote:
> You would be amazed how many times I have written FFI code that decrypts source code on load.

You can probably create a decent decompiler for D-code given the reliance on GC/stack frames etc, so I am not sure if that is a rational point (perhaps a political one).

Let me put it this way then: If havoc is available as a poorly adapted blob it will discourage development of a native idiomatic open source D physics engine, because it pays off more to spend time working around blob-related issues and get stellar performance. With no physics engine you will have something primitive in D instead, but the moment it is good enough for creating simple apps people that are interested will improve on that rather than working around Havoc issues. So, slower start, but better for the eco system in the long term.

On 2/6/14, 3:44 AM, Michel Fortin wrote:
> On 2014-02-06 04:56:28 +0000, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> said:
>
>> On 2/5/14, 4:53 PM, Michel Fortin wrote:
>>> On 2014-02-05 22:19:27 +0000, Andrei Alexandrescu
>>> <SeeWebsiteForEmail@erdani.org> said:
>>>
>>>> I want to make one positive step toward improving memory allocation in
>>>> the D language.
>>>
>>> I know. But I find your proposal confusing.
>>>
>>> Perhaps this is just one piece in your master plan where everything will
>>> make sense once we have all the pieces. But this piece by itself makes
>>> no sense to me; I have no idea where you're going with it.
>>>
>>> Is this the continuation of the old thread where you wanted ideas about
>>> how to eliminate hidden allocations in buildPath? Doesn't sound like it.
>>
>> Actually buildPath is a good example because it concatenates strings.
>> It should work transparently with RC and GC strings.
>
> That thread about buildPath started like this: "Walter and I were
> talking about eliminating the surreptitious allocations in buildPath".
> But reference counting will do nothing to eliminate surreptitious
> allocations.

That's exactly right. Currently buildPath uses ~= several times so it will produce allocations that the user is unable to free. If buildPath used reference counting through and through, temporary allocations would be freed eagerly inside buildPath, and the user will have a shot at freeing the end result.

> It can't be that problem you're trying to address.

Are you sure you want to debate with me what's in my mind?


Andrei

I know I'm coming into this a bit late, but in general I only feel like there's a problem with built-in dynamic types.  It generally isn't hard to manage the lifetime of classes manually, and I don't tend to churn through them.  Also, there are basically no routines in Phobos that operate on classes, so that's really entirely my problem as a user anyway.  What concerns me most is string processing, and in particular the routines in Phobos that do string processing.  And while I've always liked the idea of supplying a destination buffer to these routines, it doesn't help the case where the buffer is too small and allocation still needs to occur.

Instead of sorting out some crazy return type or supplying a raw destination buffer, what if we instead supply an appender?  Then the appender could grow the buffer in place or throw or whatever we want the behavior to be when out of space.  I think this would solve 90% of my concerns about unexpected GC pauses in my code.  For the rest, any temporary allocations inside Phobos routines should either be eliminated, reused on each call per a static reference, or cleaned up.  I'm really okay with the occasional "new" inside a routine so long as repeated calls to that routine don't generate tons of garbage and thus trigger a collection.

Am Wed, 05 Feb 2014 19:44:50 +0000
schrieb "Jameson Ernst" <jameson@example.com>:

> I'm just a casual end-user of D, but have been following this and related discussions with great interest. Just yesterday I was trying to "sell" D to a friend, and he basically told me that he'd be interested once the memory management situation gets resolved. I've been thinking about this a lot lately, even though I'm probably way out of my depth given the experts that frequent this forum. Still, I had an idea about this and thought I'd throw it out there.
> 
> On Wednesday, 5 February 2014 at 04:18:49 UTC, Andrei Alexandrescu wrote:
> > On 2/4/14, 5:12 PM, deadalnix wrote:
> >> That being understood, I'd rather connect things the other way around.
> >>
> >> auto x = foo();
> >> auto x = foo().toRC();
> >
> > I don't know how to implement that.
> >
> >
> > Andrei
> 
>  From the discussion currently going on about postblits, it seems
> like there's a new emerging concept of a "unique expression," the
> value of which is guaranteed not to be referenced elsewhere.
> Could this concept perhaps be leveraged to go backwards from GC
> to RC? If you perform a GC allocation in the context of a unique
> expression, would it then be safe to force it back into an RC
> context, knowing that there are no outstanding references to it?
> What's more, it would allow library writes to mostly perform
> allocations a single way, giving the choice to the caller how
> they'd like to manage the lifetime of the newly returned unique
> object.
> 
> I could be completely misunderstanding the unique-expression concept though.

The intent is to make it possible to avoid the GC, but if I understand you correctly you talk about always allocating through the GC first.

-- 
Marco

On Thursday, 6 February 2014 at 23:13:11 UTC, Marco Leise wrote:
> The intent is to make it possible to avoid the GC, but if I
> understand you correctly you talk about always allocating
> through the GC first.

One of the ideas on the table is to use ARC backed by the GC to clear cycles, which would mean the GC needs to be aware of all allocations anyway so it can scan and clear them if a cycle is formed. It's not safe to start reference counting GC memory at an arbitrary time after allocation, since references to it could be anywhere, but if you have a unique expression, then you know there are no other references to the new data, so it's safe to begin reference counting it at that time before it propagates out into the program at large.

On Thursday, 6 February 2014 at 11:44:34 UTC, Michel Fortin wrote:
> That thread about buildPath started like this: "Walter and I were talking about eliminating the surreptitious allocations in buildPath". But reference counting will do nothing to eliminate surreptitious allocations. It can't be that problem you're trying to address.

I think stuff like buildPath just shows how the language should be geared more towards compiler optimizations of allocation if performance is a real goal.

The efficient thing to do is to optimize a string concat into a stack allocation (alloca) if it is a throw-away, and just change the stack frame upon return, with some heuristics and alternative execution paths in order to avoid running out of stack space.

E.g.
a(){
    return buildPath(...) ~ "!";
}

Compiles to:

a:
   call buildPath(...)
   alloca(size_to_endof_path_returned_by_buildPath+1)
   *someaddr = '!'
   return (stackbufferstart,length)


buildPath:
   stackbuffer=alloca(sum_of_lengths)
   copy...
   return tuple(stackbufferstart,length)

On Friday, 7 February 2014 at 00:30:00 UTC, Ola Fosheim Grøstad wrote:
>    alloca(size_to_endof_path_returned_by_buildPath+1)

Of course, since the stack usually grows that should be size_to_start_of_path and you need to preallocate the necessary padding. A pity that stacks usually don't grow upwards, that would make appending much more flexible.

On 2/6/14, 4:29 PM, "Ola Fosheim Grøstad" <ola.fosheim.grostad+dlang@gmail.com>" wrote:
> On Thursday, 6 February 2014 at 11:44:34 UTC, Michel Fortin wrote:
>> That thread about buildPath started like this: "Walter and I were
>> talking about eliminating the surreptitious allocations in buildPath".
>> But reference counting will do nothing to eliminate surreptitious
>> allocations. It can't be that problem you're trying to address.
>
> I think stuff like buildPath just shows how the language should be
> geared more towards compiler optimizations of allocation if performance
> is a real goal.
>
> The efficient thing to do is to optimize a string concat into a stack
> allocation (alloca) if it is a throw-away, and just change the stack
> frame upon return, with some heuristics and alternative execution paths
> in order to avoid running out of stack space.

This is a very incomplete, naive sketch. It could be made to work only in a language that has no backward compatibility to worry about.

Andrei

On Friday, 7 February 2014 at 01:21:25 UTC, Andrei Alexandrescu wrote:
> This is a very incomplete, naive sketch. It could be made to work only in a language that has no backward compatibility to worry about.

Whaddya mean? On the object code level? The compiler should be conservative and generate the alternatives if needed.

In this day and age a language should aim for whole program optimization and static analysis. Not necessarily the proof-of-concept compiler, but the language spec.

A system level language should go out of its way to make stack and pool allocations likely/possible/probable as well as register based optimizations on calls etc.

On 2/6/14, 6:14 PM, "Ola Fosheim Grøstad" <ola.fosheim.grostad+dlang@gmail.com>" wrote:
> On Friday, 7 February 2014 at 01:21:25 UTC, Andrei Alexandrescu wrote:
>> This is a very incomplete, naive sketch. It could be made to work only
>> in a language that has no backward compatibility to worry about.
>
> Whaddya mean? On the object code level? The compiler should be
> conservative and generate the alternatives if needed.
>
> In this day and age a language should aim for whole program optimization
> and static analysis. Not necessarily the proof-of-concept compiler, but
> the language spec.
>
> A system level language should go out of its way to make stack and pool
> allocations likely/possible/probable as well as register based
> optimizations on calls etc.

I'm jaded - Walter and I bounced around for years ideas that vaguely prescribe a feature/optimization and gloss over many details and all difficulties... they're a dime a dozen.

Andrei