On 10 October 2013 01:46, Paulo Pinto <pjmlp@progtools.org> wrote:

> Am 09.10.2013 16:30, schrieb Manu:
>
>> On 9 October 2013 17:31, Walter Bright <newshound2@digitalmars.com <mailto:newshound2@**digitalmars.com <newshound2@digitalmars.com>>> wrote:
>>
>>     On 10/9/2013 12:29 AM, Manu wrote:
>>
>>         Does anyone here REALLY believe that a bunch of volunteer
>>         contributors can
>>         possibly do what apple failed to do with their squillions of
>>         dollars and engineers?
>>         I haven't heard anybody around here propose the path to an
>>         acceptable solution.
>>         It's perpetually in the too-hard basket, hence we still have the
>>         same GC as
>>         forever and it's going nowhere.
>>
>>
>>     What do you propose?
>>
>>
>> ARC. I've been here years now, and I see absolutely no evidence that the
>> GC is ever going to improve. I can trust ARC, it's predictable, I can
>> control it.
>> Also, proper support for avoiding the GC without severe inconvenience as
>> constantly keeps coming up. But I don't think there's any debate on that
>> one. Everyone seems to agree.
>>
>
> As someone that is in the sidelines and doesn't really use D, my opinion should not count that much, if at all.
>
> However, rewriting D's memory management to be ARC based will have performance impact if the various D compilers aren't made ARC aware.
>

Supporting ARC in the compiler _is_ the job. That includes a cyclic-reference solution.

Then there is the whole point of rewriting phobos and druntime to use ARC
> instead of GC.
>

It would be transparent if properly supported by the compiler.

Will the return on investment pay off, instead of fixing the existing GC?
>

If anyone can even _imagine_ a design for a 'fixed' GC, I'd love to hear it. I've talked with a lot of experts, they all mumble and groan, and just talk about how hard it is.

What will be the message sent to the outsiders wondering if D is stable
> enough to be adopted, and see these constant rewrites?
>

People didn't run screaming from Obj-C when they switched to ARC. I think they generally appreciated it.

On 10/9/13 2:29 AM, Robert Schadek wrote:
> On 10/09/2013 04:22 AM, Andrei Alexandrescu wrote:
>>
>> * Get Robert Schadek's precise GC in. Walter and I have become 101%
>> convinced a precise GC is the one way to go about GC.
>>
> I would like to claim that work, but the Rainer Schütze wrote that. I
> know, both german, same initials.

Apologies to both!

Andrei

On 10/9/13 4:30 AM, Jacob Carlborg wrote:
> On 2013-10-09 05:31, Walter Bright wrote:
>
>> Making this work is fraught with difficulty. It is normal behavior in D
>> to create local data with new(), build a data structure, and then cast
>> it to shared so it can be transferred to another thread. This will fail
>> miserably if the data is allocated on a thread local heap.
>
> I agree with Andrei here. Alternatively perhaps the runtime can move the
> data to a global pool if it's casted to shared.

I think a reasonable solution (in case we're unable to solve this reasonably within the type system) is to offer a library solution that does the shared allocation, mutation, and casting internally.

Again: USER CODE SHOULD HAVE NO BUSINESS CASTING CASUALLY TO GET WORK DONE.


Andrei

On 2013-10-09 16:51:03 +0000, Manu <turkeyman@gmail.com> said:

> On 10 October 2013 01:40, Johannes Pfau <nospam@example.com> wrote:
> 
>> But if someone really wants to strip the GC _completely_ there's a huge
>> issue with memory management of Exceptions.
> 
> Exceptions have a pretty well defined lifetime... can't they be manually
> cleaned up by the exception handler after the catching scope exits?

Exceptions don't need a well-defined lifetime for things to work.

D exceptions are classes and are heap-allocated. So if everything becomes reference-counted, exceptions would be reference-counted too. The exception handler would be the one decrementing the reference count once it has done with the exception (all this under the hood, managed by the compiler).

Alternatively an exception handler could return the exception to the parent function (as a return value), store the exception elsewhere, or throw it again, in which case the decrement operation would be balanced by an increment, and both increment and decrement should be elided by the compiler as they're cancelling each other.

I fail to see an issue.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

On 10/9/13 6:06 AM, Michel Fortin wrote:
> On 2013-10-09 07:33:29 +0000, Manu <turkeyman@gmail.com> said:
>
>> Is there more to it? Cleaning up circular references I guess... what does
>> Apple do?
>
> Apple implemented auto-nulling weak pointers for ARC (as well as
> __unsafe_unretained ones if you are not afraid of dangling pointers).

Yah, the rub being the programmer must use weak pointers judiciously. The difficulty is in "judiciously" :o).

>> It's an uncommon edge case, so there's gotta be heaps of room for
>> efficient
>> solutions to that (afaik) one edge case. Are there others?
>
> I don't know about you, but circular references are not rare at all in
> my code.

Nor Facebook's. I can't give away numbers, but the fraction of cycles per total memory allocated averaged over all of Facebook's PHP programs (PHP being reference counted) is staggering. Staggering!

If PHP didn't follow a stateless request model (which allows us to use region allocation and reclaim all memory upon the request's end), we'd be literally unable to run a server for more than a few minutes.

> Another solution that could be used for circular references is to have
> the stop-the-world GC we have now collect them. Reference counting could
> be used to free GC memory in advance (in the absence of circular
> references), so the GC would have less garbage to collect when it runs.

Yah, that seems to be a good approach for us.


Andrei

On 10/09/2013 12:40 PM, qznc wrote:
> I found no summary and stuff seems to get lost,
> so I created a page on the wiki.
>
> http://wiki.dlang.org/Versus_the_garbage_collector

+1

On 2013-10-09 14:52:35 +0000, Manu <turkeyman@gmail.com> said:

> I suspect there are plenty of creative possibilities that could be applied
> to reducing the pool of potential circular-references to as small a pool as
> possible.

I'm more pessimistic than you on this. Cycles can only be detected at runtime unless we make dramatic changes to the memory model. You can find cycles by scanning at intervals, as the GC does. Or you could detect them by updating a dependency graph each time you assign a value to a pointer. The dependency graph would have to tell you whether you're still connected to a root. When all root connections are severed you can free the object. I doubt the overhead would make this later idea practical however.

Weak-autonulling pointers are much cheaper. I'd suggest using one of the above methods as debugging aid to find cycles, and then manually insert weak pointers where needed to break those cycles. Either that, or leave the GC on if you don't mind the GC.


> It's simply not feasible to have machine scanning 4-6 gigabytes of
> allocated memory looking for garbage.

Agree.


-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

Am Wed, 9 Oct 2013 13:13:51 -0400
schrieb Michel Fortin <michel.fortin@michelf.ca>:

> On 2013-10-09 16:51:03 +0000, Manu <turkeyman@gmail.com> said:
> 
> > On 10 October 2013 01:40, Johannes Pfau <nospam@example.com> wrote:
> > 
> >> But if someone really wants to strip the GC _completely_ there's a huge issue with memory management of Exceptions.
> > 
> > Exceptions have a pretty well defined lifetime... can't they be manually cleaned up by the exception handler after the catching scope exits?
> 
> Exceptions don't need a well-defined lifetime for things to work.

What I meant was using exceptions without the GC requires huge changes (like switching all exceptions to use reference counting) or at least using the same type of allocation for all exceptions. If you have 'malloced' and GC owned exceptions it can be hard to know if an Exception needs to be freed:

try
{
    codeWithMallocException();
    codeWithGCException();
}
catch(Exception e)
{
    //Am I supposed to free this exception now?
}

Now think of exception chaining... I think this can easily lead to memory leaks or other latent issues.

So avoiding parts of phobos which allocate right now is relatively easy. Switching all code to use 'malloced' Exceptions is much more work.

> D exceptions are classes and are heap-allocated. So if everything
> becomes reference-counted, exceptions would be reference-counted too.
> [...]
> I fail to see an issue.
> 

I meant it's complicated right now, without compiler / runtime / phobos
changes. You can avoid parts of phobos as a user but getting rid of
GC allocated exceptions is more difficult. Of course if you can change
all exceptions to use reference counting that works well but that's a
rather big change.

Am 09.10.2013 19:05, schrieb Manu:
> On 10 October 2013 01:46, Paulo Pinto <pjmlp@progtools.org
> <mailto:pjmlp@progtools.org>> wrote:
>
>     Am 09.10.2013 16:30, schrieb Manu:
>
>         On 9 October 2013 17:31, Walter Bright
>         <newshound2@digitalmars.com <mailto:newshound2@digitalmars.com>
>         <mailto:newshound2@__digitalmars.com
>         <mailto:newshound2@digitalmars.com>>> wrote:
>
>              On 10/9/2013 12:29 AM, Manu wrote:
>
>                  Does anyone here REALLY believe that a bunch of volunteer
>                  contributors can
>                  possibly do what apple failed to do with their
>         squillions of
>                  dollars and engineers?
>                  I haven't heard anybody around here propose the path to an
>                  acceptable solution.
>                  It's perpetually in the too-hard basket, hence we still
>         have the
>                  same GC as
>                  forever and it's going nowhere.
>
>
>              What do you propose?
>
>
>         ARC. I've been here years now, and I see absolutely no evidence
>         that the
>         GC is ever going to improve. I can trust ARC, it's predictable,
>         I can
>         control it.
>         Also, proper support for avoiding the GC without severe
>         inconvenience as
>         constantly keeps coming up. But I don't think there's any debate
>         on that
>         one. Everyone seems to agree.
>
>
>     As someone that is in the sidelines and doesn't really use D, my
>     opinion should not count that much, if at all.
>
>     However, rewriting D's memory management to be ARC based will have
>     performance impact if the various D compilers aren't made ARC aware.
>
>
> Supporting ARC in the compiler _is_ the job. That includes a
> cyclic-reference solution.
>
>     Then there is the whole point of rewriting phobos and druntime to
>     use ARC instead of GC.
>
>
> It would be transparent if properly supported by the compiler.
>
>     Will the return on investment pay off, instead of fixing the
>     existing GC?
>
>
> If anyone can even _imagine_ a design for a 'fixed' GC, I'd love to hear
> it. I've talked with a lot of experts, they all mumble and groan, and
> just talk about how hard it is.
>
>     What will be the message sent to the outsiders wondering if D is
>     stable enough to be adopted, and see these constant rewrites?
>
>
> People didn't run screaming from Obj-C when they switched to ARC. I
> think they generally appreciated it.

Because Objective-C's GC design was broken, as I mentioned on my previous posts.

Anyway, you make good points, thanks for the reply.

--
Paulo

Am 09.10.2013 17:54, schrieb Sean Kelly:
> On Oct 9, 2013, at 8:35 AM, Paulo Pinto <pjmlp@progtools.org> wrote:
>>
>> The Azul VM does not have a problem with it, as it does pauseless concurrent GC, while being used on online trading systems.
>
> Incremental GCs are awesome.  Have you looked at IBM's Metronome for Java?  ...
>

Just the technical papers.

> ...  Maybe this would work with SafeD, but I don't think it's an option with the full D language.
>

Yeah, maybe if D was safe by default, with pointer tricks encapsulated inside system sections. Similar to unsafe blocks in other languages.

Not sure how much that would help, though.

--
Paulo