Anti-GC crowd tries to promote ARC as an deterministic alternative for memory management.
I noticed that people promoting ARC do not provide any disadvantages for proposed approach.

The thing is in gamedev and other soft-realitime software background only a handfull types of resources are really managed by RC and memory usage patterns are VERY specific to their domain (mostly linear allocation/deallocation and objects with non deterministic lifetime are preallocated in pools).

Trying to use RC as a general method of memory management leads to some problems.
A pretty detailed view by John Harrop (He is somewhat known for trolling in PL community, but nonetheless knows what he is talking about) - http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1# 


So RC could also introduce unpredictable pause times at undesired places.

This is also confirmed by research from HP - http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf

My point is that we should not ruin the language ease of use. We do need to deal with Phobos internal allocations, but we should not switch to ARC as a default memory management scheme. In practice people promoting ARC will probably not use phobos anyway. Currently its just an excuse to not use D.

Look at c++ and STL, etc. People will roll their own solutions no matter what you try.

On Thursday, 6 February 2014 at 11:37:59 UTC, Max Klyga wrote:
> Anti-GC crowd tries to promote ARC as an deterministic alternative for memory management.
> I noticed that people promoting ARC do not provide any disadvantages for proposed approach.
>
> The thing is in gamedev and other soft-realitime software background only a handfull types of resources are really managed by RC and memory usage patterns are VERY specific to their domain (mostly linear allocation/deallocation and objects with non deterministic lifetime are preallocated in pools).
>
> Trying to use RC as a general method of memory management leads to some problems.
> A pretty detailed view by John Harrop (He is somewhat known for trolling in PL community, but nonetheless knows what he is talking about) - http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>
>
> So RC could also introduce unpredictable pause times at undesired places.
>
> This is also confirmed by research from HP - http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>
> My point is that we should not ruin the language ease of use. We do need to deal with Phobos internal allocations, but we should not switch to ARC as a default memory management scheme. In practice people promoting ARC will probably not use phobos anyway. Currently its just an excuse to not use D.
>
> Look at c++ and STL, etc. People will roll their own solutions no matter what you try.

Nicely said.

I believe both approaches should be available. Those who must work without GC should be able to easily do that, but GC should be picked as default simply because it is better for general programming tasks. If D turns out to be used for something else, and majority of use-cases require GC to be off, perhaps D should switch to no-GC by default. I do not see this happening, to be honest.

This is somewhat similar to final-by-default crowd who wants class methods to be final, not virtual by default. Again, D should provide means to satisfy both crowds, because I think it is possible.

Am 06.02.2014 12:37, schrieb Max Klyga:
> Anti-GC crowd tries to promote ARC as an deterministic alternative for
> memory management.
> I noticed that people promoting ARC do not provide any disadvantages for
> proposed approach.
>
> The thing is in gamedev and other soft-realitime software background
> only a handfull types of resources are really managed by RC and memory
> usage patterns are VERY specific to their domain (mostly linear
> allocation/deallocation and objects with non deterministic lifetime are
> preallocated in pools).
>
> Trying to use RC as a general method of memory management leads to some
> problems.
> A pretty detailed view by John Harrop (He is somewhat known for trolling
> in PL community, but nonetheless knows what he is talking about) -
> http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>
>
> So RC could also introduce unpredictable pause times at undesired places.
>
> This is also confirmed by research from HP -
> http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>
> My point is that we should not ruin the language ease of use. We do need
> to deal with Phobos internal allocations, but we should not switch to
> ARC as a default memory management scheme. In practice people promoting
> ARC will probably not use phobos anyway. Currently its just an excuse to
> not use D.
>
> Look at c++ and STL, etc. People will roll their own solutions no matter
> what you try.
>

Full ACK! Reference counting should be well supported, but it shouldn't be the default scheme or built-in at a low level. From my personal experience it would be ideal to be able to customize certain types to be reference counted (allowing the user full flexibility implementing the actual reference counting and without ruling out weak references!), but have them accessible using the same syntax and type conversion semantics as normal references.

On Thursday, 6 February 2014 at 13:23:14 UTC, Sönke Ludwig wrote:
> Am 06.02.2014 12:37, schrieb Max Klyga:
>> Anti-GC crowd tries to promote ARC as an deterministic alternative for
>> memory management.
>> I noticed that people promoting ARC do not provide any disadvantages for
>> proposed approach.
>>
>> The thing is in gamedev and other soft-realitime software background
>> only a handfull types of resources are really managed by RC and memory
>> usage patterns are VERY specific to their domain (mostly linear
>> allocation/deallocation and objects with non deterministic lifetime are
>> preallocated in pools).
>>
>> Trying to use RC as a general method of memory management leads to some
>> problems.
>> A pretty detailed view by John Harrop (He is somewhat known for trolling
>> in PL community, but nonetheless knows what he is talking about) -
>> http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>>
>>
>> So RC could also introduce unpredictable pause times at undesired places.
>>
>> This is also confirmed by research from HP -
>> http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>>
>> My point is that we should not ruin the language ease of use. We do need
>> to deal with Phobos internal allocations, but we should not switch to
>> ARC as a default memory management scheme. In practice people promoting
>> ARC will probably not use phobos anyway. Currently its just an excuse to
>> not use D.
>>
>> Look at c++ and STL, etc. People will roll their own solutions no matter
>> what you try.
>>
>
> Full ACK! Reference counting should be well supported, but it shouldn't be the default scheme or built-in at a low level. From my personal experience it would be ideal to be able to customize certain types to be reference counted (allowing the user full flexibility implementing the actual reference counting and without ruling out weak references!), but have them accessible using the same syntax and type conversion semantics as normal references.

I think the best way forward would be to look at the places in D where allocations happen, and then figure out how we can optionally allow reference counting in these situations. Andrei just made a thread on this yesterday in regard to slices, which I think are the most promising for a RC solution.

I still wonder where the idea of replacing GC with something as a silver bullet came from. There is no problem with GC itself as you can remove it easily. Problem is state of language after it was removed and it is something completely different and unrelated.

All this ARC fuss came from few speculative discussions and suddenly got caught with great attention for reasons I fail to understand. And doing something like going for ARC by default is just crazy - it will make life more difficult for majority of users that don't care and won't fix many real issues for vocal minority.

Real helping problems to be addressed instead in my opinion:
1) providing RC-based gc_stub
2) -vgc and/or better control over hidden allocation
3) removing as much internal allocations as possible from Phobos, move to output ranges instead
4) provide examples of containers befriended with std.allocator

Not related to memory management but demanded in same domain - fix symbol boat, __forceinline

As you may notice, reference counting is just a one tiny part here and only desired as some way to get non-leaking basic language in absence of gc. And probably least important.

All recent threads just make me frustrated despite being one of pushers for better low-level memory management options.

On Thursday, 6 February 2014 at 11:37:59 UTC, Max Klyga wrote:
> Anti-GC crowd tries to promote ARC as an deterministic alternative for memory management.
> I noticed that people promoting ARC do not provide any disadvantages for proposed approach.


Feel free to extend this wiki page:

http://wiki.dlang.org/Versus_the_garbage_collector

On Thursday, 6 February 2014 at 11:37:59 UTC, Max Klyga wrote:
> Anti-GC crowd tries to promote ARC as an deterministic alternative for memory management.
> I noticed that people promoting ARC do not provide any disadvantages for proposed approach.
>
> The thing is in gamedev and other soft-realitime software background only a handfull types of resources are really managed by RC and memory usage patterns are VERY specific to their domain (mostly linear allocation/deallocation and objects with non deterministic lifetime are preallocated in pools).
>
> Trying to use RC as a general method of memory management leads to some problems.
> A pretty detailed view by John Harrop (He is somewhat known for trolling in PL community, but nonetheless knows what he is talking about) - http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>
>
> So RC could also introduce unpredictable pause times at undesired places.
>
> This is also confirmed by research from HP - http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>
> My point is that we should not ruin the language ease of use. We do need to deal with Phobos internal allocations, but we should not switch to ARC as a default memory management scheme. In practice people promoting ARC will probably not use phobos anyway. Currently its just an excuse to not use D.
>
> Look at c++ and STL, etc. People will roll their own solutions no matter what you try.

I think of RC as a greater evil that GC. From what I've seen it does creates leaking cycles in tree structures AND pauses. From a low-level point of view, RC pointers are ugly (separate counter that will trash your cache) and do atomics all over the place (ie. memory barriers). That they don't have to because of shared is TBD.

It is comforting to me to know that a GC pointer is still just a pointer. If we go the RC route, we will have to constantly think about the higher cost of RC pointer vs weak-ref, instead of thinking about other things instead
Right now a GC pointer is the same size as a non-GC one, it's liberating.

It looks like we want to solve a PR problem more than a real one.

Am 06.02.2014 14:35, schrieb Meta:
> On Thursday, 6 February 2014 at 13:23:14 UTC, Sönke Ludwig wrote:
>> Am 06.02.2014 12:37, schrieb Max Klyga:
>>> Anti-GC crowd tries to promote ARC as an deterministic alternative for
>>> memory management.
>>> I noticed that people promoting ARC do not provide any disadvantages for
>>> proposed approach.
>>>
>>> The thing is in gamedev and other soft-realitime software background
>>> only a handfull types of resources are really managed by RC and memory
>>> usage patterns are VERY specific to their domain (mostly linear
>>> allocation/deallocation and objects with non deterministic lifetime are
>>> preallocated in pools).
>>>
>>> Trying to use RC as a general method of memory management leads to some
>>> problems.
>>> A pretty detailed view by John Harrop (He is somewhat known for trolling
>>> in PL community, but nonetheless knows what he is talking about) -
>>> http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>>>
>>>
>>>
>>> So RC could also introduce unpredictable pause times at undesired
>>> places.
>>>
>>> This is also confirmed by research from HP -
>>> http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>>>
>>> My point is that we should not ruin the language ease of use. We do need
>>> to deal with Phobos internal allocations, but we should not switch to
>>> ARC as a default memory management scheme. In practice people promoting
>>> ARC will probably not use phobos anyway. Currently its just an excuse to
>>> not use D.
>>>
>>> Look at c++ and STL, etc. People will roll their own solutions no matter
>>> what you try.
>>>
>>
>> Full ACK! Reference counting should be well supported, but it
>> shouldn't be the default scheme or built-in at a low level. From my
>> personal experience it would be ideal to be able to customize certain
>> types to be reference counted (allowing the user full flexibility
>> implementing the actual reference counting and without ruling out weak
>> references!), but have them accessible using the same syntax and type
>> conversion semantics as normal references.
>
> I think the best way forward would be to look at the places in D where
> allocations happen, and then figure out how we can optionally allow
> reference counting in these situations. Andrei just made a thread on
> this yesterday in regard to slices, which I think are the most promising
> for a RC solution.

I'm just not convinced (far from it) that Phobos should be built on top of such an RCSlice type. I rather strongly agree with Dicebot that the API should be extended to work with ranges or pre-allocated buffers where possible + support for custom allocators where it makes sense. How the memory is managed is then totally up to the user and no Phobos function needs to be aware of that (e.g. just pass in a pre-allocated, reference counted slice).

Am Thu, 6 Feb 2014 14:37:59 +0300
schrieb Max Klyga <max.klyga@gmail.com>:

> 
> My point is that we should not ruin the language ease of use. We do need to deal with Phobos internal allocations, but we should not switch to ARC as a default memory management scheme.

What's with all this finger pointing and drawing battle lines in the last few days? GC-crowd vs ARC-crowd? Can we please all calm down?

Who even proposed to replace the GC completely with ARC? Can somebody point me to a clear statement demanding that?

90% of phobos isn't actually affected by the ARC/GC issue. Most functions just allocate memory, then return some result and are finished. They do not keep data references. As they do not keep references there's no need for ARC or GC, we just need a way to tell every function how it should allocate.

Some people seem to want some implicit way to set a 'default' allocator, but I haven't heard of any solution that works. (E.g. having a thread-local default allocator, per library default allocator, how would that even work?)

I don't think there's anything wrong with the obvious solution: All phobos functions which allocate take an optional Allocator parameter, defaulting to GC. The little extra typing won't harm anyone and if you want to use things like stack-based buffers you'll have to write extra code and think about memory allocation anyway.

auto gcString = toUpper("test");
auto mallocString = toUpper!Malloc("test");
ubtye[64] sbuf;
auto stackString = toUpper(sbuf[], "test");

What's so bad about this? It works for most of phobos, doesn't require language changes and it's easy to realize what's going on when reading the code. Having an 'application default allocator' or 'thread local default allocator' or 'per function default allocator' will actually hide the allocation strategy and I bet it would cause issues.


So the question then is: what about language feature which allocate using the GC? Wouldn't we want these to work with any kind of allocator? Answer: no, because:

This is the list of language features which allocate:

* .sort, .idup, .dup, setting .length

Sort is deprecated; we should provide duplicate!Allocator functions as a replacement for dup/idup (Or dup/idup could support an allocator argument); just don't set .length. If you need some way to grow an Array just use Appender or a library array, ...

* closures

Who needs these anyway? If the callees only use scoped delegates closures do not allocate. Otherwise just implement the closure yourself and allocate wherever you want:

int localA, localB;
struct Frame
{
    int a, b;
    int callback() {a++};
}
auto f = allocate!(Frame, Malloc)(localA, localB);
functionWithDelegate(&f.callback);

* ~, ~= on slices (not on user types)

Just avoid these. When string processing a call to format!Allocator would be better anyway. For other stuff use Appender!Allocator or some other library type which could actually overload ~,~= and then you can use these again...

* delete

deprecated

* new

we need a generic allocation function anyway, allocate!Allocator

* Array literals
* Associative array literals (not in all cases)

That should be fixed. I hope at some point these types will be implemented in druntime/phobos, support allocators and don't need TypeInfo. Until then, just use user defined Array, AssociativeArray types.


I think with relatively little effort we could solve most problems. The remaining cases must be decided on a case-by case basis. Should containers use RC or GC, some stuff like that.

Exceptions currently can't work on a system without GC cause we always use 'throw new' and nobody ever explicitly frees Exceptions. ARC could be a solution to that issue (if we enforce that exceptions may not have circular references, but they shouldn't anyway)

And then slices which are actually stored by functions are another issue.

But it's not like we just change the whole language to ARC. We already have forced RC in phobos: std.stdio.File for example. And nobody complained, except that RefCounted has bugs and an ARC implementation for File could avoid these bugs and be faster than the current implementation...

We just have to provide everyone with a way to choose their favorite implementation. Which means we provide public APIs which allow any kind of memory allocation and internally do not rely on automatic memory management (internal allocation in phobos should be done on the stack/ with malloc / made configurable, but not with a GC).

On 2/6/14, 7:11 AM, ponce wrote:
> On Thursday, 6 February 2014 at 11:37:59 UTC, Max Klyga wrote:
>> Anti-GC crowd tries to promote ARC as an deterministic alternative for
>> memory management.
>> I noticed that people promoting ARC do not provide any disadvantages
>> for proposed approach.
>>
>> The thing is in gamedev and other soft-realitime software background
>> only a handfull types of resources are really managed by RC and memory
>> usage patterns are VERY specific to their domain (mostly linear
>> allocation/deallocation and objects with non deterministic lifetime
>> are preallocated in pools).
>>
>> Trying to use RC as a general method of memory management leads to
>> some problems.
>> A pretty detailed view by John Harrop (He is somewhat known for
>> trolling in PL community, but nonetheless knows what he is talking
>> about) -
>> http://www.quora.com/Computer-Programming/How-do-reference-counting-and-garbage-collection-compare/answer/Jon-Harrop-1?srid=3Gvg&share=1#
>>
>>
>>
>> So RC could also introduce unpredictable pause times at undesired places.
>>
>> This is also confirmed by research from HP -
>> http://www.hpl.hp.com/personal/Hans_Boehm/popl04/refcnt.pdf
>>
>> My point is that we should not ruin the language ease of use. We do
>> need to deal with Phobos internal allocations, but we should not
>> switch to ARC as a default memory management scheme. In practice
>> people promoting ARC will probably not use phobos anyway. Currently
>> its just an excuse to not use D.
>>
>> Look at c++ and STL, etc. People will roll their own solutions no
>> matter what you try.
>
> I think of RC as a greater evil that GC. From what I've seen it does
> creates leaking cycles in tree structures AND pauses. From a low-level
> point of view, RC pointers are ugly (separate counter that will trash
> your cache) and do atomics all over the place (ie. memory barriers).
> That they don't have to because of shared is TBD.
>
> It is comforting to me to know that a GC pointer is still just a
> pointer. If we go the RC route, we will have to constantly think about
> the higher cost of RC pointer vs weak-ref, instead of thinking about
> other things instead
> Right now a GC pointer is the same size as a non-GC one, it's liberating.
>
> It looks like we want to solve a PR problem more than a real one.

Though I partly agree with your considerations, let me note that PR problems are real.

Andrei