Does (or will) D support weak references?

See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.

Thanks!

Brian Hammond schrieb:
> Does (or will) D support weak references?
> 
> See http://www.python.org/doc/current/lib/module-weakref.html for a
> backgrounder.

At a language level, it is technically not very feasible, since D has to be able to work with conservative garbage collectors, as well as other automatic memory management systems. In Python, it happens to be well implementable because memory is managed by reference- counting garbage collector. But for statically compiled languages, reference-counting has proven to be a major slowdown, expecially when it has to be thread-safe.

There should be other ways to implement caches and such in D. Normally you would have something like a "central repository" for objects of some sort, and you "request" them and "put back", basically doing sort-of manual reference counting, which is though considerably much less overhead than automatic one. There might also be some other ways, e.g. by mangling pointers so that garbage collector doesn't find them - but then the object must maintain a complete list of its users. A library implementation is though definately more tedious to use, would have another advantage performance-wise: it may use a smarter clean-up strategy than a garbage collector and work asynchronously compared to it, thus reducing pauses.

-eye

Ilya Minkov wrote:

<snip>
> But for statically compiled languages, reference-counting has proven to be a major slowdown, expecially when it has to be thread-safe.

And a cause of memory leaks, when you try to use circular data structures.

> There should be other ways to implement caches and such in D. Normally you would have something like a "central repository" for objects of some sort, and you "request" them and "put back", basically doing sort-of manual reference counting, which is though considerably much less overhead than automatic one. There might also be some other ways, e.g. by mangling pointers so that garbage collector doesn't find them - but then the object must maintain a complete list of its users.
<snip>

I thought of writing a class that would do something like that.  The trouble is that we'd need a means of determining if an object is reachable by strong references.  The only way we can really get near doing this is to have a common base class for weak-referenceable objects, which would clear the references in its destructor.  A compromise would be to create an object wrapper enabling something to be WR'd, but that might lead to WRs being cleared while the object is still active.

But with the base class proviso, a possibility is to assign a serial number to each object when it is first WR'd, and have the WR object hold this number.  There would be a static mapping table of some sort between
serial numbers and mangled pointers.  When it's time to clear the WRs, it would simply set the pointer in the table to null.  The only time you're going to run into trouble is if you WR more objects in the course of your program than the type of the serial number'll allow.

Stewart.

-- 
My e-mail is valid but not my primary mailbox, aside from its being the unfortunate victim of intensive mail-bombing at the moment.  Please keep replies on the 'group where everyone may benefit.

In article <c80726$1qq$1@digitaldaemon.com>, Stewart Gordon says... ..
>
>I thought of writing a class that would do something like that.  The trouble is that we'd need a means of determining if an object is reachable by strong references.  The only way we can really get near doing this is to have a common base class for weak-referenceable objects, which would clear the references in its destructor.  A compromise would be to create an object wrapper enabling something to be WR'd, but that might lead to WRs being cleared while the object is still active.

The boehm-wheeler gc (and probably most others) has a provision to add
non-scanned areas.  (In D, malloced space is non-scanned, right?)  If you add
one of these areas (needs some language support) and put an array of weak
pointers there, then the only piece missing is a way for the GC to clear
pointers (from that area) for objects that it collects.

So: Weak refs are a tiny 'smart pointer' like object that points at a slot in the table (or indexes into the unscanned space).  That slot has a non-scanned pointer to the real object.

When the GC is done reaping the heap, it does one last mark scan: it runs over the weak ref table:  instead of marking objects as "not garbage", it just zeroes the weak pointers to them.  If the object is already marked as "not garbage" it leaves the pointer alone (there must be a strong ref).

- Extra accesses are needed to use weak refs (its a pointer to a pointer).

- This is okay, because weak pointers are rarely used except for this kind of caching.  Once you fetch the regular pointer, its as fast as any other.

(Side note: The standard application for weak references (caching) seems a bit "small" to me.  It seems like the number of objects kept in-cache should be performance tuned, not cleared every time the GC gets hungry, right?  You can keep a fixed number of regular pointers, plus the weak set; but the weak set only helps for cache elements that are past the end of this initial size.  The non-weak portion of the cache is probably as large as you can afford to begin with, so..)

Kevin

> (Side note: The standard application for weak references (caching) seems a
> bit
> "small" to me.  It seems like the number of objects kept in-cache should
> be
> performance tuned, not cleared every time the GC gets hungry, right?  You
> can keep a fixed number of regular pointers, plus the weak set; but the
> weak set
> only helps for cache elements that are past the end of this initial size.
> The non-weak portion of the cache is probably as large as you can afford
> to begin with, so..)

I agree - I've only seen weak references used to manage caches. It would be neat to have GC callbacks where an object can register a delegate that runs when the GC is about to full-collect. That way a cache can clear itself out (or clear out the extra elements of the cache) without needing weak references. I guess you'd have to be careful not to leak the cache since the GC would then have a reference to it.

-Ben

"Brian Hammond" <d@brianhammond.com> wrote in message news:c7grjd$1fq9$1@digitaldaemon.com...
> Does (or will) D support weak references?
>
> See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.

Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references.

So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.

Thanks for the response.  I was thinking of weakrefs as an exploitable feature for a resource manager I am writing for a game engine in D.

Weak refs seemed like a good fit for this problem.  The resource manager maintains a cache of weak references to resource instances.  The manager knows they exist but doesn't stop the gc of such from happening by holding (strong) references to the resource instances.

Thanks, Brian

In article <c8440l$2j3e$1@digitaldaemon.com>, Walter says...
>
>
>"Brian Hammond" <d@brianhammond.com> wrote in message news:c7grjd$1fq9$1@digitaldaemon.com...
>> Does (or will) D support weak references?
>>
>> See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.
>
>Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references.
>
>So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.
>
>

I would think you can get essentially the same effect in D via destructors:

class Cache {
..
delete(int key) {...}
};

class GCwatch {
int key;
Cache cache;

this(int k, Cache c)
{
key = k;
cache = c;
}

~this()
{
Cache.delete(key);
}
};

add_weak_elem()
{
Cache c = new Cache;
int k = c.get_weak_elem();

new GCwatch(c,k);
..
}

The GCwatch is not held by a strong pointer, so the GC will delete it, causing Cache cleanups to happen at the same time as GC cleanups.

Kevin

In article <c896ua$quh$1@digitaldaemon.com>, Brian Hammond <d at brianhammond dot com> says...
>
>Thanks for the response.  I was thinking of weakrefs as an exploitable feature for a resource manager I am writing for a game engine in D.
>
>Weak refs seemed like a good fit for this problem.  The resource manager maintains a cache of weak references to resource instances.  The manager knows they exist but doesn't stop the gc of such from happening by holding (strong) references to the resource instances.
>
>Thanks, Brian
>
>In article <c8440l$2j3e$1@digitaldaemon.com>, Walter says...
>>
>>
>>"Brian Hammond" <d@brianhammond.com> wrote in message news:c7grjd$1fq9$1@digitaldaemon.com...
>>> Does (or will) D support weak references?
>>>
>>> See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.
>>
>>Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references.
>>
>>So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.
>>
>>
>
>

I should amend this: the cached object needs a pointer to the GCwatch, to prevent it from being deleted when a strong pointer exists, and the Cache needs to disguise the pointer to the element.  Also the method should not be called "delete".

Kevin

In article <c8ambr$2lq$1@digitaldaemon.com>, Kevin Bealer says...
>
>
>I would think you can get essentially the same effect in D via destructors:
>
>class Cache {
>..
>delete(int key) {...}
>};
>
>class GCwatch {
>int key;
>Cache cache;
>
>this(int k, Cache c)
>{
>key = k;
>cache = c;
>}
>
>~this()
>{
>Cache.delete(key);
>}
>};
>
>add_weak_elem()
>{
>Cache c = new Cache;
>int k = c.get_weak_elem();
>
>new GCwatch(c,k);
>..
>}
>
>The GCwatch is not held by a strong pointer, so the GC will delete it, causing Cache cleanups to happen at the same time as GC cleanups.
>
>Kevin
>
>In article <c896ua$quh$1@digitaldaemon.com>, Brian Hammond <d at brianhammond dot com> says...
>>
>>Thanks for the response.  I was thinking of weakrefs as an exploitable feature for a resource manager I am writing for a game engine in D.
>>
>>Weak refs seemed like a good fit for this problem.  The resource manager maintains a cache of weak references to resource instances.  The manager knows they exist but doesn't stop the gc of such from happening by holding (strong) references to the resource instances.
>>
>>Thanks, Brian
>>
>>In article <c8440l$2j3e$1@digitaldaemon.com>, Walter says...
>>>
>>>
>>>"Brian Hammond" <d@brianhammond.com> wrote in message news:c7grjd$1fq9$1@digitaldaemon.com...
>>>> Does (or will) D support weak references?
>>>>
>>>> See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.
>>>
>>>Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references.
>>>
>>>So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.
>>>
>>>
>>
>>
>
>

Your idea is fine it's just manual.  I was mostly curious if my issue could be solved using weakrefs in D.  As has been pointed out, weakrefs don't apply well to D.  Thus, to implement this, the monitored object must notify interested parties (say a cache) that it's been collected (notification is done in the destructor).

In article <c8ampu$3c0$1@digitaldaemon.com>, Kevin Bealer says...
>
>
>I should amend this: the cached object needs a pointer to the GCwatch, to prevent it from being deleted when a strong pointer exists, and the Cache needs to disguise the pointer to the element.  Also the method should not be called "delete".
>
>Kevin

Keeping an explicit pointer won't keep the object alive if there are no references to it and the GC runs.  E.g.

import std.gc;

class A
{
this()  { printf("A.this() called.\n");  }
~this() { printf("A.~this() called.\n"); }
}

int main(char[][] args)
{
printf("creating a and pointing b at a\n");
A a = new A();
A* b = &a;
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("Nullifying a...\n");
a = null;
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("Running gc...\n");
std.gc.fullCollect();
printf("A has %.*s collected\n", !a ? "been" : "not been");
printf("b's target object is %.*s\n", *b ? "not null" : "null");

printf("end.\n");
return 0;
}

output:

creating a and pointing b at a
A.this() called.
b's target object is not null
Nullifying a...
b's target object is null
Running gc...
A.~this() called.
A has been collected
b's target object is null
end.

Thanks! Brian


>In article <c8ambr$2lq$1@digitaldaemon.com>, Kevin Bealer says...
>>
>>
>>I would think you can get essentially the same effect in D via destructors:
>>
>>class Cache {
>>..
>>delete(int key) {...}
>>};
>>
>>class GCwatch {
>>int key;
>>Cache cache;
>>
>>this(int k, Cache c)
>>{
>>key = k;
>>cache = c;
>>}
>>
>>~this()
>>{
>>Cache.delete(key);
>>}
>>};
>>
>>add_weak_elem()
>>{
>>Cache c = new Cache;
>>int k = c.get_weak_elem();
>>
>>new GCwatch(c,k);
>>..
>>}
>>
>>The GCwatch is not held by a strong pointer, so the GC will delete it, causing Cache cleanups to happen at the same time as GC cleanups.
>>
>>Kevin
>>
>>In article <c896ua$quh$1@digitaldaemon.com>, Brian Hammond <d at brianhammond dot com> says...
>>>
>>>Thanks for the response.  I was thinking of weakrefs as an exploitable feature for a resource manager I am writing for a game engine in D.
>>>
>>>Weak refs seemed like a good fit for this problem.  The resource manager maintains a cache of weak references to resource instances.  The manager knows they exist but doesn't stop the gc of such from happening by holding (strong) references to the resource instances.
>>>
>>>Thanks, Brian
>>>
>>>In article <c8440l$2j3e$1@digitaldaemon.com>, Walter says...
>>>>
>>>>
>>>>"Brian Hammond" <d@brianhammond.com> wrote in message news:c7grjd$1fq9$1@digitaldaemon.com...
>>>>> Does (or will) D support weak references?
>>>>>
>>>>> See http://www.python.org/doc/current/lib/module-weakref.html for a backgrounder.
>>>>
>>>>Weak references always seemed to me to be a massive and brittle kludge to deal with memory management issues ill-suited to garbage collection. Since D allows explicit memory management as an alternative for gc, instead of weak references for gc, use explicit memory management for those references.
>>>>
>>>>So, weak references are not planned for D. I think they are only required for languages that offer no alternative to gc.
>>>>
>>>>
>>>
>>>
>>
>>
>
>