grauzone wrote:
> Christopher Wright wrote:
>> grauzone wrote:
>>> Jarrett Billingsley wrote:
>>>> On Sun, Mar 29, 2009 at 4:42 PM, Leandro Lucarella <llucax@gmail.com> wrote:
>>>>> This was discussed several times in the past. For example:
>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/weak_references_13301.html 
>>>>>
>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/Soft_weak_references_8264.html 
>>>>>
>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/announce/ANN_WeakObjectReference_-_class_to_hold_weak_references_9103.html 
>>>>>
>>>>> etc.
>>>>>
>>>>> I hope it helps.
>>>>
>>>> The one provided by Bill:
>>>>
>>>> http://www.dsource.org/projects/scrapple/browser/trunk/weakref
>>>>
>>>> seems to work fine, and has the advantage of working in both Phobos and Tango.
>>>
>>> First, I doubt this actually works. The WeakRef stores the pointer as size_t, but the GC is conservative and will still recognize the size_t as a pointer. The unittest in the existing code only works, because he uses an explicit delete on the referenced object.
>>
>> If the WeakRef is on the stack, this is true.
>>
>> If the WeakRef is part of an aggregate type that contains pointers, this is true.
> 
> If WeakRef is a class, this is also true. Because all objects contain a hidden monitor pointer, and the monitor is subject to garbage collection AFAIK.
> 
>> Otherwise, the GC will see that the relevant block is marked as having no pointers.
>>
>> True -- weakref is a difficult thing to make thread-safe.
> 
> It seems there's still work to do, and a thread-safe WeakRef can't be created with the current interfaces. Is this true?
> 
> I'm thinking rt_attachDisposeEvent() should take a *pointer* to the reference instead of the reference itself (effectively a double pointer), and clear this pointer during garbage collection, when all threads are still globally locked.

Hold on, I think this isn't a real issue.

You have essentially:
class WeakRef
{
	const xor = 0x101010101;
	size_t value;
	bool collected;
	Object toObject()
	{
		if (collected) return null;
		auto ptr = cast(void*)(value ^ xor);
		auto obj = *cast(Object*)&ptr;
		if (collected) return null;
		return obj;
	}
}


After casting, you have a strong reference to the object, so it's impossible for the object to be collected if it hasn't yet been collected. If the object was collected between the two checks, you just have an invalid object reference that nobody ever uses. As long as the compiler doesn't do nasty reordering things, you should be fine.

On Tue, Mar 31, 2009 at 7:37 AM, Christopher Wright <dhasenan@gmail.com> wrote:
> grauzone wrote:
>>
>> Christopher Wright wrote:
>>>
>>> grauzone wrote:
>>>>
>>>> Jarrett Billingsley wrote:
>>>>>
>>>>> On Sun, Mar 29, 2009 at 4:42 PM, Leandro Lucarella <llucax@gmail.com> wrote:
>>>>>>
>>>>>> This was discussed several times in the past. For example:
>>>>>>
>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/weak_references_13301.html
>>>>>>
>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/Soft_weak_references_8264.html
>>>>>>
>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/announce/ANN_WeakObjectReference_-_class_to_hold_weak_references_9103.html
>>>>>> etc.
>>>>>>
>>>>>> I hope it helps.
>>>>>
>>>>> The one provided by Bill:
>>>>>
>>>>> http://www.dsource.org/projects/scrapple/browser/trunk/weakref
>>>>>
>>>>> seems to work fine, and has the advantage of working in both Phobos and Tango.
>>>>
>>>> First, I doubt this actually works. The WeakRef stores the pointer as size_t, but the GC is conservative and will still recognize the size_t as a pointer. The unittest in the existing code only works, because he uses an explicit delete on the referenced object.
>>>
>>> If the WeakRef is on the stack, this is true.
>>>
>>> If the WeakRef is part of an aggregate type that contains pointers, this is true.
>>
>> If WeakRef is a class, this is also true. Because all objects contain a hidden monitor pointer, and the monitor is subject to garbage collection AFAIK.
>>
>>> Otherwise, the GC will see that the relevant block is marked as having no pointers.
>>>
>>> True -- weakref is a difficult thing to make thread-safe.
>>
>> It seems there's still work to do, and a thread-safe WeakRef can't be created with the current interfaces. Is this true?
>>
>> I'm thinking rt_attachDisposeEvent() should take a *pointer* to the reference instead of the reference itself (effectively a double pointer), and clear this pointer during garbage collection, when all threads are still globally locked.
>
> Hold on, I think this isn't a real issue.
>
> You have essentially:
> class WeakRef
> {
>        const xor = 0x101010101;
>        size_t value;
>        bool collected;
>        Object toObject()
>        {
>                if (collected) return null;
>                auto ptr = cast(void*)(value ^ xor);
>                auto obj = *cast(Object*)&ptr;
>                if (collected) return null;
>                return obj;
>        }
> }
>
>
> After casting, you have a strong reference to the object, so it's impossible for the object to be collected if it hasn't yet been collected. If the object was collected between the two checks, you just have an invalid object reference that nobody ever uses. As long as the compiler doesn't do nasty reordering things, you should be fine.

Pardon my ignorance, but how do you know that ptr ^ xor is not a pointer to some other real object in memory?   Does the language give you some guarantee that it won't ever be a valid pointer?

--bb

Sun, 29 Mar 2009 17:42:48 -0400, Chad J wrote:

> Simon TRENY wrote:
>> Hello,
>> 
>> I have a class "A" and I'd like to keep a list of all the created instances of this class. To do that, I have a static List!(A) in the A class and, in the constructor, I add each new instance to this list. This gives me the following code:
>> 
>> class A {
>>    private static List!(A) s_instances;
>> 
>>    public this() {
>>       s_instances.add(this);
>>    }
>> 
>>    public ~this() {
>>       s_instances.remove(this);
>>    }
>> 
>>    public static void printAll() {
>>       foreach (A instance; s_instances)
>>          print(instance.toString());
>>    }
>> }
>> 
>> But then, since all the instances are referenced by the static list, they are never garbage-collected, which could be a problem. In some other languages, this can be solved using weak references, but I haven't found any informations about using weak references in D. Is there any way to solve this problem?
>> 
>> Thanks,
>> Simon
>> 
> 
> Maybe what you are looking for are the GC.removeRoot or GC.removeRange functions which are available in both Phobos and Tango? http://www.dsource.org/projects/tango/docs/current/tango.core.Memory.html http://www.digitalmars.com/d/2.0/phobos/std_gc.html http://www.digitalmars.com/d/1.0/phobos/std_gc.html

You can remove only something previously added.  Since a static array is not a root nor a range of roots, you can't make it invisible to GC this way.

== Quote from Sergey Gromov (snake.scaly@gmail.com)'s article
> Sun, 29 Mar 2009 17:42:48 -0400, Chad J wrote:
> >
> > Maybe what you are looking for are the GC.removeRoot or GC.removeRange functions which are available in both Phobos and Tango? http://www.dsource.org/projects/tango/docs/current/tango.core.Memory.html http://www.digitalmars.com/d/2.0/phobos/std_gc.html http://www.digitalmars.com/d/1.0/phobos/std_gc.html
> You can remove only something previously added.  Since a static array is not a root nor a range of roots, you can't make it invisible to GC this way.

What you could do is turn off the HAS_POINTERS flag for the WeakPtr
object, assuming it's set to begin with (if your class just contains a size_t
for the pointer then it might not be).  This should avoid the need for
any weird tricks to obscure the pointer data.  On D2 you do this via:

GC.setAttr(myWeakPtr, GC.BlkAttr.NO_SCAN);

Bill Baxter wrote:
> On Tue, Mar 31, 2009 at 7:37 AM, Christopher Wright <dhasenan@gmail.com> wrote:
>> grauzone wrote:
>>> Christopher Wright wrote:
>>>> grauzone wrote:
>>>>> Jarrett Billingsley wrote:
>>>>>> On Sun, Mar 29, 2009 at 4:42 PM, Leandro Lucarella <llucax@gmail.com>
>>>>>> wrote:
>>>>>>> This was discussed several times in the past. For example:
>>>>>>>
>>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/weak_references_13301.html
>>>>>>>
>>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/learn/Soft_weak_references_8264.html
>>>>>>>
>>>>>>> http://www.digitalmars.com/d/archives/digitalmars/D/announce/ANN_WeakObjectReference_-_class_to_hold_weak_references_9103.html
>>>>>>> etc.
>>>>>>>
>>>>>>> I hope it helps.
>>>>>> The one provided by Bill:
>>>>>>
>>>>>> http://www.dsource.org/projects/scrapple/browser/trunk/weakref
>>>>>>
>>>>>> seems to work fine, and has the advantage of working in both Phobos and
>>>>>> Tango.
>>>>> First, I doubt this actually works. The WeakRef stores the pointer as
>>>>> size_t, but the GC is conservative and will still recognize the size_t as a
>>>>> pointer. The unittest in the existing code only works, because he uses an
>>>>> explicit delete on the referenced object.
>>>> If the WeakRef is on the stack, this is true.
>>>>
>>>> If the WeakRef is part of an aggregate type that contains pointers, this
>>>> is true.
>>> If WeakRef is a class, this is also true. Because all objects contain a
>>> hidden monitor pointer, and the monitor is subject to garbage collection
>>> AFAIK.
>>>
>>>> Otherwise, the GC will see that the relevant block is marked as having no
>>>> pointers.
>>>>
>>>> True -- weakref is a difficult thing to make thread-safe.
>>> It seems there's still work to do, and a thread-safe WeakRef can't be
>>> created with the current interfaces. Is this true?
>>>
>>> I'm thinking rt_attachDisposeEvent() should take a *pointer* to the
>>> reference instead of the reference itself (effectively a double pointer),
>>> and clear this pointer during garbage collection, when all threads are still
>>> globally locked.
>> Hold on, I think this isn't a real issue.
>>
>> You have essentially:
>> class WeakRef
>> {
>>        const xor = 0x101010101;
>>        size_t value;
>>        bool collected;
>>        Object toObject()
>>        {
>>                if (collected) return null;
>>                auto ptr = cast(void*)(value ^ xor);
>>                auto obj = *cast(Object*)&ptr;
>>                if (collected) return null;
>>                return obj;
>>        }
>> }
>>
>>
>> After casting, you have a strong reference to the object, so it's impossible
>> for the object to be collected if it hasn't yet been collected. If the
>> object was collected between the two checks, you just have an invalid object
>> reference that nobody ever uses. As long as the compiler doesn't do nasty
>> reordering things, you should be fine.

Ooops, that looks OK. This makes me happy; at least I can use weak pointers in my own program now (and remove that nasty hack that emulated them).

> Pardon my ignorance, but how do you know that ptr ^ xor is not a
> pointer to some other real object in memory?   Does the language give
> you some guarantee that it won't ever be a valid pointer?

Not really, but it could be "good enough". Here's another way to trick the GC in a more reliable way:

union hide_ptr {
	void* ptr;
	byte[4] stuff;
}

class WeakRef {
	byte[5] hidden_pointer;
	this(void* ptr) {
		hide_ptr p;
		p.ptr = ptr;
		hidden_pointer[1..5] = p.stuff;
	}
}

This should work because the GC expects all pointers to be on 4 byte aligned boundaries (or 8 bytes in 64 bit mode).

> --bb

Sean Kelly wrote:
> == Quote from Sergey Gromov (snake.scaly@gmail.com)'s article
>> Sun, 29 Mar 2009 17:42:48 -0400, Chad J wrote:
>>> Maybe what you are looking for are the GC.removeRoot or GC.removeRange
>>> functions which are available in both Phobos and Tango?
>>> http://www.dsource.org/projects/tango/docs/current/tango.core.Memory.html
>>> http://www.digitalmars.com/d/2.0/phobos/std_gc.html
>>> http://www.digitalmars.com/d/1.0/phobos/std_gc.html
>> You can remove only something previously added.  Since a static array is
>> not a root nor a range of roots, you can't make it invisible to GC this
>> way.
> 
> What you could do is turn off the HAS_POINTERS flag for the WeakPtr
> object, assuming it's set to begin with (if your class just contains a size_t
> for the pointer then it might not be).  This should avoid the need for
> any weird tricks to obscure the pointer data.  On D2 you do this via:
> 
> GC.setAttr(myWeakPtr, GC.BlkAttr.NO_SCAN);

Wouldn't this code break it?

synchronized(mWeakPtr) {}

The GC would collect the monitor in the next GC run, and you had a dangling pointer. But then I don't really know enough about how objects are GC-scanned and how monitors are memory managed.

Actually, scratch that. Any 4 byte pattern can look like a pointer. Unless you manage to encode it in a way the 4 byte cells look like they're pointing into an address range not managed by the GC. For example, most OSes reserve the last 1 or 2 GBs for the kernel. If your byte quadruple looks like a pointer into the kernel, it's always safe not to be scanned.

The only simple and reliable way is to store the pointer in a malloc'ed or NO_SCAN'ed memory area.

grauzone wrote:
> Sean Kelly wrote:
>> == Quote from Sergey Gromov (snake.scaly@gmail.com)'s article
>>> Sun, 29 Mar 2009 17:42:48 -0400, Chad J wrote:
>>>> Maybe what you are looking for are the GC.removeRoot or GC.removeRange
>>>> functions which are available in both Phobos and Tango?
>>>> http://www.dsource.org/projects/tango/docs/current/tango.core.Memory.html 
>>>>
>>>> http://www.digitalmars.com/d/2.0/phobos/std_gc.html
>>>> http://www.digitalmars.com/d/1.0/phobos/std_gc.html
>>> You can remove only something previously added.  Since a static array is
>>> not a root nor a range of roots, you can't make it invisible to GC this
>>> way.
>>
>> What you could do is turn off the HAS_POINTERS flag for the WeakPtr
>> object, assuming it's set to begin with (if your class just contains a size_t
>> for the pointer then it might not be).  This should avoid the need for
>> any weird tricks to obscure the pointer data.  On D2 you do this via:
>>
>> GC.setAttr(myWeakPtr, GC.BlkAttr.NO_SCAN);
> 
> Wouldn't this code break it?
> 
> synchronized(mWeakPtr) {}
> 
> The GC would collect the monitor in the next GC run, and you had a dangling pointer. But then I don't really know enough about how objects are GC-scanned and how monitors are memory managed.

The monitor is allocated via malloc().  But you're right that if this were allocated from the GC then it would result in a dangling pointer (which is possible since monitors can be overridden by the user).