How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code?

extern(C) external(Foo* foo);

struct Foo
{
}

class Bar
{
	Foo foo;
}

main()
{
	Foo* foo = new Foo();

	// Should foo be pinned before the call?
	external(foo);

	Bar bar = new Bar();

	// Should bar be pinned before the call?
	external(&bar.foo);
}


btw, ares seems to provides a moving gc and does have a gc.pin method. Should i consider looking closer at ares? Thanks

Max Samuha wrote:

> How to ensure D code that interoperates with external libraries will work when a moving gc is eventually used with D? Will i need to add lots of pinning/unpinning code?
> 
> extern(C) external(Foo* foo);
> 
> struct Foo
> {
> }
> 
> class Bar
> {
> Foo foo;
> }
> 
> main()
> {
> Foo* foo = new Foo();
> 
> // Should foo be pinned before the call?
> external(foo);
> 
> Bar bar = new Bar();
> 
> // Should bar be pinned before the call?
> external(&bar.foo);
> }
> 
> 
> btw, ares seems to provides a moving gc and does have a gc.pin method. Should i consider looking closer at ares? Thanks

The GC in Ares is currently the same as the one in Phobos, but the interface in Ares should make it possible to easier switch between other types of GC's.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi

Typically, garbage collectors run only on memory allocation/deallocation/similar.  They won't run in the middle of a foreign program... unless it's threaded, then all bets are off.

-[Unknown]


> How to ensure D code that interoperates with external libraries will
> work when a moving gc is eventually used with D? Will i need to add
> lots of pinning/unpinning code?
> 
> extern(C) external(Foo* foo); 
> 
> struct Foo
> {
> }
> 
> class Bar
> {
> 	Foo foo;
> }
> 
> main()
> {
> 	Foo* foo = new Foo();
> 	
> 	// Should foo be pinned before the call?
> 	external(foo);
> 
> 	Bar bar = new Bar();
> 
> 	// Should bar be pinned before the call?
> 	external(&bar.foo);
> }
> 
> 
> btw, ares seems to provides a moving gc and does have a gc.pin method.
> Should i consider looking closer at ares? Thanks

On Sat, 03 Jun 2006 11:09:56 -0700, "Unknown W. Brackets" <unknown@simplemachines.org> wrote:

>Typically, garbage collectors run only on memory allocation/deallocation/similar.  They won't run in the middle of a foreign program... unless it's threaded, then all bets are off.
>
>-[Unknown]
>

Thank you for prompt replies. Most programs will run more than one thread, so it is not guaranteed that garbage collection will not be triggered when control is in external code. What's the best way to ensure that the pointers are valid until control is returned to D code?

version (MovingGC/Ares/whatever)
{
	gc.pin(foo);
}
external(foo);
version (MovingGC/Ares/whatever)
{
	gc.unpin(foo);
}

that's no good. D claims it is compatible with C code but it cannot be true unless either the specs require that D use only non-moving garbage collector or there is a standard way to ensure that pointers to garbage collected memory passed to external functions remain valid while control is in external code. please correct me if i missed something. sorry for my poor English

Well, you are correct that you need to do this for threaded code interfacing with C functions via pointers.

That doesn't mean D can't interface with C code properly, just that it might not properly do it, necessarily, under this circumstance.

The truth of the matter is, D is still in development, and so is the standard library.  This is something that has not yet been provided for, you are right.  But, as yet, no D implementation uses a compacting GC.

I would suggest that a compiler that adds a compacting GC in could, theoretically, detect the passing of a pointer to a C function, and automatically "pin" it before the call.  Unpinning it afterward does not seem practical.

Another option would be to declare the variable which holds the pointer as extern (C).  This could be a clear indication to the compiler that this pointer should not be moved.

However, the above solutions may indeed be impractical.  More thought will be needed, and this thought will no doubt come from Walter.  This is why, as I said, D is still in an alpha stage.

Even so, I would personally just make a note near the affected area, e.g.:

// !!! Passing a pointer to C.

Or however you prefer to make such notes in your convention; possibly noting this specifically as something to check on whenever a new compiler is used.

It may very well be that gc.pin() and gc.unpin() functions will be added to Phobos, in which case you will need to go through these areas and make the changes - just as I had to do with my code when on_scope_success was changed to scope (success), and when === was changed to is.

It may seem like a pain, but the changes are typically minor, and the benefits I see from programming in D vs. C/C++ still far outweigh these problems.  I cannot predict if it's the same for you, of course.

I'm American, so I may have to ask you to forgive me my poor English as well.  Yours is as good as mine, whether that's a compliment or an insult.

-[Unknown]


> Thank you for prompt replies. Most programs will run more than one
> thread, so it is not guaranteed that garbage collection will not be
> triggered when control is in external code. What's the best way to
> ensure that the pointers are valid until control is returned to D
> code?
> 
> version (MovingGC/Ares/whatever)
> {
> 	gc.pin(foo);
> }
> external(foo);
> version (MovingGC/Ares/whatever)
> {
> 	gc.unpin(foo);
> }
> 
> that's no good. D claims it is compatible with C code but it cannot be
> true unless either the specs require that D use only non-moving
> garbage collector or there is a standard way to ensure that pointers
> to garbage collected memory passed to external functions remain valid
> while control is in external code. please correct me if i missed
> something. sorry for my poor English

On Sat, 03 Jun 2006 17:06:03 -0700, "Unknown W. Brackets" <unknown@simplemachines.org> wrote:

>Well, you are correct that you need to do this for threaded code interfacing with C functions via pointers.
>
>That doesn't mean D can't interface with C code properly, just that it might not properly do it, necessarily, under this circumstance.
>
>The truth of the matter is, D is still in development, and so is the standard library.  This is something that has not yet been provided for, you are right.  But, as yet, no D implementation uses a compacting GC.
>
>I would suggest that a compiler that adds a compacting GC in could, theoretically, detect the passing of a pointer to a C function, and automatically "pin" it before the call.  Unpinning it afterward does not seem practical.
>
>Another option would be to declare the variable which holds the pointer as extern (C).  This could be a clear indication to the compiler that this pointer should not be moved.
>
>However, the above solutions may indeed be impractical.  More thought will be needed, and this thought will no doubt come from Walter.  This is why, as I said, D is still in an alpha stage.
>
>Even so, I would personally just make a note near the affected area, e.g.:
>
>// !!! Passing a pointer to C.
>
>Or however you prefer to make such notes in your convention; possibly noting this specifically as something to check on whenever a new compiler is used.
>
>It may very well be that gc.pin() and gc.unpin() functions will be added to Phobos, in which case you will need to go through these areas and make the changes - just as I had to do with my code when on_scope_success was changed to scope (success), and when === was changed to is.
>
>It may seem like a pain, but the changes are typically minor, and the benefits I see from programming in D vs. C/C++ still far outweigh these problems.  I cannot predict if it's the same for you, of course.
>
>I'm American, so I may have to ask you to forgive me my poor English as well.  Yours is as good as mine, whether that's a compliment or an insult.
>
>-[Unknown]

Thank you for the extensive explanation!
Marking my code with '// !!! Passing a pointer to C.' is what i've
been doing recently:) and it seems the best solution for now until
ares is mature or phobos has pin/unpin, or pointers passed to C are
pinned implicitly.

Max Samuha wrote:
> How to ensure D code that interoperates with external libraries will
> work when a moving gc is eventually used with D? Will i need to add
> lots of pinning/unpinning code?
<snip>

Pinning has been requested a few times on these 'groups.  There has also been a discussion before on issues that still need to be addressed before a GC that moves stuff around will work:

http://www.digitalmars.com/d/archives/26273.html

Stewart.

On Tue, 06 Jun 2006 14:27:22 +0100, Stewart Gordon <smjg_1998@yahoo.com> wrote:

>
>Pinning has been requested a few times on these 'groups.  There has also been a discussion before on issues that still need to be addressed before a GC that moves stuff around will work:
>
>http://www.digitalmars.com/d/archives/26273.html
>
>Stewart.

Yes, the thread you mention was posted two years ago. It would be really nice to hear a word from Walter on this issue.

Max Samuha wrote:
> Thank you for the extensive explanation!
> Marking my code with '// !!! Passing a pointer to C.' is what i've
> been doing recently:) and it seems the best solution for now until
> ares is mature or phobos has pin/unpin, or pointers passed to C are
> pinned implicitly.

They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.

Walter Bright wrote:
> Max Samuha wrote:
>> Thank you for the extensive explanation!
>> Marking my code with '// !!! Passing a pointer to C.' is what i've
>> been doing recently:) and it seems the best solution for now until
>> ares is mature or phobos has pin/unpin, or pointers passed to C are
>> pinned implicitly.
> 
> They will be pinned implicitly, no need to call pin/unpin. The only time you're going to have a problem is if the C routine hides the pointer in a malloc'd buffer where the gc won't see it. The gc *will* see pointers on the C stack.

Would it remain pinned indefinitely, until the C function exits, or until the program explicitly unpins it by some means?

Stewart.