On Jan 16, 2011, at 9:54 AM, Rainer Schuetze wrote:

> Hi,
> 
> I have not updated to the latest version yet, but it seems that it's the first time that gc_addRange is called during initialization, and this hits the bug in the gc stub code (obviously, the return statements are missing for the "proxy is null" case).
> 
> Even if fixed, the proxy has the problem that anything that has been allocated until the proxy is switched, uses a different heap, because the C-Runtime is not shared between the DLLs. This will cause problems when trying to scan/collect objects.

Allocating memory from the GC before initializing the runtime results in undefined behavior.  If you're using gcstub though, the only thing that won't work is that allocated blocks won't be scanned for roots by the new GC, but this behavior isn't expected anyway.  The new GC doesn't own the memory allocated by the old GC.  Also, The roots and ranges are already transferred to the new GC. (see gc_setProxy), so the static data segment of the DLL will be scanned by the app's GC.

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/d-runtime/attachments/20110117/7838e39c/attachment.html>

On Jan 17, 2011, at 12:34 PM, Walter Bright wrote:
> 
> Sean Kelly wrote:
>> On Jan 16, 2011, at 9:54 AM, Rainer Schuetze wrote:
>> 
>> 
>> 
>>> Hi,
>>> 
>>> I have not updated to the latest version yet, but it seems that it's the first time that gc_addRange is called during initialization, and this hits the bug in the gc stub code (obviously, the return statements are missing for the "proxy is null" case).
>>> 
>>> Even if fixed, the proxy has the problem that anything that has been allocated until the proxy is switched, uses a different heap, because the C-Runtime is not shared between the DLLs. This will cause problems when trying to scan/collect objects.
>>> 
>>> 
>> 
>> Allocating memory from the GC before initializing the runtime results in undefined behavior.  If you're using gcstub though, the only thing that won't work is that allocated blocks won't be scanned for roots by the new GC, but this behavior isn't expected anyway.  The new GC doesn't own the memory allocated by the old GC.  Also, The roots and ranges are already transferred to the new GC. (see gc_setProxy), so the static data segment of the DLL will be scanned by the app's GC.
>> 
>> 
> 
> Here's the code for rt_init(), which is called by LoadLibrary(), i.e. using gcstub instead of the gc. It calls _moduleCtor() and _moduleTlsCtor(). Therefore, any memory allocated by any static constructors in the DLL will not be scanned for roots.
> 
> extern (C) bool rt_init(ExceptionHandler dg = null)
> {
>     _d_criticalInit();
> 
>     try
>     {
>         gc_init();
>         initStaticDataGC();
>         version (Windows)
>             _minit();
>         _moduleCtor();
>         _moduleTlsCtor();
>         runModuleUnitTests();
>         return true;
>     }
>     catch (Throwable e)
>     {
>         if (dg)
>             dg(e);
>         else
>             throw e;    // rethrow, don't silently ignore error
>     }
>     _d_criticalTerm();
>     return false;
> }

Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.

I can think of two solutions, though neither are terribly appealing.  The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).

Thoughts?

On Mon, 17 Jan 2011, Sean Kelly wrote:

> Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.
> 
> I can think of two solutions, though neither are terribly appealing. The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).
> 
> Thoughts?

#3) require that d dll's call the appropriate registration function first. Any user ctors must be 2-N.

Brad Roberts wrote:
> On Mon, 17 Jan 2011, Sean Kelly wrote:
>
> 
>> Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.
>>
>> I can think of two solutions, though neither are terribly appealing. The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).
>>
>> Thoughts?
>> 
>
> #3) require that d dll's call the appropriate registration function first. Any user ctors must be 2-N.
>
> 

I agree with Brad. The only sure way to make it work is to have an initialization function, D_Initialize_Druntime(params...), that is required before accessing the functionality of the DLL. One of those params should be the gc proxy. This still works with a non-D caller.

On Jan 17, 2011, at 5:07 PM, Walter Bright wrote:
> 
> Brad Roberts wrote:
>> On Mon, 17 Jan 2011, Sean Kelly wrote:
>> 
>> 
>>> Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.
>>> 
>>> I can think of two solutions, though neither are terribly appealing.  The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).
>>> 
>>> Thoughts?
>>> 
>> 
>> #3) require that d dll's call the appropriate registration function first.  Any user ctors must be 2-N.
>> 
>> 
> 
> I agree with Brad. The only sure way to make it work is to have an initialization function, D_Initialize_Druntime(params...), that is required before accessing the functionality of the DLL. One of those params should be the gc proxy. This still works with a non-D caller.

The unfortunate part is that this isn't necessary when using Runtime.loadLibrary(), since that will do the right thing.  It would only be necessary for a non-D hosting app.  I guess if it has an "already ran" flag in it then the library writer could have his init_my_lib() routine call it instead.  *sigh*  No automatic alternatives, huh?

Sean Kelly wrote:
> The unfortunate part is that this isn't necessary when using Runtime.loadLibrary(), since that will do the right thing.  It would only be necessary for a non-D hosting app.  I guess if it has an "already ran" flag in it then the library writer could have his init_my_lib() routine call it instead.  *sigh*  No automatic alternatives, huh?

What about creating a DLL that just contains the GC and exports the proxy pointer? Any DLL/EXE that wants to use the shared GC links against the import library of this DLL and sets its proxy pointer before any other initialisation.

In case of multi threading, a bit more needs to done to forward TLS ranges to the proxy. But this will probably not work anyway, because the threads also need to be shared.

So, better create a druntime.dll/phobos.dll to share everything...

Rainer

On Jan 18, 2011, at 11:37 AM, Rainer Schuetze wrote:

> Sean Kelly wrote:
>> The unfortunate part is that this isn't necessary when using Runtime.loadLibrary(), since that will do the right thing.  It would only be necessary for a non-D hosting app.  I guess if it has an "already ran" flag in it then the library writer could have his init_my_lib() routine call it instead.  *sigh*  No automatic alternatives, huh?
> 
> What about creating a DLL that just contains the GC and exports the proxy pointer? Any DLL/EXE that wants to use the shared GC links against the import library of this DLL and sets its proxy pointer before any other initialisation.
> 
> In case of multi threading, a bit more needs to done to forward TLS ranges to the proxy. But this will probably not work anyway, because the threads also need to be shared.
> 
> So, better create a druntime.dll/phobos.dll to share everything...

A druntime/phobos DLL would be a clean way to handle this, though shared libraries complicate distribution of user code.  Although, didn't Visual Studio eliminate the option of statically linking its runtime library?  We'd still need a reasonable short-term solution though.  I don't even know what would be required to ship druntime/phobos in a DLL, but I'm guessing it's more than a few simple changes.  We'd probably need export labels all over the place, etc, right?

On 2011-01-18 17:16, Sean Kelly wrote:
> On Jan 17, 2011, at 5:07 PM, Walter Bright wrote:
>>
>> Brad Roberts wrote:
>>> On Mon, 17 Jan 2011, Sean Kelly wrote:
>>>
>>>
>>>> Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.
>>>>
>>>> I can think of two solutions, though neither are terribly appealing.  The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).
>>>>
>>>> Thoughts?
>>>>
>>>
>>> #3) require that d dll's call the appropriate registration function first.  Any user ctors must be 2-N.
>>>
>>>
>>
>> I agree with Brad. The only sure way to make it work is to have an initialization function, D_Initialize_Druntime(params...), that is required before accessing the functionality of the DLL. One of those params should be the gc proxy. This still works with a non-D caller.
>
> The unfortunate part is that this isn't necessary when using Runtime.loadLibrary(), since that will do the right thing.  It would only be necessary for a non-D hosting app.  I guess if it has an "already ran" flag in it then the library writer could have his init_my_lib() routine call it instead.  *sigh*  No automatic alternatives, huh?

I'm just dropping in here in the middle of the discussion, don't know if this helps. Windows doesn't have any way to register a callback to the loader/dynamic linker? Something like the "_dyld_register_func_for_add_image" function that is available on Mac OS X which can be used to initialize the runtime.

-- 
/Jacob Carlborg

Sean Kelly wrote:
> On Jan 17, 2011, at 5:07 PM, Walter Bright wrote:
> 
>> Brad Roberts wrote:
>> 
>>> On Mon, 17 Jan 2011, Sean Kelly wrote:
>>>
>>> 
>>> 
>>>> Darnit, you're right.  So the ideal scenario would be if the GC proxy could be set before the DLL load event was fired, which I think is impossible.  I also think it's important that a D DLL be usable by both a D and a non-D host app without a recompile, so we can't rely on gc_setProxy() ever being called (which is why I'm not terribly fond of gcstub to begin with).  This problem applies to any GC in the DLL though, so it still needs a solution.
>>>>
>>>> I can think of two solutions, though neither are terribly appealing.  The first would be to require that D DLLs not create threads in their module ctors, and have gc_setProxy() call the module dtors, set the proxy, then re-construct the modules.  The second would be for gc_setProxy() to suspend all threads in the DLL, have the GC pass all of its allocated blocks as roots/ranges to the proxy, and then restart threads in the DLL.  The second seems far more robust, but imposes a rather large functionality requirement on the GC (a pseudo-collection).
>>>>
>>>> Thoughts?
>>>> 
>>>> 
>>> #3) require that d dll's call the appropriate registration function first.  Any user ctors must be 2-N.
>>>
>>> 
>>> 
>> I agree with Brad. The only sure way to make it work is to have an initialization function, D_Initialize_Druntime(params...), that is required before accessing the functionality of the DLL. One of those params should be the gc proxy. This still works with a non-D caller.
>> 
>
> The unfortunate part is that this isn't necessary when using Runtime.loadLibrary(), since that will do the right thing.

It can't do the right thing, as there's no way to pass a parameter to DLL_PROCESS_ATTACH.

>   It would only be necessary for a non-D hosting app.  I guess if it has an "already ran" flag in it then the library writer could have his init_my_lib() routine call it instead.  *sigh*  No automatic alternatives, huh?
>
>