Don Clugston wrote:
> I've also had problems with DLLs in the git version of druntime.
> When I use Runtime.initialize() in a thread, in a DLL which was
> dynamically loaded, it crashes. I haven't traced into it, but it feels
> like exactly the same behaviour I saw when TLS wasn't initialized.
> Almost as if something in the runtime is using a TLS variable before
> initialization has happened.

Recently, I also started creating threads in a DLL, and that revealed a
few inconsistencies between dll_helper.d and thread.d. The major problem
is that DLLMain is called before the thread is added to the list of
library created threads, causing initialization to be done twice.
I have a patch for this, but I'm not yet on the latest git-version. I'll
try to update the bugzilla entry
http://d.puremagic.com/issues/show_bug.cgi?id=5582 with it and maybe
even create a pull request.


> I also notice that the docs in 2.052 had an inconsistency: the first example used dll_helper, the later ones use Runtime.initialize().

When updating the DLL init code, I also changed the "DLLs with a C Interface" part in the docs, but as I'm not convinced that the approach used by "D code calling D code in DLLs" is good enough, I left it as is. My suggestion here would be to use a common phobos.DLL to share all runtime library functions including the GC (as shown here: http://d.puremagic.com/issues/show_bug.cgi?id=4071 )

On 20 April 2011 08:47, Rainer Schuetze <r.sagitario at gmx.de> wrote:
>
>
> Don Clugston wrote:
>>
>> I've also had problems with DLLs in the git version of druntime.
>> When I use Runtime.initialize() in a thread, in a DLL which was
>> dynamically loaded, it crashes. I haven't traced into it, but it feels
>> like exactly the same behaviour I saw when TLS wasn't initialized.
>> Almost as if something in the runtime is using a TLS variable before
>> initialization has happened.
>
> Recently, I also started creating threads in a DLL, and that revealed a few
> inconsistencies between dll_helper.d and thread.d. The major problem is that
> DLLMain is called before the thread is added to the list of library created
> threads, causing initialization to be done twice.
> I have a patch for this, but I'm not yet on the latest git-version. I'll try
> to update the bugzilla entry
> http://d.puremagic.com/issues/show_bug.cgi?id=5582 with it and maybe even
> create a pull request.
>
>
>> I also notice that the docs in 2.052 had an inconsistency: the first example used dll_helper, the later ones use Runtime.initialize().
>
> When updating the DLL init code, I also changed the "DLLs with a C Interface" part in the docs, but as I'm not convinced that the approach used by "D code calling D code in DLLs" is good enough, I left it as is. My suggestion here would be to use a common phobos.DLL to share all runtime library functions including the GC (as shown here: http://d.puremagic.com/issues/show_bug.cgi?id=4071 )

The "DLLs with a C interface" part imports core.dll_helper, which no longer exists. What should it be now?

The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.

Sent from my iPhone

On Apr 19, 2011, at 11:55 PM, Don Clugston <dclugston at googlemail.com> wrote:

> On 20 April 2011 08:47, Rainer Schuetze <r.sagitario at gmx.de> wrote:
>> 
>> 
>> Don Clugston wrote:
>>> 
>>> I've also had problems with DLLs in the git version of druntime.
>>> When I use Runtime.initialize() in a thread, in a DLL which was
>>> dynamically loaded, it crashes. I haven't traced into it, but it feels
>>> like exactly the same behaviour I saw when TLS wasn't initialized.
>>> Almost as if something in the runtime is using a TLS variable before
>>> initialization has happened.
>> 
>> Recently, I also started creating threads in a DLL, and that revealed a few
>> inconsistencies between dll_helper.d and thread.d. The major problem is that
>> DLLMain is called before the thread is added to the list of library created
>> threads, causing initialization to be done twice.
>> I have a patch for this, but I'm not yet on the latest git-version. I'll try
>> to update the bugzilla entry
>> http://d.puremagic.com/issues/show_bug.cgi?id=5582 with it and maybe even
>> create a pull request.
>> 
>> 
>>> I also notice that the docs in 2.052 had an inconsistency: the first example used dll_helper, the later ones use Runtime.initialize().
>> 
>> When updating the DLL init code, I also changed the "DLLs with a C Interface" part in the docs, but as I'm not convinced that the approach used by "D code calling D code in DLLs" is good enough, I left it as is. My suggestion here would be to use a common phobos.DLL to share all runtime library functions including the GC (as shown here: http://d.puremagic.com/issues/show_bug.cgi?id=4071 )
> 
> The "DLLs with a C interface" part imports core.dll_helper, which no
> longer exists. What should it be now?
> _______________________________________________
> D-runtime mailing list
> D-runtime at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/d-runtime

On 20 April 2011 16:22, Sean Kelly <sean at invisibleduck.org> wrote:
> The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.

They're ugly and look like internal functions. I think just give them nice names.

In fact, how about a wrapper function for the whole mess?

extern (Windows)
BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved)
{
     return standardDllMain(hInstance, ulReason);
}

Any other required work can be done after the standard handler.
I make a lot of C-callable DLLs (in fact, it's ~75% of my D
programming) and they're all like this.
We might need a different wrapper for D-callable DLLs.

Obviously the low-level stuff needs to be available, but if there's a simple wrapper, they can stay ugly.

On Apr 20, 2011, at 7:49 AM, Don Clugston wrote:

> On 20 April 2011 16:22, Sean Kelly <sean at invisibleduck.org> wrote:
>> The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.
> 
> They're ugly and look like internal functions. I think just give them nice names.
> 
> In fact, how about a wrapper function for the whole mess?
> 
> extern (Windows)
> BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved)
> {
>     return standardDllMain(hInstance, ulReason);
> }
> 
> Any other required work can be done after the standard handler.
> I make a lot of C-callable DLLs (in fact, it's ~75% of my D
> programming) and they're all like this.
> We might need a different wrapper for D-callable DLLs.

I hope that the same wrapper can be used for both.  It would stink if the DLL creator had to build with the intent of being used by D or non-D apps.  Since Runtime.loadLibrary() knows whether it's loading a D or a non-D DLL, I hope that it will be able to take care of any rejiggering that needs to take place for everything to work properly.

So how about we rename windows._dll to windows.dll, and windows._thread to something else (dllthread? threadsupp?), then add the standardDllMain to windows.dll.

On 20 April 2011 19:55, Sean Kelly <sean at invisibleduck.org> wrote:
> On Apr 20, 2011, at 7:49 AM, Don Clugston wrote:
>
>> On 20 April 2011 16:22, Sean Kelly <sean at invisibleduck.org> wrote:
>>> The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.
>>
>> They're ugly and look like internal functions. I think just give them nice names.
>>
>> In fact, how about a wrapper function for the whole mess?
>>
>> extern (Windows)
>> BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved)
>> {
>> ? ? return standardDllMain(hInstance, ulReason);
>> }
>>
>> Any other required work can be done after the standard handler.
>> I make a lot of C-callable DLLs (in fact, it's ~75% of my D
>> programming) and they're all like this.
>> We might need a different wrapper for D-callable DLLs.
>
> I hope that the same wrapper can be used for both. ?It would stink if the DLL creator had to build with the intent of being used by D or non-D apps.

It is a decision you have to make right at the beginning.  If you have
a D app, you can (in theory) pass gc-allocated memory through extern
functions. If you have a non-D app, all memory management has to use
memory allocated by Windows, and all extern functions have to be
extern(C) or extern(Windows).
So in the D case, the GC for the DLL and the GC for the app need to be
connected, and it's far more complicated.

>?Since Runtime.loadLibrary() knows whether it's loading a D or a non-D DLL, I hope that it will be able to take care of any rejiggering that needs to take place for everything to work properly.

>
> So how about we rename windows._dll to windows.dll, and windows._thread to something else (dllthread? threadsupp?), then add the standardDllMain to windows.dll.

I'd like to hear Rainer's opinion on this --  I'd go along with anything he says.

Don Clugston wrote:
> On 20 April 2011 19:55, Sean Kelly <sean at invisibleduck.org> wrote:
>> On Apr 20, 2011, at 7:49 AM, Don Clugston wrote:
>>
>>> On 20 April 2011 16:22, Sean Kelly <sean at invisibleduck.org> wrote:
>>>> The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.
>>> They're ugly and look like internal functions. I think just give them nice names.
>>>
>>> In fact, how about a wrapper function for the whole mess?
>>>
>>> extern (Windows)
>>> BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved)
>>> {
>>>     return standardDllMain(hInstance, ulReason);
>>> }
>>>
>>> Any other required work can be done after the standard handler.
>>> I make a lot of C-callable DLLs (in fact, it's ~75% of my D
>>> programming) and they're all like this.
>>> We might need a different wrapper for D-callable DLLs.
>> I hope that the same wrapper can be used for both.  It would stink if the DLL creator had to build with the intent of being used by D or non-D apps.
> 
> It is a decision you have to make right at the beginning.  If you have
> a D app, you can (in theory) pass gc-allocated memory through extern
> functions. If you have a non-D app, all memory management has to use
> memory allocated by Windows, and all extern functions have to be
> extern(C) or extern(Windows).
> So in the D case, the GC for the DLL and the GC for the app need to be
> connected, and it's far more complicated.


I agree. A DLL "connected" to a D app must not do any initialization in
DllMain(PROCESS_ATTACH), because it has not received the gc-proxy yet.
(There is some code in the proxy to move the ranges to scan to the
application GC, but any object created during initialization lives on
the wrong heap and will not be scanned or collected.)
On the other hand a "standalone" DLL must be initialized in DllMain,
unless you want to add a lazy initialization check to every export that
might get called, waiting for setProxy() being called first or not.

I guess, I repeat myself, but I don't think the GC-proxy mechanism is good enough. What if the DLL creates a thread? The app does not know about the new thread (won't suspend it in a GC collection), the GC does not scan the new TLS data segment of the DLL (it even does not get notified of the apps own new TLS segment). I think, there is more trouble lurking (e.g. socket initialization).

The best approach I see is to share the complete runtime in a DLL, so any executable or DLL can use shared or unshared memory, threads, etc. as defined by the interfaces between the binaries.


> 
>> So how about we rename windows._dll to windows.dll, and windows._thread to something else (dllthread? threadsupp?), then add the standardDllMain to windows.dll.
> 
> I'd like to hear Rainer's opinion on this --  I'd go along with anything he says.


Adding standardDllMain() sounds good to me. I agree with windows.dll, too.

I'm not so sure about windows._thread. OSX has an osx.mach.thread_act, so we might want to have a similar naming for windows, but I have no idea what "act" stands for here.

Looking through the code, it mostly deals with the thread environment block, so windows.teb might be ok, but not really obvious.

dllthread: the code has nothing to do with DLLs explicitely, so it would
not fit.
threadsupp: maybe better: threadaux or threadhelp. The latter would at
least match the naming of the dbghelp module.

Though not perfect, threadaux or threadhelp are my favorites.

On Apr 20, 2011, at 12:45 PM, Don Clugston wrote:

> On 20 April 2011 19:55, Sean Kelly <sean at invisibleduck.org> wrote:
>> On Apr 20, 2011, at 7:49 AM, Don Clugston wrote:
>> 
>>> On 20 April 2011 16:22, Sean Kelly <sean at invisibleduck.org> wrote:
>>>> The helper modules were moved to core.sys.windows. I prepended an underscore because I thought they'd be for internal use. Alternate suggestions welcome.
>>> 
>>> They're ugly and look like internal functions. I think just give them nice names.
>>> 
>>> In fact, how about a wrapper function for the whole mess?
>>> 
>>> extern (Windows)
>>> BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved)
>>> {
>>>     return standardDllMain(hInstance, ulReason);
>>> }
>>> 
>>> Any other required work can be done after the standard handler.
>>> I make a lot of C-callable DLLs (in fact, it's ~75% of my D
>>> programming) and they're all like this.
>>> We might need a different wrapper for D-callable DLLs.
>> 
>> I hope that the same wrapper can be used for both.  It would stink if the DLL creator had to build with the intent of being used by D or non-D apps.
> 
> It is a decision you have to make right at the beginning.  If you have
> a D app, you can (in theory) pass gc-allocated memory through extern
> functions. If you have a non-D app, all memory management has to use
> memory allocated by Windows, and all extern functions have to be
> extern(C) or extern(Windows).
> So in the D case, the GC for the DLL and the GC for the app need to be
> connected, and it's far more complicated.

In the non-D case, any threads calling the DLL code can be registered via thread_attachThis().  That should cause them to be scanned by the DLLs GC for roots.  Though I imagine the app's main thread is registered regardless, inside rt_init().  That is only the stack though, not malloc-obtained DMA, so I guess it is something the DLL writer has to think about.

On Apr 20, 2011, at 2:06 PM, Rainer Schuetze wrote:
> 
> I agree. A DLL "connected" to a D app must not do any initialization in DllMain(PROCESS_ATTACH), because it has not received the gc-proxy yet. (There is some code in the proxy to move the ranges to scan to the application GC, but any object created during initialization lives on the wrong heap and will not be scanned or collected.)
> On the other hand a "standalone" DLL must be initialized in DllMain, unless you want to add a lazy initialization check to every export that might get called, waiting for setProxy() being called first or not.

Oh right... I knew I'd thought about this before and found a problem with my last suggestion.  It's the module ctors.  They are run by rt_init(), and if they allocate any memory, that memory won't be scanned by the app's GC for roots.  It's a horrible chicken & egg problem.

> I guess, I repeat myself, but I don't think the GC-proxy mechanism is good enough. What if the DLL creates a thread? The app does not know about the new thread (won't suspend it in a GC collection), the GC does not scan the new TLS data segment of the DLL (it even does not get notified of the apps own new TLS segment). I think, there is more trouble lurking (e.g. socket initialization).

I think there's a comment in the runtime somewhere that says "do something about DLL-created threads someday."  It's solvable via a proxy mechanism similar to the GC, but that approach is so horrible that I've been holding off on doing it.  Is there a problem with sockets as well?

> The best approach I see is to share the complete runtime in a DLL, so any executable or DLL can use shared or unshared memory, threads, etc. as defined by the interfaces between the binaries.

That would be great.  Then I could drop the gc_proxy stuff as well.

>>> So how about we rename windows._dll to windows.dll, and windows._thread to something else (dllthread? threadsupp?), then add the standardDllMain to windows.dll.
>> I'd like to hear Rainer's opinion on this --  I'd go along with anything he says.
> 
> Adding standardDllMain() sounds good to me. I agree with windows.dll, too.
> 
> I'm not so sure about windows._thread. OSX has an osx.mach.thread_act, so we might want to have a similar naming for windows, but I have no idea what "act" stands for here.
> 
> Looking through the code, it mostly deals with the thread environment block, so windows.teb might be ok, but not really obvious.
> 
> dllthread: the code has nothing to do with DLLs explicitely, so it would not fit.
> threadsupp: maybe better: threadaux or threadhelp. The latter would at least match the naming of the dbghelp module.
> 
> Though not perfect, threadaux or threadhelp are my favorites.

Alrighty.

Sean Kelly wrote:
> Is there a problem with sockets as well?
> 

I remember problems with WSACleanup being called from the static destructor in std.socket. This killed all connections when a thread terminated. But this was a bug and was fixed by using the shared static destructor.

Rereading the MSDN documentation for WSAStartup and WSACleanup, it seems save to have a pair of these calls in each DLL. But this statement

"The WSAStartup function typically leads to protocol-specific helper DLLs being loaded. As a result, the WSAStartup function should not be called from the DllMain function in a application DLL. This can potentially cause deadlocks."

suggests it would be better to have a lazy init call to WSAStartup when first using sockets.