http://d.puremagic.com/issues/show_bug.cgi?id=4071

           Summary: Missing support to share memory and objects between
                    DLLs and executable
           Product: D
           Version: unspecified
          Platform: Other
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: druntime
        AssignedTo: sean@invisibleduck.org
        ReportedBy: r.sagitario@gmx.de


--- Comment #0 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 11:35:29 PDT ---
Sharing gc-allocated or c-runtime-allocated memory, files, threads and other objects between different D-DLLs and the D-executable can be desirable, especially when working in a larger environment with dynamically loaded DLLs.

The current support for this is very limited, supplying a proxy for the GC, but its implementation is incomplete, e.g. with respect to TLS memory and new threads.

The following comments will show an implementation of a phobos.dll that can be accessed by a number of DLLs and the executable to create objects than can be shared freely. It also allows the usage of a single GC.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #1 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 11:40:36 PDT ---
Created an attachment (id=600)
add options -exportall and -sharedlib to dmd

Let's start with a few patches to dmd:

1. add option -exportall to export any mangled symbol defined by an object file
  (a workaround for #3956 is needed, patching obj_export to avoid
indeterministic removal of leading '_' from exported symbols by optlink)

2. add option -sharedlib to switch to different default libraries phobos_shared.lib and snn_shared.lib instead of phobos.lib and snn.lib

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #2 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 11:47:04 PDT ---
Created an attachment (id=601)
druntime support for building phobos.dll

The major changes are in druntime and involve splitting files into the part that can be shared in a single DLL and the functions that must exist per binary:

- move moduleCtor/Dtor, etc. from object_.di into new file moduleinit.d (had to
copy some ModuleInfo (back?) declarations to object.di to compile)
- move extern(C) main() from dmain2.d into new file cmain.d
- exclude __LDBLULLNG() in llmath.d because it is already in snn.lib
- build druntime.obj with "-exportall" instead of druntime.lib excluding
moduleinit.d, cmain.d and a few more
- extracted stuff from dll_helper.d into new file shared_dll_helper.d and added
a few more functions to support patching relocations

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #3 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 11:54:49 PDT ---
Created an attachment (id=602)
changes to phobos to build phobos.dll

The patches to phobos are more-or-less limited to the makefile and adding 2 files:

- phobos: build phobos.obj with "-exportall" instead of phobos.lib
- add def-file exporting any sensible symbol from the C compiled modules and
snn.lib
- add dllmain.d that works along the line of the new dll-support in dmd 2.042
- build phobos.dll from druntime.obj, C compiled modules and the object files
not included in druntime.obj using the def-file
- use implib to create import library phobos_shared.lib (the optlink /IMLIB
creates a corrupted lib)
- add modules to the lib that must exist in each binary: minit.obj,
moduleinit.obj, cmain.obj, dll_helper.obj, etc.
- extracted some modules from snn.lib (constart, dllstart, winstart, excptlst,
cinit, ehinit, setargv, tlsdata, tlsseg, clearbss) and put them in new lib
snn_shared.lib

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #4 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 12:00:45 PDT ---
Created an attachment (id=603)
example using the shared dll

This is the example from dll.html modified to work with the shared DLL.

Please note that an executable/DLL needs a slightly different initialization:
- patch relocations to stubs to import table to the destination value to allow
data symbols to be accessed correctly
- do not init gc
- inform gc about TLS usage
- add check to module(tls)ctors to avoid calling modules in other dlls

These are already covered by the patches to druntime.

If you apply the patches, you might need to check the path to the tools at the top of the makefiles.

current restrictions:
- there is no global moduleinfo-array spanning all binaries
- trace module not included (always produces output)
- no version check, so welcome to DLL hell!

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #5 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-07 12:15:59 PDT ---
forgot to mention that the patches are against dmd rev 431, druntime rev 282 and phobos rev 1477.

you can build the dll by executing

   make -f win32.mak dll

in the phobos directory.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071


Jacob Carlborg <doob@me.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |doob@me.com


--- Comment #6 from Jacob Carlborg <doob@me.com> 2010-04-07 13:08:02 PDT ---
I have no experiences of DLLs but here are my experiences on converting Tango to a dynamic library on Mac OS X. Don't know if any of this would work on Windows.

Comment 1
An option to build as an dynamic library would be nice

Comment 2
The way I solved the problems here was to declare the D main function as a weak
symbol in a C file. The module constructors are handled by looping through all
loaded images (binaries, dynamic libraries) and collect the module info arrays.
Then combining all the arrays into one and running all the module constructors.
The same is done with the exception handling tables.

The following have I not done yet, but I hope it will work:
In a C file, declare a function to be a module initializer that initializes the
D runtime if no D main function is present.

Don't know if this helps anything.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #7 from Rainer Schuetze <r.sagitario@gmx.de> 2010-04-08 14:29:59 PDT ---
I'm not sure the way you did it on OSX is feasable on Windows aswell, because you might not have access to all the info in the DLLs (maybe it's possible if you export all necessary symbols in each DLL). But I guess it will not work for dynamically loaded DLLs. My implementation does not need a "supervising" function to run initializers, each binary just initializes the part it contains.

I haven't done anything with respect to exception handling, but if I understand it correctly, the exception frames are local to the functions and binary they belong to, so they still are unwinded correctly. A simple test (catching an exception from inside phobos.dll) was working correctly.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #8 from Jacob Carlborg <doob@me.com> 2010-04-09 02:26:03 PDT ---
That may actually a better idea, to let every binary handle its own initialization. Then I guess there will be no problem with module constructors that are run when they shouldn't.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=4071



--- Comment #9 from Jacob Carlborg <doob@me.com> 2010-04-11 03:56:04 PDT ---
Another question: since you have moved the module initialization into a new file and let every binary take care of its own initialization, couldn't that cause problems like circular dependencies that aren't detected? For example, two modules from two different libraries, each compiled into its own dll, which import the other module.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------