September 30, 2012
On 2012-09-30 08:41, Rob T wrote:

> There are plenty of cases where you have to use a dynamically loaded lib
> even if you know before hand what will be loaded. I think you understand
> this if I read your posts correctly.
>
> In my case I have a pre-existing C++ app that is designed to load user
> defined C++ plugins. I wanted to interface D to one of my own C++
> plugins, but I apparently cannot do it reliably because of
> initialization issues with the GC and possibly some other obscure items.

But now we're back at plugins. I think this part of the discussion is starting to run in circles and become quite pointless. I think most of us know that we need to support all types of libraries. Static and dynamic, both for link time and runtime.

> If I can figure out what needs to be done to resolve the problem in
> enough detail, then maybe I can hack the runtime source and roll out a
> solution.

You can have a look at the work done by Martin Nowak:

https://github.com/dawgfoto/druntime/commits/SharedRuntime

He has a couple of other useful branches as well, for example:

https://github.com/dawgfoto/druntime/commits/dmain2Refactoring

On Mac OS X there's also the problem with TLS. There is no native support for TLS in Mac OS X prior to Lion (10.7). DMD has rolled its own implementation that needs to be adapted to support dynamic libraries.

> The GC always seems to pop up as a source of problems. For long term
> solution, the GC should be made 100% optional (in practice as opposed to
> in theory), the standard library ought to be made to work with or
> wothout a GC (or simply without), and the GC itself should be easily
> replaceable with alternate versions. I think this idea has already been
> discussed elsewhere, and is on the TODO list (I hope!).

You can replace the GC at link time. Here's an example of a stubbed implemented of the GC:

http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

This is for Tango but the druntime is based on the Tango runtime so I would guess most of this would be the same.

-- 
/Jacob Carlborg
September 30, 2012
On Sunday, 30 September 2012 at 10:57:24 UTC, Jacob Carlborg wrote:
> On 2012-09-30 08:41, Rob T wrote:
>
> I think most of us know that we need to support all types of libraries. Static and dynamic, both for link time and runtime.

OK we're all in agreement on this point.

> You can have a look at the work done by Martin Nowak:
>
> https://github.com/dawgfoto/druntime/commits/SharedRuntime

It seems that an attempt to make the runtime shared is well under way. Did anything get into the main dmd branch or has the effort been stalled or ...?

I've only recently been learning what's going on with D and why it is what it is, so a breif history lesson may be needed once in a while.

> You can replace the GC at link time. Here's an example of a stubbed implemented of the GC:
> http://dsource.org/projects/tango/browser/trunk/tango/core/rt/gc/stub

I will look at this too. Thanks for the pointers.

--rt

October 01, 2012
On 2012-10-01 01:42, Rob T wrote:

> It seems that an attempt to make the runtime shared is well under way.
> Did anything get into the main dmd branch or has the effort been stalled
> or ...?

Seems pretty stalled.

> I will look at this too. Thanks for the pointers.

No problem.

-- 
/Jacob Carlborg
October 01, 2012
Am Sat, 29 Sep 2012 15:19:30 +0200
schrieb Jacob Carlborg <doob@me.com>:

> On 2012-09-28 20:25, Maxim Fomin wrote:
> 
> > I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are working and here is some kind of test: https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't executed when it is situated in a shared library and which calls some throwing function from another shared library. Unittests aren't working either. Regarding other parts - they seem to work.
> 
> That's a fairly uninteresting test. You are linking to the dynamic library. What's interesting is loading a dynamic library using dlopen, or similar. What's the point of using dynamic libraries if you're linking with them?
> 

There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-)

So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test

It should also be enhanced to test multiple shared libraries.

The good news:
* Exception handling is working
* ModuleInfos are working
* unit tests are working
* Static variables, gshared variables, tls variables are working
* Object.factory is working
* Calling functions, passing function pointers, passing classes between
  dso/app is working

The bad news:
* The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
  it only scans the main app.
October 01, 2012
On 2012-10-01 12:42, Johannes Pfau wrote:

> There are some reasons for dynamic libraries linked at compile time,
> one is that we have to start somewhere and they are required for
> plugins / dynamically loaded libraries as well ;-)
>
> So I started a small test suite for GDC (could be adapted to other
> compilers). It currently only tests compile time linking of dynamic
> libraries, but adjusting the test to use runtime loading should be
> easy. But it's pointless as long as we have no runtime support.
> https://github.com/jpf91/dso-test
>
> It should also be enhanced to test multiple shared libraries.
>
> The good news:
> * Exception handling is working
> * ModuleInfos are working
> * unit tests are working
> * Static variables, gshared variables, tls variables are working
> * Object.factory is working
> * Calling functions, passing function pointers, passing classes between
>    dso/app is working

That's good. Have you tested this with DMD?

> The bad news:
> * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
>    it only scans the main app.

That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux.

BTW, is the runtime and phobos statically linked both with the dynamic library and the executable?

-- 
/Jacob Carlborg
October 01, 2012
On 1 October 2012 13:34, Jacob Carlborg <doob@me.com> wrote:
> On 2012-10-01 12:42, Johannes Pfau wrote:
>
>> There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-)
>>
>> So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test
>>
>> It should also be enhanced to test multiple shared libraries.
>>
>> The good news:
>> * Exception handling is working
>> * ModuleInfos are working
>> * unit tests are working
>> * Static variables, gshared variables, tls variables are working
>> * Object.factory is working
>> * Calling functions, passing function pointers, passing classes between
>>    dso/app is working
>
>
> That's good. Have you tested this with DMD?
>
>
>> The bad news:
>> * The GC doesn't scan TLS/__gshared/static data in dynamic libraries,
>>    it only scans the main app.
>
>
> That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux.
>
> BTW, is the runtime and phobos statically linked both with the dynamic library and the executable?
>
> --
> /Jacob Carlborg

On Linux, there has already been an runtime implementation written that scans /proc/self/maps and adds all data sections to the GC that way.  Whether or not DMD wishes to go down that route is their own decision.  I am looking into a solution that doesn't have any bearing on what platform it's running on...


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
October 01, 2012
On 2012-10-01 14:40, Iain Buclaw wrote:

> On Linux, there has already been an runtime implementation written
> that scans /proc/self/maps and adds all data sections to the GC that
> way.  Whether or not DMD wishes to go down that route is their own
> decision.  I am looking into a solution that doesn't have any bearing
> on what platform it's running on...

Well, /proc isn't available on Mac OS X so I think you have to continue looking.

-- 
/Jacob Carlborg
October 01, 2012
On 2012-10-01 14:40, Iain Buclaw wrote:

> On Linux, there has already been an runtime implementation written
> that scans /proc/self/maps and adds all data sections to the GC that
> way.  Whether or not DMD wishes to go down that route is their own
> decision.  I am looking into a solution that doesn't have any bearing
> on what platform it's running on...

I think this is already working on Mac OS X. It's handled by:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L82

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memory_osx.d#L133

-- 
/Jacob Carlborg
October 01, 2012
On 2012-10-01 12:42, Johannes Pfau wrote:

> There are some reasons for dynamic libraries linked at compile time,
> one is that we have to start somewhere and they are required for
> plugins / dynamically loaded libraries as well ;-)
>
> So I started a small test suite for GDC (could be adapted to other
> compilers). It currently only tests compile time linking of dynamic
> libraries, but adjusting the test to use runtime loading should be
> easy. But it's pointless as long as we have no runtime support.
> https://github.com/jpf91/dso-test

Isn't "dmain2" used when building shared libraries using GDC? That's where the implementation of "rt_init" is located.

-- 
/Jacob Carlborg
October 01, 2012
Am Mon, 01 Oct 2012 14:34:21 +0200
schrieb Jacob Carlborg <doob@me.com>:

> On 2012-10-01 12:42, Johannes Pfau wrote:
> 
> > There are some reasons for dynamic libraries linked at compile time, one is that we have to start somewhere and they are required for plugins / dynamically loaded libraries as well ;-)
> >
> > So I started a small test suite for GDC (could be adapted to other compilers). It currently only tests compile time linking of dynamic libraries, but adjusting the test to use runtime loading should be easy. But it's pointless as long as we have no runtime support. https://github.com/jpf91/dso-test
> >
> > It should also be enhanced to test multiple shared libraries.
> >
> > The good news:
> > * Exception handling is working
> > * ModuleInfos are working
> > * unit tests are working
> > * Static variables, gshared variables, tls variables are working
> > * Object.factory is working
> > * Calling functions, passing function pointers, passing classes
> > between dso/app is working
> 
> That's good. Have you tested this with DMD?

Not yet.
> 
> > The bad news:
> > * The GC doesn't scan TLS/__gshared/static data in dynamic
> > libraries, it only scans the main app.
> 
> That should be fairly trivial on Mac OS X. But I'm suspecting it won't be that easy on Linux.
> 
> BTW, is the runtime and phobos statically linked both with the dynamic library and the executable?

I tested two different configurations:

druntime and phobos are shared libraries as well (this is the correct solution, it's mostly working except for the GC issues)

druntime and phobos static linking: phobos and druntime are statically linked into the app, libdso.so is not linked against druntime/phobos at all. (this is a hack to get better test results: With a shared druntime, you can't call GC.collect twice, because the first call frees important objects in druntime and then the second call segfaults)