On 2015-05-21 17:51, bitwise wrote:

> I'm not sure exactly what you mean about "integrated runtime".
>
> Looking in /usr/share/dmd/lib, I see phobos as a static library (which
> I'm assuming includes druntime, which I don't see anywhere). If the
> runtime is linked as a static library, shouldn't it be able to link to a
> dynamic library and just work?
>
> I think a separate runtime is what I was expecting.

No, see my answer to Timothee [1].

[1] http://forum.dlang.org/post/mjlc10$2pi5$1@digitalmars.com

-- 
/Jacob Carlborg

On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg <doob@me.com> wrote:

> On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote:
>> dlopen() of a D shared library works when called from a C++ file, and
>> that a C++ file can call multiple D shared libraries simultaneously when
>> using RTLD_LAZY option.
>>
>> Before waiting for a full solution with an integrated druntime, is there
>> at least a way to have a separate runtime in each shared library, so
>> that dlopen() can be called from a D file?
>
> No, not out of the box.
>
> I'm not sure if I remember all the details correctly, but this is basically how it works:
>
> The runtime uses the "_dyld_register_func_for_add_image" function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore.
>
> Then, the other problem with "_dyld_register_func_for_add_image" is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore.
>
> There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded.

So does that mean that "_dyld_register_func_for_add_image" shouldn't be called at all from shared libs at all then? The fact that there is no "unregister" function seems to suggest that it's meant to be called once from the main executable to add a callbacck that survives the entire life of the program. Can't whatever the callback does be done from the runtime linked to the main executable?

  Bit

On Thu, 21 May 2015 21:13:56 -0400, bitwise <bitwise.pvt@gmail.com> wrote:

> Can't whatever the callback does be done from the runtime linked to the main executable?

Or simply use dlsym from the main executable to call back into the dylib to set things up?

  Bit

On Thursday, 21 May 2015 at 15:51:38 UTC, bitwise wrote:
> I'm not sure exactly what you mean about "integrated runtime".

I think he means Phobos/Druntime as shared library.

> Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work?
>
> I think a separate runtime is what I was expecting.

Yes separate shared libraries (with multiple runtimes) work on every other platform.
The problem for OSX is that onAddImage gets called for the executable and every shared library. It would be trivial to only register the image containing the current runtime, by comparing the address of a private symbol with the memory region of the images.

https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186

On Thursday, 21 May 2015 at 01:31:37 UTC, bitwise wrote:
> I've been reading over all the bugs and conversations I can find, but I'm still confused about exactly what's going on here.
>
> What I need:
> -load a dynamic library on OSX using dlopen or Runtime.loadLibrary
> -use dlsym to retrieve my functions and set up a gc proxy

Forget about the GC proxy, it's a hack to work around ODR issues.
What you really need is a shared phobos library as we have on linux or freebsd.

> -pull interfaces of classes defined in the dynamic library into my program for use.

If you want to exchange data and code across multiple shared libraries and your executable the runtime must be fully aware of all shared libraries to support casting ,exceptions and finalizers.

On Thursday, 21 May 2015 at 08:07:49 UTC, Jacob Carlborg wrote:
> I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1].

One could also make emulated TLS work with shared libraries.

On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote:
> Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).

Have you tried LDC? They recently got shared library support, maybe also on OSX.

On 2015-05-22 18:12, Martin Nowak wrote:

> One could also make emulated TLS work with shared libraries.

Yeah, but then you would need to implement something similar to the TLS code already present in the dynamic linker in druntime. Both alternatives would require changes to both the compiler and runtime. Hopefully implementing native TLS in the compiler would not be any more difficult than adding support for shared libraries to the emulated TLS.

-- 
/Jacob Carlborg

On 2015-05-22 03:13, bitwise wrote:

> So does that mean that "_dyld_register_func_for_add_image" shouldn't be
> called at all from shared libs at all then? The fact that there is no
> "unregister" function seems to suggest that it's meant to be called once
> from the main executable to add a callbacck that survives the entire
> life of the program.

Yes.

> Can't whatever the callback does be done from the
> runtime linked to the main executable?

The runtime extracts some data from the loaded image and stores it in some global variable. If the executable would do this it needs to store the data in the global variable in the dynamic library.

-- 
/Jacob Carlborg

I've read through these now, which I missed the first time around, so sorry for making you guys repeat yourselves ;)

<Runtime issue on Mac OS X>
http://comments.gmane.org/gmane.comp.lang.d.runtime/1214

<ideas for runtime loading of shared libraries.>
http://forum.dlang.org/thread/mailman.2052.1325532031.24802.digitalmars-d@puremagic.com

So in terms of a shared lib having it's own runtime, we have these problems:

[1] problem

On Fri, 22 May 2015 12:04:24 -0400, Martin Nowak <code@dawg.eu> wrote:
> Yes separate shared libraries (with multiple runtimes) work on every other platform.
> The problem for OSX is that onAddImage gets called for the executable and every shared library. It would be trivial to only register the image containing the current runtime, by comparing the address of a private symbol with the memory region of the images.
>
> https://github.com/D-Programming-Language/druntime/blob/6331ab1ae19f3ff82449a5734b59d81b128685f4/src/rt/sections_osx.d#L186

[2] problem

On Thu, 21 May 2015 15:34:56 -0400, Jacob Carlborg <doob@me.com> wrote:
> The runtime uses the "_dyld_register_func_for_add_image" function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore.
>
> Then, the other problem with "_dyld_register_func_for_add_image" is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore.


I think I have found solutions for these problems.


[1] solution

I've modified sections_osx.d as follows:

extern (C) void _sections_osx_onAddImage_STUB(in mach_header* h, intptr_t slide) {
    // empty
}

extern (C) void sections_osx_onAddImage(in mach_header* h, intptr_t slide)
{
// on mac osx, Dl_info.dli_fbase is a pointer to the mach_header for the library. [I]
// here we return unless onAddImage is being called for the current library
    mach_header* myHeader = null;

    Dl_info info;
    // this line also makes sure that the stub isn't
    // removed by the linker(it's needed for [2])
    if(dladdr(&_sections_osx_onAddImage_STUB, &info))
    {
        mach_header* mh = cast(mach_header*)info.dli_fbase;
        if(mh == cast(mach_header*)h)
            myHeader = mh;
    }

    if(!myHeader)
        return;

    // initialize sections.....

}

[2] solution

Although the callback passed to "_dyld_register_func_for_add_image" cannot be removed, it can be replaced, so I've replaced it with a pointer to the stub located in the main program.

I've modified initSections() in sections_osx.d as follows:

void initSections()
{
    pthread_key_create(&_tlsKey, null);

    // register the callback as usual. This will call the callback
    // for every library currently loaded before returning, so
    // once it has returned, it should be safe to set the callback
    // to something else(the empty stub)
    _dyld_register_func_for_add_image(&sections_osx_onAddImage);

    // dlopen(null, ..) will retrieve a handle to the main program [II]
    // OSX docs says it returns the first symbol found using
    // "RTLD_DEFAULT" or "the default library search order" which
    // should(and does as far as I can tell) return the handle
    // to the main program
    void *main = dlopen(null, RTLD_NOW);
    assert(main);
    alias typeof(&_sections_osx_onAddImage_STUB) addImgFn;
    addImgFn func = cast(addImgFn)dlsym(main, "_sections_osx_onAddImage_STUB");
    assert(func);

    // set the callback to the empty stub in the main program
    _dyld_register_func_for_add_image(func);

    _isRuntimeInitialized = true;
}

[I]  https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dladdr.3.html
[II] https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html
     http://linux.die.net/man/3/dlopen



So at this point, it seems like these two fixes work as expected, but now, I'm having some new and very strange problems.

I have a simple shared library and program I've been using to test this:

[main.d]
module main;
import std.stdio;
import std.conv;
import std.string;
import core.sys.posix.dlfcn;

void main(string[] args)
{
    alias void function() fnType;

    void *handle = dlopen("myShared.dylib", RTLD_NOW);
    assert(handle);

    fnType init = cast(fnType)dlsym(handle, "initLib");
    assert(init);
    init();

    fnType term = cast(fnType)dlsym(handle, "termLib");
    assert(term);
    term();

    dlclose(handle);
    writeln("done");
}

[myShared.d]
module myShared;
import core.runtime;
import std.stdio;

extern(C) void initLib() {
    writeln("Initializing Runtime");
    Runtime.initialize();
}

extern(C) void termLib() {
    writeln("Terminating Runtime");
    Runtime.terminate();
}


So, when I run the above program, rt_init() should be called once for the main program, and once for the shared library. However, when I run the above program, rt_init() from the main program seems to get called twice. To clarify, I mean that when I retrieve "initLib()" with dlsym() and call it, rt_init() from the main module gets called.

This seems to prove the above:

In dmain2.d, I have modified rt_init() as follows:

extern (C) int rt_init()
{
    import core.sys.posix.dlfcn;
    Dl_info info;
    if(dladdr(&rt_init, &info))
        fprintf(stdout, "RT INIT: %s\n", info.dli_fname);   // this prints "main" for both calls

    if (atomicOp!"+="(_initCount, 1) > 1)
    {
        fprintf(stdout, "RT ALREADY INITIALIZED\n");
        return 1;
    }

// ...

    fprintf(stdout, "RT INIT COMPLETE\n");
}

When the main program calls rt_init(), the output correctly reads "RT INIT COMPLETE".
When I load the dynamic library however, I get the output "RT ALREADY INITIALIZED"

How is this possible? I am not using a shared druntime afaik..

  Bit