I tried using a shared library for OSX yesterday. I opened it with dlopen, retrieved my extern(C) function, and called it. All was well, and it seemed to work(wrote to the console with writeln).

But, I got a message in the console saying shared libraries were not yet implemented for OSX. What exactly is it that's not working?

 Thanks,
   Bit

On 2015-05-20 16:44, bitwise wrote:
> I tried using a shared library for OSX yesterday. I opened it with
> dlopen, retrieved my extern(C) function, and called it. All was well,
> and it seemed to work(wrote to the console with writeln).
>
> But, I got a message in the console saying shared libraries were not yet
> implemented for OSX. What exactly is it that's not working?

TLS, module constructors, module/class info, perhaps the GC, similar things.

-- 
/Jacob Carlborg

On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
> On 2015-05-20 16:44, bitwise wrote:
>> I tried using a shared library for OSX yesterday. I opened it with
>> dlopen, retrieved my extern(C) function, and called it. All was well,
>> and it seemed to work(wrote to the console with writeln).
>>
>> But, I got a message in the console saying shared libraries were not yet
>> implemented for OSX. What exactly is it that's not working?
>
> TLS, module constructors, module/class info, perhaps the GC, similar things.

Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).

On Wed, 20 May 2015 17:35:37 -0400, bitwise <bitwise.pvt@gmail.com> wrote:

> On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
>> On 2015-05-20 16:44, bitwise wrote:
>>> I tried using a shared library for OSX yesterday. I opened it with
>>> dlopen, retrieved my extern(C) function, and called it. All was well,
>>> and it seemed to work(wrote to the console with writeln).
>>>
>>> But, I got a message in the console saying shared libraries were not yet
>>> implemented for OSX. What exactly is it that's not working?
>>
>> TLS, module constructors, module/class info, perhaps the GC, similar things.
>
> Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).


I've been reading over all the bugs and conversations I can find, but I'm still confused about exactly what's going on here.

What I need:
-load a dynamic library on OSX using dlopen or Runtime.loadLibrary
-use dlsym to retrieve my functions and set up a gc proxy
-pull interfaces of classes defined in the dynamic library into my program for use.

Not sure how much of this is working properly or not.

I'd be willing to throw a couple of bucks at this for a bounty, but I'm not sure how much it's worth though.
Any takers for $25? ;)

  Bit

On Wednesday, 20 May 2015 at 21:35:38 UTC, bitwise wrote:
> On Wednesday, 20 May 2015 at 18:53:30 UTC, Jacob Carlborg wrote:
>> On 2015-05-20 16:44, bitwise wrote:
>>> I tried using a shared library for OSX yesterday. I opened it with
>>> dlopen, retrieved my extern(C) function, and called it. All was well,
>>> and it seemed to work(wrote to the console with writeln).
>>>
>>> But, I got a message in the console saying shared libraries were not yet
>>> implemented for OSX. What exactly is it that's not working?
>>
>> TLS, module constructors, module/class info, perhaps the GC, similar things.
>
> Heh.. That's pretty useless. Any idea on the state of things? Like if there are plans to support this in the near future(few months)? I couldn't find many conversations on this. This kinda-totally ruins my plans(on OSX at least).

Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS using undocumented OS X APIs.  I'm unsure of how full-fledged the support is on those other issues, but I believe the big issue is that loading multiple shared libraries is not supported, as that will take more work:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198

If you want to talk to someone about getting more of shared libraries working, I believe Martin is your man.

On 2015-05-21 07:06, Joakim wrote:

> Actually, TLS works on OS X: dmd emulates it, while ldc has native TLS
> using undocumented OS X APIs.

The emulated TLS won't work with dynamic libraries. The runtime can only fetch the TLS data from one source, which will be the executable itself.

> I'm unsure of how full-fledged the
> support is on those other issues, but I believe the big issue is that
> loading multiple shared libraries is not supported, as that will take
> more work:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_osx.d#L198

No, that issue is loading dynamic libraries regardless if it's one or several. As you can see in the comment, it will pick the sections from the executable.

> If you want to talk to someone about getting more of shared libraries
> working, I believe Martin is your man.

Yeah, but I'm not sure how familiar he is with the OS X specific things. And native TLS is a requirement, which requires changes in the compiler and the runtime. Alternatively we need to reimplement the TLS handling from the dynamic linker.

-- 
/Jacob Carlborg

On 2015-05-20 23:35, bitwise wrote:

> Heh.. That's pretty useless. Any idea on the state of things? Like if
> there are plans to support this in the near future(few months)? I
> couldn't find many conversations on this. This kinda-totally ruins my
> plans(on OSX at least).

I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1].

[1] https://issues.dlang.org/show_bug.cgi?id=9476

-- 
/Jacob Carlborg

dlopen() of a D shared library works when called from a C++ file, and that a C++ file can call multiple D shared libraries simultaneously when using RTLD_LAZY option.

Before waiting for a full solution with an integrated druntime, is there at least a way to have a separate runtime in each shared library, so that dlopen() can be called from a D file?

On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 2015-05-20 23:35, bitwise wrote:
>
>  Heh.. That's pretty useless. Any idea on the state of things? Like if
>> there are plans to support this in the near future(few months)? I
>> couldn't find many conversations on this. This kinda-totally ruins my
>> plans(on OSX at least).
>>
>
> I don't think anyone is working on this. Native TLS is a prerequisite which requires changes both to the compiler and the runtime. There's an enhancement request for native TLS [1].
>
> [1] https://issues.dlang.org/show_bug.cgi?id=9476
>
> --
> /Jacob Carlborg
>

On Thu, 21 May 2015 04:12:19 -0400, Timothee Cour via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> dlopen() of a D shared library works when called from a C++ file, and that
> a C++ file can call multiple D shared libraries simultaneously when using
> RTLD_LAZY option.
>
> Before waiting for a full solution with an integrated druntime, is there at
> least a way to have a separate runtime in each shared library, so that
> dlopen() can be called from a D file?

I'm not sure exactly what you mean about "integrated runtime".

Looking in /usr/share/dmd/lib, I see phobos as a static library (which I'm assuming includes druntime, which I don't see anywhere). If the runtime is linked as a static library, shouldn't it be able to link to a dynamic library and just work?

I think a separate runtime is what I was expecting.

> On Thu, May 21, 2015 at 1:07 AM, Jacob Carlborg via Digitalmars-d <
> digitalmars-d@puremagic.com> wrote:
>>
>> I don't think anyone is working on this. Native TLS is a prerequisite
>> which requires changes both to the compiler and the runtime. There's an
>> enhancement request for native TLS [1].
>>
>> [1] https://issues.dlang.org/show_bug.cgi?id=9476
>>
>> --
>> /Jacob Carlborg
>>


(In reply to John Colvin from comment #11)
> I think the time has come that 10.6 support could be dropped, no? It's no
> longer supported by Apple.

+1

  Bit

On 2015-05-21 10:12, Timothee Cour via Digitalmars-d wrote:
> dlopen() of a D shared library works when called from a C++ file, and
> that a C++ file can call multiple D shared libraries simultaneously when
> using RTLD_LAZY option.
>
> Before waiting for a full solution with an integrated druntime, is there
> at least a way to have a separate runtime in each shared library, so
> that dlopen() can be called from a D file?

No, not out of the box.

I'm not sure if I remember all the details correctly, but this is basically how it works:

The runtime uses the "_dyld_register_func_for_add_image" function provided by the dynamic linker. This function is used to register a callback. The callback will be called for each currently loaded image (executable/dynamic library) in the executable. The callback will also be called for every newly loaded image, i.e. using dlopen. You need to keep track of which image is yourself and which are other images you should ignore.

Then, the other problem with "_dyld_register_func_for_add_image" is that it's not possible to unregister the callback. Which means, if you register a callback using this function from a dynamic library loaded with dlopen and then unload it. Next time dlopen is called it will crash, because the callback points to an address that doesn't exist anymore.

There's an other, undocumented, function that does basically the same but prevents the issue with the callback by not allowing the dynamic library to be unloaded.

-- 
/Jacob Carlborg