December 13, 2011
On 2011-12-13 13:42, Martin Nowak wrote:
> On Mon, 12 Dec 2011 12:31:44 +0100, Jacob Carlborg <doob@me.com> wrote:
>
>> On 2011-12-10 22:32, Walter Bright wrote:
>>> On 12/10/2011 12:53 PM, maarten van damme wrote:
>>>> Could also someone shed some light on creating shared library's on
>>>> linux? there
>>>> was some news about -fPIC beeing fixed but no real confirmation.
>>>
>>> Turns out there was a bug where EBX was not set correctly when calling a
>>> function that existed in a shared library. This bug was fixed. So we're
>>> ready to try again at creating a shared library with dmd.
>>
>> That's great to hear. We'll see how far we get this time.
>>
>
> What's the state of the OSX dylib code in dmd and druntime?

Most of the needed code is there but it's not "activated". I've been trying to get it to work several times but it's always something that I can't get to work.

I currently have problems with TLS. Except from that it seems like it works, but there are several improvements that could be made. I have

I have a fork of druntime where you can see my progress, I've rewritten what's already in druntime: https://github.com/jacob-carlborg/druntime/tree/dylib

What's needed or can be improved:

* Cleanup module infos, exception handling tables and TLS when a dynamic library is unloaded

* A low level associative array and regular array that can be used without having druntime full initialized

I wrote to the DMD internals mailing list about this: http://www.mail-archive.com/dmd-internals@puremagic.com/msg02701.html

-- 
/Jacob Carlborg
December 13, 2011
On Tue, 13 Dec 2011 14:29:09 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2011-12-13 13:42, Martin Nowak wrote:
>> On Mon, 12 Dec 2011 12:31:44 +0100, Jacob Carlborg <doob@me.com> wrote:
>>
>>> On 2011-12-10 22:32, Walter Bright wrote:
>>>> On 12/10/2011 12:53 PM, maarten van damme wrote:
>>>>> Could also someone shed some light on creating shared library's on
>>>>> linux? there
>>>>> was some news about -fPIC beeing fixed but no real confirmation.
>>>>
>>>> Turns out there was a bug where EBX was not set correctly when calling a
>>>> function that existed in a shared library. This bug was fixed. So we're
>>>> ready to try again at creating a shared library with dmd.
>>>
>>> That's great to hear. We'll see how far we get this time.
>>>
>>
>> What's the state of the OSX dylib code in dmd and druntime?
>
> Most of the needed code is there but it's not "activated". I've been trying to get it to work several times but it's always something that I can't get to work.
>
> I currently have problems with TLS. Except from that it seems like it works, but there are several improvements that could be made. I have
>
> I have a fork of druntime where you can see my progress, I've rewritten what's already in druntime: https://github.com/jacob-carlborg/druntime/tree/dylib
>
> What's needed or can be improved:
>
> * Cleanup module infos, exception handling tables and TLS when a dynamic library is unloaded
>
TLS is not too difficult. We can either use bracketed sections again or
let the compiler emit small thunks that fetch the complete TLS section for
a executable/DSO. The important point is that fetching the TLS addresses must
be done for each thread and from a local function within each DSO.
people.redhat.com/drepper/tls.pdf
My current approach is to register callbacks, and let the threads update
their ranges before GC.
Not sure, but it will probably causes eagerly allocation of the TLS blocks.

Some similar issues happen with the other section brackets (.deh, .minfo).
We need local symbols as brackets not global ones as they would collide.

I've tried several approaches.

 o Using a linker script with a recent binutils, add needed d sections
   and use PROVIDE_HIDDEN to bracket them. Then it's only a matter
   of adding constructor/destructor routines that register with druntime.

   It's not too feasible as newer binutils are default on linux only
   and distributing linker scripts is not too nice either.

 o Create a C module similar to crt0.o that creates d sections and bracketing
   symbols. Add constructor/destructor to register with druntime.

   The object file needs to be the first when linking, but the linker
   can merge sections, thus there is only one registration per
   executable/shared library.

 o Let the compiler generate a constructor/destructor for executables
   and shared libraries. The object would also need to get linked first.

   This is less transparent than the 2nd option without much benefit.

 o Generate registration routines per object file. No bracketed sections.



So currently I'm in favor of having to install a 'drt0.o' or so somewhere
in the library path and linking it into every executable/DSO.

> * A low level associative array and regular array that can be used without having druntime full initialized
>
> I wrote to the DMD internals mailing list about this: http://www.mail-archive.com/dmd-internals@puremagic.com/msg02701.html
December 13, 2011
On Tue, 13 Dec 2011 14:12:43 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2011-12-13 13:12, Martin Nowak wrote:
>> On Sat, 10 Dec 2011 22:32:17 +0100, Walter Bright
>> <newshound2@digitalmars.com> wrote:
>>
>>> On 12/10/2011 12:53 PM, maarten van damme wrote:
>>>> Could also someone shed some light on creating shared library's on
>>>> linux? there
>>>> was some news about -fPIC beeing fixed but no real confirmation.
>>>
>>> Turns out there was a bug where EBX was not set correctly when calling
>>> a function that existed in a shared library. This bug was fixed. So
>>> we're ready to try again at creating a shared library with dmd.
>>>
>>
>> On x86-64 the PIC code is working fine, but there were some relocation
>> issues left.
>> https://github.com/dawgfoto/dmd/commits/SharedElf
>>
>> I'm looking into wiring things up with the runtime,
>> but it requires quite some changes.
>>
>> One thing I don't have a good answer too is how to handle registration
>> of a shared library loaded during runtime. Apparently we don't want
>> to synchronize all threads whenever a library gets loaded/unloaded.
>>
>> martin
>
> On Mac OS X there's a function available for that, "_dyld_register_func_for_add_image". I don't know if there's something similar available on Linux and FreeBSD.
>
> http://developer.apple.com/library/mac/#documentation/developertools/Reference/MachOReference/Reference/reference.html
>

This can be done by __attribute__ ((constructor)) or by adding a function
pointer to the .init section.
But the issue is initialization for already running threads.
December 13, 2011
On 2011-12-13 15:15, Martin Nowak wrote:
> TLS is not too difficult. We can either use bracketed sections again or
> let the compiler emit small thunks that fetch the complete TLS section for
> a executable/DSO. The important point is that fetching the TLS addresses
> must
> be done for each thread and from a local function within each DSO.
> people.redhat.com/drepper/tls.pdf
> My current approach is to register callbacks, and let the threads update
> their ranges before GC.
> Not sure, but it will probably causes eagerly allocation of the TLS blocks.

Mac OS X doesn't support TLS (it does now in 10.7) so Walter rolled his own implementation. I don't know how close that is to the EFL implementation specified in the above link.

> Some similar issues happen with the other section brackets (.deh, .minfo).
> We need local symbols as brackets not global ones as they would collide.

There is no need for brackets, they just cause problems. The correct way to get section data on Mac OS X is to use the "getsectbynamefromheader" function. You can have a look at my fork to see how its used.

https://github.com/jacob-carlborg/druntime/blob/dylib/src/rt/memory.d#L318

> I've tried several approaches.

They way I've done it is to register a callback for the "_dyld_register_func_for_add_image" function. The documentation describes the function as:

"When you call _dyld_register_func_for_add_image, the dynamic linker runtime calls the specified callback (func) once for each of the images that are currently loaded into the program. When a new image is added to the program, your callback is called again with the mach_header for the new image, and the virtual memory slide amount of the new image."

In this callback I extract module infos, exception handling tables and TLS data from the mach_header. This is now where the problem comes. I need an associative array which maps mach_headers to a struct (or similar) containing module infos, exception handling tables and TLS data. But I don't have an associative array that works this earily in the initialization phase of the runtime.

The next problem is with accessing TLS. When a TLS variable is accessed the compiler calls the ___tls_get_addr function, implemented in druntime. This function receives the address and then converts it to TLS using the TLS data bracketed in the object file. In this function I need to somehow access the current object file/mach_header, then use this mach_header to access the TLS data in the associative array mentioned above.

You can see my current implementation in my fork, everything below this line:

https://github.com/jacob-carlborg/druntime/blob/dylib/src/rt/memory.d#L267

As you can see I don't have an associative array so the above currently doesn't work.

-- 
/Jacob Carlborg
December 13, 2011
On 2011-12-13 15:19, Martin Nowak wrote:
> On Tue, 13 Dec 2011 14:12:43 +0100, Jacob Carlborg <doob@me.com> wrote:
>> On Mac OS X there's a function available for that,
>> "_dyld_register_func_for_add_image". I don't know if there's something
>> similar available on Linux and FreeBSD.
>>
>> http://developer.apple.com/library/mac/#documentation/developertools/Reference/MachOReference/Reference/reference.html
>>
>>
>
> This can be done by __attribute__ ((constructor)) or by adding a function
> pointer to the .init section.

If __attribute__ ((constructor)) is used you need a pre-compiled object file (written in C) which you can link all your D dynamic libraries with?

> But the issue is initialization for already running threads.

When druntime is first initialized there won't be any D threads and the runtime doesn't need to care about non-D threads?

-- 
/Jacob Carlborg
December 13, 2011
On Tue, 13 Dec 2011 15:53:22 +0100, Jacob Carlborg <doob@me.com> wrote:

> On 2011-12-13 15:15, Martin Nowak wrote:
>> TLS is not too difficult. We can either use bracketed sections again or
>> let the compiler emit small thunks that fetch the complete TLS section for
>> a executable/DSO. The important point is that fetching the TLS addresses
>> must
>> be done for each thread and from a local function within each DSO.
>> people.redhat.com/drepper/tls.pdf
>> My current approach is to register callbacks, and let the threads update
>> their ranges before GC.
>> Not sure, but it will probably causes eagerly allocation of the TLS blocks.
>
> Mac OS X doesn't support TLS (it does now in 10.7) so Walter rolled his own implementation. I don't know how close that is to the EFL implementation specified in the above link.
>
Which would open some room for hooks.

>> Some similar issues happen with the other section brackets (.deh, .minfo).
>> We need local symbols as brackets not global ones as they would collide.
>
> There is no need for brackets, they just cause problems. The correct way to get section data on Mac OS X is to use the "getsectbynamefromheader" function. You can have a look at my fork to see how its used.
>
Correctly used they don't and are a really cheap way to get the ranges.
Having an OS function to do this is nice but as far as I know it doesn't port
to ELF (without linking in some elf libs) while the C object ports to OSX.

> https://github.com/jacob-carlborg/druntime/blob/dylib/src/rt/memory.d#L318
>
>> I've tried several approaches.
>
> They way I've done it is to register a callback for the "_dyld_register_func_for_add_image" function. The documentation describes the function as:
>
> "When you call _dyld_register_func_for_add_image, the dynamic linker runtime calls the specified callback (func) once for each of the images that are currently loaded into the program. When a new image is added to the program, your callback is called again with the mach_header for the new image, and the virtual memory slide amount of the new image."
>
> In this callback I extract module infos, exception handling tables and TLS data from the mach_header. This is now where the problem comes. I need an associative array which maps mach_headers to a struct (or similar) containing module infos, exception handling tables and TLS data. But I don't have an associative array that works this earily in the initialization phase of the runtime.
But it will get called from the thread that runs dlopen.
So how do you add/remove an EH table while another thread is
catching exceptions?

>
> The next problem is with accessing TLS. When a TLS variable is accessed the compiler calls the ___tls_get_addr function, implemented in druntime. This function receives the address and then converts it to TLS using the TLS data bracketed in the object file. In this function I need to somehow access the current object file/mach_header, then use this mach_header to access the TLS data in the associative array mentioned above.
Yeah, __tls_get_addr on OSX will require kind of a DSO specific handle or
call a DSO local function, i.e. C static.
>
> You can see my current implementation in my fork, everything below this line:
>
> https://github.com/jacob-carlborg/druntime/blob/dylib/src/rt/memory.d#L267
>
> As you can see I don't have an associative array so the above currently doesn't work.
December 13, 2011
On 2011-12-13 16:29, Martin Nowak wrote:
> Correctly used they don't and are a really cheap way to get the ranges.
> Having an OS function to do this is nice but as far as I know it doesn't
> port
> to ELF (without linking in some elf libs) while the C object ports to OSX.

First, it does cause problems because you will get missing symbols for _minfo_beg and _minfo_end when it's compiled as a dynamic library, trust me, I've already done this for Tango. Second, It doesn't have to port to EFL, it's already platform specific.

>> In this callback I extract module infos, exception handling tables and
>> TLS data from the mach_header. This is now where the problem comes. I
>> need an associative array which maps mach_headers to a struct (or
>> similar) containing module infos, exception handling tables and TLS
>> data. But I don't have an associative array that works this earily in
>> the initialization phase of the runtime.
> But it will get called from the thread that runs dlopen.
> So how do you add/remove an EH table while another thread is
> catching exceptions?

I have no idea from what thread that callback will be called from. I haven't thought about what should happen.

-- 
/Jacob Carlborg
December 13, 2011
On 12/10/2011 5:33 PM, Adam Wilson wrote:
> On Sat, 10 Dec 2011 11:36:19 -0800, Trass3r <un@known.com> wrote:
>
>>> Floating point arithmetic benchmarks have long hurt us,
>>> and 64-bit generation on OSX is a gating issue.
>>
>> The disastrous Windoze toolchain as well.
>> F*** OMF.
>
> +1! x64 on Windows will never happen until Optlink is overhauled or COFF
> is added to the Backend.
>

Yes. Please for the love of god be compatible with the rest of everything.
December 13, 2011
On 12/11/2011 4:13 AM, Paulo Pinto wrote:
> Am 10.12.2011 21:35, schrieb Andrei Alexandrescu:
>> On 12/10/11 2:22 PM, maarten van damme wrote:
>>> Just for fun I
>>> wanted to create a program as little as possible, compiled without
>>> garbage collector/phobos/... and it turned out that compiling without
>>> garbage collector is pretty much impossible (memory leaks all around the
>>> place in druntime). dynamic linking for the d standard library would be
>>> a great new option for a dmd release in the future :).
>>
>> Using D without GC is an interesting direction, and dynamic linking
>> should be available relatively soon.
>>
>>
>> Andrei
>
> As a long time beliver in systems programming languages with GC support
> (Modula-3, Oberon, Sing#, ...), I think allowing this in D is the wrong
> direction.
>
> Sure provinding APIs to control GC behavior makes sense, but not turn it
> off, we already have enough languages to do systems programming without GC.

If you are doing Embedded Systems programming Dynamic memory and GC are pretty much a no no. Perhaps the compiler needs a No GC check to make sure when flaged that no GC/GC Managed memory is used. This would disable soem features but allow for this need.
December 13, 2011
On 12/11/2011 11:32 AM, Andrei Alexandrescu wrote:
> On 12/11/11 12:17 PM, Chad J wrote:
>> On 12/11/2011 11:21 AM, Andrei Alexandrescu wrote:
>>> On 12/11/11 9:14 AM, Chad J wrote:
>>>>
>>>> I think one thing that could would be (optional) reference counting for
>>>> transitively atomic types. Of course, this is just another kind of
>>>> garbage collection, but it is /deterministic/ and parallelizes well,
>>>> and
>>>> I bet this would open up a large amount of Phobos while the
>>>> stop-the-world collector is offline, and with little or no code change.
>>>
>>> I think the language has enough (in theory) and nearly enough (in
>>> practice) expressiveness to implement reference counted classes in a
>>> library, virtually transparently. I mean one could define a template
>>> RefCounted (an extended version of what's now in std) such that this
>>> pattern is possible:
>>>
>>> // client code
>>> class WidgetImpl {
>>> ...
>>> }
>>>
>>> alias RefCounted!WidgetImpl Widget;
>>>
>>> Aside from using WidgetImpl in type relationships, using Widget would
>>> transparently insert the appropriate reference counting without
>>> otherwise interfering with normal WidgetImpl semantics.
>>>
>>>
>>> Andrei
>>
>> This is good. So stick it in the runtime and make the compiler emit
>> RefCounted!___ templates whenever it finds a transitively atomic type.
>>
>> This can be done alongside the existing garbage collector.
>>
>> Then we can benchmark different combinations of memory management
>> strategies.
>>
>> Two compiler flags can work in tandem:
>> -nogc
>> -norefcount
>>
>> (Turn them both off if you really want to break Phobos.)
>
> We considered that route, and concluded it's marred by too many issues.
>
> Andrei
>

What issues did it cause?