Hi Martin,

I'm working on the LDC 2.063 merge right now, and it would obviosuly be nice to finally support shared libraries in LDC as well (PIC/shared object codegen works since ages ago, but runtime support was missing so far).

However, I would like to avoid the use of bracketing sections/symbols for locating the ModuleInfos of all the linked modules (we don't need to handle EH tables anyway). While it is certainly possible to crack open the default LLVM object file emission pipeline and directly write out some custom data in an object file format dependent way, this would also mean that we cannot use the standard LLVM tools for generating object files from bitcode (LLVM IR) anymore, which would be quite annoying for tooling.

Thus, I thought about using a scheme where there is one global (C runtime) ctor per module just like there is in the old one. Instead of setting up the _Dmodule_ref chain directly, it would call the equivalent of _d_dso_registry. If a DSO info record (struct DSO in sections_linux) has not been initialized for the given image yet, the function would set it up, and in any case add the passed ModuleInfo* to a pointer array in the record.

While I'm quite sure this would work for load-time bound shared objects, I didn't really think about the dlopen() case yet. Do you think this would be feasible, or do you have any better ideas?

Thanks,
David
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 14 June 2013 13:58, David Nadlinger <code@klickverbot.at> wrote:
> Hi Martin,
>
> I'm working on the LDC 2.063 merge right now, and it would obviosuly be nice to finally support shared libraries in LDC as well (PIC/shared object codegen works since ages ago, but runtime support was missing so far).
>

Further to this, have you got some documentation somewhere on how DMD's compiler/runtime solution ties together?

>
> While I'm quite sure this would work for load-time bound shared objects, I didn't really think about the dlopen() case yet. Do you think this would be feasible, or do you have any better ideas?
>

dlopen() should call the all .ctors on load no problem.


--
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On Fri, Jun 14, 2013 at 4:09 PM, Iain Buclaw <ibuclaw@ubuntu.com> wrote:
> Further to this, have you got some documentation somewhere on how DMD's compiler/runtime solution ties together?

No, unfortunately. The mechanism isn't really complex, though, so just grep the DMD source for "d_dso_registry" and it should be rather obvious from there.

David
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

Am 14.06.2013 14:58, schrieb David Nadlinger:
> Thus, I thought about using a scheme where there is one global (C
> runtime) ctor per module just like there is in the old one. Instead of
> setting up the _Dmodule_ref chain directly, it would call the
> equivalent of _d_dso_registry. If a DSO info record (struct DSO in
> sections_linux) has not been initialized for the given image yet, the
> function would set it up, and in any case add the passed ModuleInfo*
> to a pointer array in the record.
>
Instead of emitting one constructor per module you could also create a special object file with that constructor, then always link against that object file when compiling a shared library.

This is also what gcc -shared does: It links in crtbeginS.o and crtendS.o.

-- 
Johannes Pfau

_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

Am 17.06.2013 14:45, schrieb Johannes Pfau:
> Am 14.06.2013 14:58, schrieb David Nadlinger:
>> Thus, I thought about using a scheme where there is one global (C
>> runtime) ctor per module just like there is in the old one. Instead of
>> setting up the _Dmodule_ref chain directly, it would call the
>> equivalent of _d_dso_registry. If a DSO info record (struct DSO in
>> sections_linux) has not been initialized for the given image yet, the
>> function would set it up, and in any case add the passed ModuleInfo*
>> to a pointer array in the record.
>>
> Instead of emitting one constructor per module you could also create a special object file with that constructor, then always link against that object file when compiling a shared library.
>
> This is also what gcc -shared does: It links in crtbeginS.o and crtendS.o.
>
Of course then you couldn't access all ModuleInfos... sorry for the noise.

It could actually be implemented similar to crtbeginS.o and crtendS.o which are used to run the global c constructors though, but then it's platform specific again.

-- 
Johannes Pfau

_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 06/14/2013 02:58 PM, David Nadlinger wrote:
> Hi Martin,
Sorry for the delay, I got pretty busy recently.
> I'm working on the LDC 2.063 merge right now, and it would obviosuly
> be nice to finally support shared libraries in LDC as well (PIC/shared
> object codegen works since ages ago, but runtime support was missing
> so far).
>
> However, I would like to avoid the use of bracketing sections/symbols
> for locating the ModuleInfos of all the linked modules (we don't need
> to handle EH tables anyway). While it is certainly possible to crack
> open the default LLVM object file emission pipeline and directly write
> out some custom data in an object file format dependent way, this
> would also mean that we cannot use the standard LLVM tools for
> generating object files from bitcode (LLVM IR) anymore, which would be
> quite annoying for tooling.
>
> Thus, I thought about using a scheme where there is one global (C
> runtime) ctor per module just like there is in the old one. Instead of
> setting up the _Dmodule_ref chain directly, it would call the
> equivalent of _d_dso_registry. If a DSO info record (struct DSO in
> sections_linux) has not been initialized for the given image yet, the
> function would set it up, and in any case add the passed ModuleInfo*
> to a pointer array in the record.
I don't think bracketing section are the problem (except for the ld bugs in older OSX versions).
In fact they come with two benefits.
There is only a single call that contains all information about a shared library at once.
Because the call is the only one you can use it to perform initialization.
When you have one call per module you first have to gather all information and can
only perform initialization when it's complete.
So this performs worse and is annoying to implement.

The basic problem to solve is that _minfo* sections must be merged by the linker but _minfo_beg and _minfo_end must not be be external symbols as the executable would superimpose them.
This is currently done with section groups of internal symbols.
I recently had the idea that it should also work with hidden external symbols but I still have to verify that.
If it works we could move the functionality into the glue layer so that it can hopefully be reused by GDC and LDC.
Otherwise we could also try to add support for section groups and internal symbols in the glue layer but that might be a lot of work.

Best,
Martin

_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 07/24/2013 04:42 PM, Martin Nowak wrote:
> The basic problem to solve is that _minfo* sections must be merged by
> the linker but _minfo_beg and _minfo_end must not be be external
> symbols as the executable would superimpose them.
> This is currently done with section groups of internal symbols.
> I recently had the idea that it should also work with hidden external
> symbols but I still have to verify that.
> If it works we could move the functionality into the glue layer so
> that it can hopefully be reused by GDC and LDC.
> Otherwise we could also try to add support for section groups and
> internal symbols in the glue layer but that might be a lot of work.
>
> Best,
> Martin
>
I tried that out and it works.
The linker will correctly merge the external weak symbols, but because
they are hidden they remain inaccessible
from outside of a shared library. Thus the runtime linker won't
interpose the bracketing symbols with the ones from
the executable or another shared library.
This should be fairly simple to implement in the glue layer, we only
need to mark the COMDAT variables as hidden.

On Mon, Jun 17, 2013 at 2:52 PM, Johannes Pfau <johannespfau@googlemail.com> wrote:
> Of course then you couldn't access all ModuleInfos... sorry for the noise.
>
> It could actually be implemented similar to crtbeginS.o and crtendS.o which are used to run the global c constructors though, but then it's platform specific again.

Mhm, at least on Linux, crtbegin.o/crtend.o are special-cased in the linker script so that their contents end up at the appropriate place in the executable (i.e. bracketing the .cdots/.dtors.

David
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

Hi Martin,

Sorry for not getting back to you earlier (not really following the lists right now), but while this is a nice idea to reduce the implementation complexity, I can't quite see how you would ensure that the ModuleInfos actually end up between the bracketing symbols without directly controlling the order in which they are written out (see the original post on why this is problematic for LDC). Maybe I'm missing something obvious?

David
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime

On 08/26/2013 12:40 AM, David Nadlinger wrote:
> Hi Martin,
>
> Sorry for not getting back to you earlier (not really following the
> lists right now), but while this is a nice idea to reduce the
> implementation complexity, I can't quite see how you would ensure that
> the ModuleInfos actually end up between the bracketing symbols without
> directly controlling the order in which they are written out (see the
> original post on why this is problematic for LDC). Maybe I'm missing
> something obvious?
>
> David
>
So you can't control the order in which sections are created?
This makes me wonder because defining variables in C code
with attribute section will also preserve the order.
_______________________________________________
D-runtime mailing list
D-runtime@puremagic.com
http://lists.puremagic.com/mailman/listinfo/d-runtime