On 2013-12-19 20:51, Johannes Pfau wrote:

> Sounds like that could work. But as the module section is a custom
> section anyway we wouldn't have to replace/modify the default linker
> script - we can pass a custom script to ld which just handles the
> ".minfo" section. That should be just as portable as relying on the
> "don't reorder sections" behavior: Works everywhere where GNU Binutils
> LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels
> like a hack imho. I'm also not sure if we can control the order in
> which sections are emitted in GCC)

The trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.

> It seems like getting the TLS section is the more interesting part. We
> can't emit sections around the TLS section so IIRC the current dmd
> implementation therefore relies on the runtime linker and libc specific
> interfaces?

Yes, using dl_iterate_phdr: http://linux.die.net/man/3/dl_iterate_phdr.

> I think asking the binutils maintainers to add __tdata_begin,
> __tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss
> sections would be a nice long-term solution, or is there some issue
> with that?

How does that work? Do people need to update their linkers for that to work?

-- 
/Jacob Carlborg

On 2013-12-19 18:59, David Nadlinger wrote:

> That is pretty much how it is done in DMD and LDC as well. The only
> difference is that instead of modifying the linker scripts to
> accommodate this, we emit the ModuleInfo references to custom sections.
> The GNU toolchain (and we are in highly system-specific territory here
> anyway) never changes the order of custom sections, which you can also
> verify using __attribute__((section("...))) in GCC. Thus, if you emit
> your relevant symbols into three sections like this,

Does that really work reliably? As far as I remember, the trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.

But the dynamic linker on Mac OS X has an API to access these sections easily anyway.

-- 
/Jacob Carlborg

Am Fri, 20 Dec 2013 08:40:28 +0100
schrieb Jacob Carlborg <doob@me.com>:

> > I think asking the binutils maintainers to add __tdata_begin, __tdata_end, __tbss_begin and __tbss_end markers to the tdata and tbss sections would be a nice long-term solution, or is there some issue with that?
> 
> How does that work? Do people need to update their linkers for that to work?
> 

Likely yes, that's why I said 'longterm' solution. You need to edit the default linker script. Then there are 2 options: We can force certain object files to be at the start or the end of the section. So we could introduce dso_start.o and dso_end.o files containing the start/end sections and those would always be placed as the first/last objects in a section. (This is how the C++/C runtime calls the constructors in the .ctors section: crtbegin.o contains a __CTOR_LIST__ symbol)

It's also possible to directly instruct the linker to insert a symbol at the start of the section and at the end of the section. This way you don't need special object files but the symbols will appear always, even for C/C++ programs.

In theory it's always possible to use your own linker script without updating the compiler, but that's not a practical solution. Implicit linker-script which only extend the default script can be used for custom sections, but AFAIK this doesn't work for standard sections.

Am Fri, 20 Dec 2013 08:43:12 +0100
schrieb Jacob Carlborg <doob@me.com>:

> On 2013-12-19 18:59, David Nadlinger wrote:
> 
> > That is pretty much how it is done in DMD and LDC as well. The only difference is that instead of modifying the linker scripts to accommodate this, we emit the ModuleInfo references to custom sections. The GNU toolchain (and we are in highly system-specific territory here anyway) never changes the order of custom sections, which you can also verify using __attribute__((section("...))) in GCC. Thus, if you emit your relevant symbols into three sections like this,
> 
> Does that really work reliably? As far as I remember, the trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.
> 
> But the dynamic linker on Mac OS X has an API to access these sections easily anyway.
> 

Seems like the OSX linker is not based on the GNU linker and doesn't have linker scripts either. There's some "sectorder" argument which might be useful though.

But relying on the runtime on OSX isn't bad as there's only one runtime anyway. On linux we have different libcs and non-standard interfaces which isn't optimal.

On 20 December 2013 07:40, Jacob Carlborg <doob@me.com> wrote:
> On 2013-12-19 20:51, Johannes Pfau wrote:
>
>> Sounds like that could work. But as the module section is a custom section anyway we wouldn't have to replace/modify the default linker script - we can pass a custom script to ld which just handles the ".minfo" section. That should be just as portable as relying on the "don't reorder sections" behavior: Works everywhere where GNU Binutils LD/GOLD are used. (Emitting 3 sections is a clever trick, but it feels like a hack imho. I'm also not sure if we can control the order in which sections are emitted in GCC)
>
>
> The trick with brackting sections doesn't work on Mac OS X. DMD had some problems with that. It basically broke on every new major release of Mac OS X, the linker changed all the time. If I recall correctly it either removed the empty sections or reordered the sections.
>
>

If using GDC on OSX, then you be going through the emutls module I've got set-up.

As for module discovery, we already generate this in the GDC backend, which makes it then the job of runtime to pick-up, sort and run the ctors on all modules:

@attribute("constructor")
void __modinit()
{
  extern (C) __gshared ModuleReference* _Dmodule_ref;
  static private ModuleReference __mymod = ModuleReference(null, minfo);

  __mymod.next = _Dmodule_ref;
  _Dmodule_ref = &__mymod;
}



Unless I'm missing something in how module discovery is supposed to work, this could be instead amended to:

@attribute("constructor")
void __modinit()
{
  extern (C) void __register_module(ModuleInfo *);

  __register_module(&minfo);
}

So that any lazily loaded modules would be recorded to an existing list and trigger a run-once ctor run.

Regards
Iain

On 2013-12-20 10:27, Johannes Pfau wrote:

> Seems like the OSX linker is not based on the GNU linker and doesn't
> have linker scripts either. There's some "sectorder" argument which
> might be useful though.

It's probably BSD.

> But relying on the runtime on OSX isn't bad as there's only one runtime
> anyway. On linux we have different libcs and non-standard interfaces
> which isn't optimal.

Ok, I see.

-- 
/Jacob Carlborg

On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:
> As for module discovery, we already generate this in the GDC backend,
> which makes it then the job of runtime to pick-up, sort and run the
> ctors on all modules:
>
> @attribute("constructor")
> void __modinit()
> {
>   extern (C) __gshared ModuleReference* _Dmodule_ref;
>   static private ModuleReference __mymod = ModuleReference(null, minfo);
>
>   __mymod.next = _Dmodule_ref;
>   _Dmodule_ref = &__mymod;
> }
>
>
>
> Unless I'm missing something in how module discovery is supposed to
> work, this could be instead amended to:
>
> @attribute("constructor")
> void __modinit()
> {
>   extern (C) void __register_module(ModuleInfo *);
>
>   __register_module(&minfo);
> }
>
> So that any lazily loaded modules would be recorded to an existing
> list and trigger a run-once ctor run.

The _Dmodule_ref approach is virtually the same as we have been doing in LDC up to now. The problem with regard to runtime loading of shared libraries is that you need to run certain code (i.e. the current _d_dso_registry in DMD's/LDC's druntime) when you know that you have collected all the modules. You can't do this from a .ctor, because if you also use them for constructing the module list, you'd need to call the registry function in the last .ctor to run. And at least if you don't know the order in which they are written to the executable, you can't know that.

It might be possible to work around this requirement by rewriting the druntime internals to perform all the collision checking, etc. one-by-one, but I didn't investigate this in more detail.

David

On 20 December 2013 13:06, David Nadlinger <code@klickverbot.at> wrote:
> On Friday, 20 December 2013 at 10:54:56 UTC, Iain Buclaw wrote:
>>
>> As for module discovery, we already generate this in the GDC backend, which makes it then the job of runtime to pick-up, sort and run the ctors on all modules:
>>
>> @attribute("constructor")
>> void __modinit()
>> {
>>   extern (C) __gshared ModuleReference* _Dmodule_ref;
>>   static private ModuleReference __mymod = ModuleReference(null, minfo);
>>
>>   __mymod.next = _Dmodule_ref;
>>   _Dmodule_ref = &__mymod;
>> }
>>
>>
>>
>> Unless I'm missing something in how module discovery is supposed to work, this could be instead amended to:
>>
>> @attribute("constructor")
>> void __modinit()
>> {
>>   extern (C) void __register_module(ModuleInfo *);
>>
>>   __register_module(&minfo);
>> }
>>
>> So that any lazily loaded modules would be recorded to an existing list and trigger a run-once ctor run.
>
>
> The _Dmodule_ref approach is virtually the same as we have been doing in LDC up to now. The problem with regard to runtime loading of shared libraries is that you need to run certain code (i.e. the current _d_dso_registry in DMD's/LDC's druntime) when you know that you have collected all the modules. You can't do this from a .ctor, because if you also use them for constructing the module list, you'd need to call the registry function in the last .ctor to run. And at least if you don't know the order in which they are written to the executable, you can't know that.
>
> It might be possible to work around this requirement by rewriting the druntime internals to perform all the collision checking, etc. one-by-one, but I didn't investigate this in more detail.
>
> David

It should be possible if it's druntime handling all module loading (if you circumvent the module load handlers, don't expect it to work properly).

eg:

loadModule(mod);  // .ctors are ran and modules self register
themselves to 'mod'
mod.sortCtors();
mod.runSharedCtors();
// etc...

On Friday, 20 December 2013 at 14:55:46 UTC, Iain Buclaw wrote:
> It should be possible if it's druntime handling all module loading (if
> you circumvent the module load handlers, don't expect it to work
> properly).
>
> eg:
>
> loadModule(mod);  // .ctors are ran and modules self register
> themselves to 'mod'
> mod.sortCtors();
> mod.runSharedCtors();
> // etc...

If you just load modules using druntime (or another helper function we control), then you can of course do the initialization like this.

You'd lose the benefit of seamlessly being able to load D libraries from e.g. C/C++ plugin hosts, though.

David

On 2013-12-20 10:13, Johannes Pfau wrote:

> Likely yes, that's why I said 'longterm' solution. You need to edit the
> default linker script. Then there are 2 options: We can force certain
> object files to be at the start or the end of the section. So we could
> introduce dso_start.o and dso_end.o files containing the start/end
> sections and those would always be placed as the first/last objects in
> a section. (This is how the C++/C runtime calls the constructors in the
> .ctors section: crtbegin.o contains a __CTOR_LIST__ symbol)
>
> It's also possible to directly instruct the linker to insert a symbol
> at the start of the section and at the end of the section. This way you
> don't need special object files but the symbols will appear always,
> even for C/C++ programs.
>
> In theory it's always possible to use your own linker script without
> updating the compiler, but that's not a practical solution. Implicit
> linker-script which only extend the default script can be used for
> custom sections, but AFAIK this doesn't work for standard sections.

Ok, I see.

-- 
/Jacob Carlborg