On Nov 9, 2010, at 3:47 AM, Michel Fortin wrote:

> Le 2010-11-09 ? 3:33, Jacob Carlborg a ?crit :
> 
>> See my replay to Michel Fortin.
>> 
>> Just out of curiosity why does DMD use this approach with begin, content, end to determine a section on Mac OS X? Looking at the source code for object_.d in Tango for LDC, the same implementation is used for all Posix systems and that is the same approach which is used in druntime for Linux, Solaris and FreeBSD.
> 
> I believed the reason was that Walter could not find at the time a way to get the size of a section, so he developed this little hack.
> 
> Now, that's surprising because if you look at rt/memory_osx.c in druntime you'll find it does exactly that: it calls getsectbynamefromheader to get a section's address and size to add ranges to the GC, and _dyld_register_func_for_add_image/_dyld_register_func_for_remove_image to register callbacks for when an image is loaded/unloaded. That file's authors are Walter Bright and Sean Kelly. I don't see a reason the same technique can't be used to handle modules infos, tls, and the like.

It certainly could.  The current approach is used mostly because it works everywhere, weird OSX linker behavior notwithstanding.

> Looking at druntime makes me wonder why the initialization code is scattered everywhere... surely it'd be easier to figure out these kinds of problems if initialization was all happening in one place.

Better dynamic library support will probably mean integrating these two bits of code somehow.

Jacob Carlborg wrote:
> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code:
>
> http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 .
>
> The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
>
> 

Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.

On Nov 9, 2010, at 11:55 AM, Walter Bright wrote:
> 
> Jacob Carlborg wrote:
>> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 . The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
> 
> Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.

The simplest (appropriate) fix for the way things work today would probably be to add a new compiler runtime routine like "extern (C) void[] rt_tlsseg()".  Then all of the compiler-specific logic for TLS segment management can be moved into src/rt/memory.d (and possibly memory_osx.c).  I've been meaning to do something about this code being in core.thread anyway, since it really is compiler-specific.

Sean Kelly wrote:
> On Nov 9, 2010, at 11:55 AM, Walter Bright wrote:
> 
>> Jacob Carlborg wrote:
>> 
>>> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code:
>>> http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 .
>>> The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
>>> 
>> Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.
>> 
>
> The simplest (appropriate) fix for the way things work today would probably be to add a new compiler runtime routine like "extern (C) void[] rt_tlsseg()".  Then all of the compiler-specific logic for TLS segment management can be moved into src/rt/memory.d (and possibly memory_osx.c).  I've been meaning to do something about this code being in core.thread anyway, since it really is compiler-specific.
>
> 

The segment begin/end issue is exactly the same for exception handling and moduleinfo. A solution for one should do all three.

I'll see what I can do.

On 9 nov 2010, at 20:55, Walter Bright wrote:

> 
> 
> Jacob Carlborg wrote:
>> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 . The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
>> 
>> 
> 
> Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:

* I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?

* If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?

* This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?

* What to name the function, where to put it and when to call it?

On 9 nov 2010, at 21:23, Walter Bright wrote:

> 
> 
> Sean Kelly wrote:
>> On Nov 9, 2010, at 11:55 AM, Walter Bright wrote:
>> 
>>> Jacob Carlborg wrote:
>>> 
>>>> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 . The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
>>>> 
>>> Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.
>>> 
>> 
>> The simplest (appropriate) fix for the way things work today would probably be to add a new compiler runtime routine like "extern (C) void[] rt_tlsseg()".  Then all of the compiler-specific logic for TLS segment management can be moved into src/rt/memory.d (and possibly memory_osx.c).  I've been meaning to do something about this code being in core.thread anyway, since it really is compiler-specific.
>> 
>> 
> 
> The segment begin/end issue is exactly the same for exception handling and moduleinfo. A solution for one should do all three.
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :

> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
> 
> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?

The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.


> * If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?

That's what the Objective-C runtime does (it stores them in a linked list and compare linearly all the entries in the unload callback).


> * This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?

I think you can. I see nowhere in the Objective-C runtime where this is checked, and it'd just be quite strange if the dynamic linker did that.


> * What to name the function, where to put it and when to call it?

It's called 'map_image' and 'unmap_image' in the Objective-C runtime. But I don't know how they should be named in Druntime.


-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/

I'll have to leave these to Sean, I am not very competent on how things work on OSX.

One way to deal with the moduleinfos is to have it be an array of array of moduleinfos, then adding/removing as .so's are added/removed becomes a cinch. Using associative arrays may produce problems with initialization - if a future aa implementation requires module construction.

Jacob Carlborg wrote:
> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>
> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
>
> * If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?
>
> * This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?
>
> * What to name the function, where to put it and when to call it?
>
> On 9 nov 2010, at 21:23, Walter Bright wrote:
>
> 
>> Sean Kelly wrote:
>> 
>>> On Nov 9, 2010, at 11:55 AM, Walter Bright wrote:
>>> 
>>> 
>>>> Jacob Carlborg wrote:
>>>> 
>>>> 
>>>>> It should work using getsectdatafromheader, that's what I changed when I added support for dynamic libraries to Tango. Have a look at the Tango code: http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/object_.d#L1270 and http://dsource.org/projects/tango/browser/trunk/tango/core/rt/compiler/dmd/darwin/Image.d#L103 . The file in the last link is completely made by me so you won't have to be afraid for looking at it. Or if you don't want to look at Tango source files at all you can look at this druntime change: http://www.dsource.org/projects/druntime/changeset/372 , to be more specific: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/image.d?rev=372#L105
>>>>> 
>>>>> 
>>>> Thanks, Jacob! If you want to submit a patch to druntime that, for OSX, eliminates the dependence on the begin and end sections, that would be awesome.
>>>> 
>>>> 
>>> The simplest (appropriate) fix for the way things work today would probably be to add a new compiler runtime routine like "extern (C) void[] rt_tlsseg()".  Then all of the compiler-specific logic for TLS segment management can be moved into src/rt/memory.d (and possibly memory_osx.c).  I've been meaning to do something about this code being in core.thread anyway, since it really is compiler-specific.
>>>
>>> 
>>> 
>> The segment begin/end issue is exactly the same for exception handling and moduleinfo. A solution for one should do all three.
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>> 
>
> 

On Nov 10, 2010, at 1:55 AM, Jacob Carlborg wrote:

> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
> 
> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?

The real tricky part is adding the new TLS data to running threads.  It might be necessary to call thread_suspendAll() and then some new routine to graft the TLS data in place.  And things get even worse if dlclose is called.

> * If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?

Seems like that should work, since the address will be globally unique... unless dlclose is called, the data is retained, and then dlopen called on a new library.

> * This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?

I think that's a reasonable assumption.

> * What to name the function, where to put it and when to call it?

Probably src/rt/memory_osx.c.  Give it a name like _d_whatever and it can be changed later if needed.

On Nov 10, 2010, at 4:23 AM, Michel Fortin wrote:

> Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :
> 
>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>> 
>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
> 
> The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.

The compiler runtime (src/rt/memory.d) uses a linked list for static data segments... or it used to.  I think it now may simply call gc.addRange.  Either way, I think a linked list is a good approach.

>> * What to name the function, where to put it and when to call it?
> 
> It's called 'map_image' and 'unmap_image' in the Objective-C runtime. But I don't know how they should be named in Druntime.

Oh, it may be appropriate to call this inside rt_loadLibrary(), which I believe is in src/rt/dmain2.d.  That's called by Runtime.loadLibrary().  Is anything else needed, say if a dynamic library is loaded automatically at run time?