I've considered putting a hashtable written in C (or stripped-down D backed by malloc/free) in the runtime for things like this, but I don't think apps will have enough libraries loaded to justify an AA.

On Nov 10, 2010, at 9:42 AM, Walter Bright wrote:

> I'll have to leave these to Sean, I am not very competent on how things work on OSX.
> 
> One way to deal with the moduleinfos is to have it be an array of array of moduleinfos, then adding/removing as .so's are added/removed becomes a cinch. Using associative arrays may produce problems with initialization - if a future aa implementation requires module construction.

On 10 nov 2010, at 18:55, Sean Kelly wrote:

> On Nov 10, 2010, at 4:23 AM, Michel Fortin wrote:
> 
>> Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :
>> 
>>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>>> 
>>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
>> 
>> The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.
> 
> The compiler runtime (src/rt/memory.d) uses a linked list for static data segments... or it used to.  I think it now may simply call gc.addRange.  Either way, I think a linked list is a good approach.

Ok,  I guess a we'll use a linked list.

>>> * What to name the function, where to put it and when to call it?
>> 
>> It's called 'map_image' and 'unmap_image' in the Objective-C runtime. But I don't know how they should be named in Druntime.
> 
> Oh, it may be appropriate to call this inside rt_loadLibrary(), which I believe is in src/rt/dmain2.d.  That's called by Runtime.loadLibrary().  Is anything else needed, say if a dynamic library is loaded automatically at run time?

No, I don't think so. As long as I register a callback it will be called for dynamic libraries loaded at run time.

> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

Ok, thanks for the answers.

On 10 nov 2010, at 13:23, Michel Fortin wrote:

> Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :
> 
>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>> 
>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
> 
> The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.
> 
> 
>> * If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?
> 
> That's what the Objective-C runtime does (it stores them in a linked list and compare linearly all the entries in the unload callback).
> 
> 
>> * This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?
> 
> I think you can. I see nowhere in the Objective-C runtime where this is checked, and it'd just be quite strange if the dynamic linker did that.
> 
> 
>> * What to name the function, where to put it and when to call it?
> 
> It's called 'map_image' and 'unmap_image' in the Objective-C runtime. But I don't know how they should be named in Druntime.
> 
> 
> -- 
> Michel Fortin
> michel.fortin at michelf.com
> http://michelf.com/
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

On 10 nov 2010, at 18:52, Sean Kelly wrote:

> On Nov 10, 2010, at 1:55 AM, Jacob Carlborg wrote:
> 
>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>> 
>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
> 
> The real tricky part is adding the new TLS data to running threads.  It might be necessary to call thread_suspendAll() and then some new routine to graft the TLS data in place.  And things get even worse if dlclose is called.
> 
>> * If we change to an associative array could the image received in the callback function registered by _dyld_register_func_for_add_image be used as the key in the associative array?
> 
> Seems like that should work, since the address will be globally unique... unless dlclose is called, the data is retained, and then dlopen called on a new library.

I don't think that will be a problem. I if dlclose is called I will remove the image from the data structure and if the library is loaded again with dlopen I will at the image to the data structure again.

>> * This is a question about the _dyld_register_func_for_add_image function. Can I assume that all images sent to the registered callback functions are of the same architecture that I'm currently compiling? For example, I'm running a universal binary and it's running the 32bit part of the binary. Then I'm loading a universal dynamic library, it will only load the 32bit part since that's the part I'm running?
> 
> I think that's a reasonable assumption.
> 
>> * What to name the function, where to put it and when to call it?
> 
> Probably src/rt/memory_osx.c.  Give it a name like _d_whatever and it can be changed later if needed.
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-- 
/Jacob Carlborg

On 11/10/10 12:21 PM, Jacob Carlborg wrote:
>
> On 10 nov 2010, at 18:55, Sean Kelly wrote:
>
>> On Nov 10, 2010, at 4:23 AM, Michel Fortin wrote:
>>
>>> Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :
>>>
>>>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>>>>
>>>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
>>>
>>> The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.
>>
>> The compiler runtime (src/rt/memory.d) uses a linked list for static data segments... or it used to.  I think it now may simply call gc.addRange.  Either way, I think a linked list is a good approach.
>
> Ok,  I guess a we'll use a linked list.

I suggest we go with a hash. It costs next to nothing and has no future scalability money. I bet money at least a few companies (Facebook included) would run into severe scalability issues if linear search is to be used.

Andrei

On 10 nov 2010, at 21:44, Andrei Alexandrescu wrote:

> On 11/10/10 12:21 PM, Jacob Carlborg wrote:
>> 
>> On 10 nov 2010, at 18:55, Sean Kelly wrote:
>> 
>>> On Nov 10, 2010, at 4:23 AM, Michel Fortin wrote:
>>> 
>>>> Le 2010-11-10 ? 4:55, Jacob Carlborg a ?crit :
>>>> 
>>>>> I've been thinking about this and I'm trying to think of everything to get this right the first time so I have a couple of questions:
>>>>> 
>>>>> * I though it might be a good idea to add support for running module constructors for dynamically loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to add the new module infos to the array of existing ones and when/if the library is unloaded remove the module infos added by the library. Now for the question: is an array still a good data structure for this or should we use an associative array or something else?
>>>> 
>>>> The Objective-C runtime uses a linked list. I think the expectation is that you won't have thousands of libraries open and that you won't unload them often. But going with an AA doesn't look like a bad idea to me.
>>> 
>>> The compiler runtime (src/rt/memory.d) uses a linked list for static data segments... or it used to.  I think it now may simply call gc.addRange.  Either way, I think a linked list is a good approach.
>> 
>> Ok,  I guess a we'll use a linked list.
> 
> I suggest we go with a hash. It costs next to nothing and has no future scalability money. I bet money at least a few companies (Facebook included) would run into severe scalability issues if linear search is to be used.
> 
> Andrei
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos


I would prefer to use a hash as well but I don't know if the runtime is sufficiently initialized when this operation would be called. I guess Sean or Walter have to answer/decide this.

-- 
/Jacob Carlborg

On Nov 10, 2010, at 12:44 PM, Andrei Alexandrescu wrote:
> 
> I suggest we go with a hash. It costs next to nothing and has no future scalability money. I bet money at least a few companies (Facebook included) would run into severe scalability issues if linear search is to be used.

I can't imagine an app would load more than a few dynamic libraries, and the data shouldn't need random access, should it?  Would performance really be an issue here?

On Nov 10, 2010, at 12:49 PM, Jacob Carlborg wrote:
> 
> I would prefer to use a hash as well but I don't know if the runtime is sufficiently initialized when this operation would be called. I guess Sean or Walter have to answer/decide this.

A C-style hashtable using malloc/free is pretty easy to implement.  I don't think it's safe to use the built-in AA though.

On Wed, 10 Nov 2010 15:49:34 -0500, Jacob Carlborg <doob at me.com> wrote:
>
> I would prefer to use a hash as well but I don't know if the runtime is sufficiently initialized when this operation would be called. I guess Sean or Walter have to answer/decide this.
>

David Simcha has put some work into alternative AA implementations, including a sealed one which should only require malloc/free.

One benefit to using a linked list is that no heap allocations would be necessary.

If you make it a doubly linked list and the method that removes a node from the linked list has a pointer directly to its own node, then no linear search is necessary.  I may be way off on this, I'm not at all familiar with the requirements for loading DLs.

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Wed, November 10, 2010 3:44:47 PM
> Subject: Re: [phobos] Showstopper bug: Hello world fails on OSX!
> 
> On 11/10/10 12:21 PM, Jacob Carlborg wrote:
> >
> > On 10 nov 2010, at  18:55, Sean Kelly wrote:
> >
> >> On Nov 10, 2010, at 4:23 AM, Michel  Fortin wrote:
> >>
> >>> Le 2010-11-10 ? 4:55, Jacob Carlborg a  ?crit :
> >>>
> >>>> I've been thinking about this and  I'm trying to think of everything to
>get this right the first time so I have a  couple of questions:
> >>>>
> >>>> * I though it might  be a good idea to add support for running module
>constructors for dynamically  loaded libraries (i.e. libraries loaded with dlopen). Then I was think I need to  add the new module infos to the array of existing ones and when/if the library  is unloaded remove the module infos added by the library. Now for the question:  is an array still a good data structure for this or should we use an associative  array or something else?
> >>>
> >>> The Objective-C runtime  uses a linked list. I think the expectation is
>that you won't have thousands of  libraries open and that you won't unload them often. But going with an AA  doesn't look like a bad idea to me.
> >>
> >> The compiler runtime  (src/rt/memory.d) uses a linked list for static data
>segments... or it used  to.  I think it now may simply call gc.addRange.  Either way, I think  a linked list is a good approach.
> >
> > Ok,  I guess a we'll use  a linked list.
> 
> I suggest we go with a hash. It costs next to nothing and  has no future scalability money. I bet money at least a few companies  (Facebook included) would run into severe scalability issues if linear  search is to be  used.
> 
> Andrei
> _______________________________________________
> phobos  mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>