On 2012-01-03 22:46, Sean Kelly wrote:
> __thread is supported under Lion via Clang.

Yeah, yeah I know, sorry. I'm keep forgetting that. But we're currently not using that and we're still rolling our own implementation.

-- 
/Jacob Carlborg

On 01/02/2012 08:20 PM, Martin Nowak wrote:
> I think that I'll defer the support for runtime loading of shared
> library (plugins)
> in favor of getting linked shared library support done now.
> There are several issues that require more thoughts.
>
>   - Per-thread initialization of modules is somewhat tricky.
>     Doing it in Runtime.loadLibrary requires knowledge of shared library
> dependencies
>     because different threads might share dependencies but this is not
> provided by libc/libdl.
>
>   - D libraries used by a C library should provide proper runtime
> initialization
>     even if the C library is used by a D application.
>

I now think we should do the initialization directly when loading, even though that might cause deadlocks in certain scenarios. It would solve all mentioned initialization issues. This will also serialize initialization on the loader mutex but I don't expect this to become a performance issue.

>   - Getting symbols through mangled names is difficult/unstable.
>

ref T loadSym(T)(string fqn);
void loadSym(T)(string fqn, out T res);

T loadFunc(T)(string fqn);
void loadFunc(T)(string fqn, out T res);

auto lval = loadSym!int("std.foo.bar");
loadSym("std.foo.bar", lval);

auto fptr = loadFunc!(bool function(string) pure)("std.foo.func");
loadFunc("std.foo.func", fptr);

>   - Libraries might not be unloaded as long as GC collected class
> instances still exist because
>     finalization fails otherwise.
>

We should extend the GC to allow marking and finalizing of foreign memory. This is needed for shared memory too.

// gc.gc
// ...
void trackRange(void* p, size_t sz, void delegate(void*, size_t) dg);
// ...

auto p = malloc(1000);
GC.trackRange(p, 1000, (p, sz) => .free(p));

> Any ideas or use-cases for plugins are welcome.
>

Still holds.

> martin

Am Thu, 07 Mar 2013 12:13:03 +0100
schrieb Martin Nowak <code@dawg.eu>:

> On 01/02/2012 08:20 PM, Martin Nowak wrote:
> 
> I now think we should do the initialization directly when loading, even though that might cause deadlocks in certain scenarios. It would solve all mentioned initialization issues. This will also serialize initialization on the loader mutex but I don't expect this to become a performance issue.

That's the approach used by other runtime libraries, right? So it should work in most cases.

> 
> >   - Getting symbols through mangled names is difficult/unstable.
> >

With D methods the problem is that the mangled name contains the module name, so it'll change for every plugin. This is something a reflection library has to solve:

@plugin("testFunction") void testFunction(){};

auto lib = SharedLib.load();
reflect.getFunction!(void function())("testFunction", lib);

Where the reflection lib would have to build a name->function pointer
map in the shared library and export a C function "void*
getFunction(string name)" which can be accessed via dlsym. (Some more
work could be done to ensure type safety)
(or we just use extern(C) functions)

For OOP features it might be nice if we can iterate all classes that implement an Interface

foreach(IWebPage page; lib)

Although the question then is how to construct instances of these classes. Maybe the factory pattern is better for this.

> 
> >   - Libraries might not be unloaded as long as GC collected class
> > instances still exist because
> >     finalization fails otherwise.
> >
> 
> We should extend the GC to allow marking and finalizing of foreign memory. This is needed for shared memory too.

Would it be possible to implement this in a way that shared libraries can be used without the GC in code which doesn't use the GC (of course with a loss of convenience & safety?)

For example we could introduce this API:

/**
 * Force unload library. All types contained in this library must have
 * been finalized by now
 */
@system void forceUnload(SharedLib l);

/**
 * Internally called by the gc on every scan
 */
@safe private void gcCheckUnload()
{
   foreach(SharedLibrary l; loadedLibs)
     if(shouldCollect(l))
         forceUnload(l);
}

/**
 * Optional additional method to force unloading with GC help:
 * Finalize all types from this SharedLib, then unload. If user
 * code still keeps a reference to a type from this lib it'll crash.
 */
@system void forceGCUnload(SharedLib l)
{
    foreach(void* instance; objects)
    {
        if(inLib(instance), l))
            rt_finalize2(instance);
    }
    forceUnload(l);
}

As far as I understand the issue, C is necessary if we use the GC but want to force-unload a library: Even if user code no longer references types from the shared lib the GC might not have ran the finalizers yet. So this methods finalizes all objects from the shared lib (even if the user still keeps a reference) and then unloads the lib.

I think this is useful in a environment with a massive number of loaded / unloaded plugins. For example webpages could be written as plugins and the webserver could load/unload those. The webserver has to make sure that the libraries are really unloaded to free some resources (AFAIR mono/asp.net implements webpages as such plugins)

> 
> > Any ideas or use-cases for plugins are welcome.
> >
> 
> Still holds.

Use cases?

* Web pages as plugins
* video/audio filtering framework (avisynth, gstreamer)
* syntax highlighting for unknown languages in text editor
* adding support for a new languages / tools to IDEs
* Filters in image editing tools
* plugins for webbrowsers (adblocker, dom manipulation)
* Adding new import / export formats to almost any kind of
  editor / viewer
* Adding web site / media information scrapers to media players (xbmc,
  rhythmbox)

On 2013-03-07 14:53, Johannes Pfau wrote:

> For OOP features it might be nice if we can iterate all classes that
> implement an Interface
>
> foreach(IWebPage page; lib)
>
> Although the question then is how to construct instances of these
> classes. Maybe the factory pattern is better for this.

Just iterate all classes, like ClassInfo.find does. Add a condition to check if a given class implements a given interface. If the correct class if found create it with Object.factory. Tango contains a module, tango.core.RuntimeTraits, with a function that checks if a class implements an interface. I'm pretty sure that Phobos has something similar.

We could create a standard plugin interface in Phobos or druntime, something like:

interface Plugin
{
    void initialize ();
    void terminate ();
}

Every plugin should implement this directly or indirectly. "initialize" would be called just after the plugin is loaded and "terminate" would be called just before unloading.

-- 
/Jacob Carlborg

On 1/2/2012 11:20 AM, Martin Nowak wrote:
> I think that I'll defer the support for runtime loading of shared library (plugins)
> in favor of getting linked shared library support done now.
> There are several issues that require more thoughts.
>
>   - Per-thread initialization of modules is somewhat tricky.
>     Doing it in Runtime.loadLibrary requires knowledge of shared library
> dependencies
>     because different threads might share dependencies but this is not provided
> by libc/libdl.
>
>   - Libraries might not be unloaded as long as GC collected class instances
> still exist because
>     finalization fails otherwise.
>
>   - Getting symbols through mangled names is difficult/unstable.
>
>   - D libraries used by a C library should provide proper runtime initialization
>     even if the C library is used by a D application.
>
> Any ideas or use-cases for plugins are welcome.

My idea for this is straightforward. A D plugin should use a C interface. That means, for example, that a D plugin should NOT pass GC allocated references to its caller. Only malloc'd data should pass between the plugin and its caller.

The D plugin could still use the shared druntime.so, and could share that usage with the D caller of the plugin.

On 1/2/2012 11:20 AM, Martin Nowak wrote:
>   - Libraries might not be unloaded as long as GC collected class instances
> still exist because
>     finalization fails otherwise.

D doesn't guarantee that finalizers will run on GC allocated objects. Therefore, when unloading a dll:

1. run a gc collection
2. for all objects remaining on the heap
       if they have a finalizer and that finalizer points into the dll code
            mark them as not having a finalizer

On Thursday, March 07, 2013 13:48:21 Walter Bright wrote:
> My idea for this is straightforward. A D plugin should use a C interface. That means, for example, that a D plugin should NOT pass GC allocated references to its caller. Only malloc'd data should pass between the plugin and its caller.
> 
> The D plugin could still use the shared druntime.so, and could share that usage with the D caller of the plugin.

That may very well end up being a necessary restriction, but not being able to pass
GC allocated objects across library boundaries would be a _huge_ downside to using
shared libraries if that happens. It also would likely be a major source of bugs
as it's pretty much a guarantee that a lot of people would do it unless the compiler
prevented them in some manner.

- Jonathan M Davis

On 3/7/2013 4:44 PM, Jonathan M Davis wrote:
> That may very well end up being a necessary restriction, but not being able to pass
> GC allocated objects across library boundaries would be a _huge_ downside to using
> shared libraries if that happens. It also would likely be a major source of bugs
> as it's pretty much a guarantee that a lot of people would do it unless the compiler
> prevented them in some manner.

You're probably right. See my other idea on this posted here.

"Walter Bright" <newshound2@digitalmars.com> wrote in message news:khban1$1lm2$1@digitalmars.com...
> On 1/2/2012 11:20 AM, Martin Nowak wrote:
>>   - Libraries might not be unloaded as long as GC collected class
>> instances
>> still exist because
>>     finalization fails otherwise.
>
> D doesn't guarantee that finalizers will run on GC allocated objects. Therefore, when unloading a dll:
>
> 1. run a gc collection
> 2. for all objects remaining on the heap
>        if they have a finalizer and that finalizer points into the dll
> code
>             mark them as not having a finalizer

What if their vtbl points into the dll code?

What about delegates or function pointers that point there?

On Friday, March 08, 2013 13:18:31 Daniel Murphy wrote:
> What if their vtbl points into the dll code?
> 
> What about delegates or function pointers that point there?

Assuming that that sort of thing could be detected, then an exception could be thrown when attempting to unload the dll.

- Jonathan M Davis