One of the primary uses of dynamic loading of libraries might well be to provide plugins. By plugins I mean extensions to an existing program that can be added to the program at run-time, and can be written by separate authors who don't necessarily have access to the source code of the program, but who do understand the rules provided by the program documentation as to what capabilities the plugin must have.

The most obvious way to allow a D program to cope with plugins, is to specify an interface to which the plugin is expected to conform. Communication between the program and the plugin, after the latter is loaded would then be restricted to calls provided by that interface.

One further library method would likely be necessary to allow for the acquisition of an instance of the plugin, and its attachment to the program.

So, we could have

module ifd;
interface I
{
   string saySomething();
}

The plugin could then be:

module plugin;
import ifd;
import std.stdio;

class Plugin: I
{
   this() { writeln("plugin ctor"); }

   string saySomething() { return "I am plugin"; }
}

I getInstance()
{
   return new Plugin();
}

And our program could be:

module main;
import core.runtime;
import std.stdio;
import ifd;

extern(C) void* dlsym(void*, const char*);
extern(C) void* dlopen(const char*, int);

alias I function() pfi;

I getPlugin(string name)
{
   // Take your pick from these two - makes no odds
   //void* lib = dlopen("plugin.so\0".ptr, 1);
   void* lib = Runtime.loadLibrary(name~".so");

   if (lib is null)
   {
      writeln("failed to load plugin shared object");
      return null;
   }

   void* vp = dlsym(lib, "_D6plugin11getInstanceFZC3ifd1I\0".ptr);
   if (vp is null)
   {
      writeln("plugin creator function not found");
      return null;
   }
   pfi f = cast(pfi) vp;
   I x = f();
	if (x is null)
	{
		writeln("creation of plugin failed");
		return null;
	}
	return x;
}

void main()
{
   I x = getPlugin("plugin");
   writeln(x.saySomething());
}

Unfortunately, the result of running the program is:

steve@steve-desktop:~/scratch/piif$ ./main
plugin ctor

Segmentation fault (core dumped)

Which suggests that the library was loaded, the symbol found, and an instance of plugin created.

The two pieces were built using dmd2.064 with:

main : ifd.d main.d
	dmd -c ifd.d
	dmd -c main.d
	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

plugin : plugin.d
	dmd -c -shared -fPIC plugin.d
	dmd plugin.o -shared -defaultlib=libphobos2.so -map

clean :
	rm *.o
	rm main
	rm plugin.so

Does anyone have any suggestions as to what might be going wrong here? I have further examples, but I guess I should do them one at a time.

Steve

Hi Steve. Well, I had so many experience with this and after many tries, I gave up unfortunately. It continuous giving Segmentation error.

What I have found is that while the main programme is closing, it tries to call "rt_finalize" to remove that object from memory, then it says that method is not defined. I complied programme in Debug version and used GDB to understand above situation.

(Linux x64, DMD 2.065)

On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
> One of the primary uses of dynamic loading of libraries might well be to provide plugins. By plugins I mean extensions to an existing program that can be added to the program at run-time, and can be written by separate authors who don't necessarily have access to the source code of the program, but who do understand the rules provided by the program documentation as to what capabilities the plugin must have.
>
> The most obvious way to allow a D program to cope with plugins, is to specify an interface to which the plugin is expected to conform. Communication between the program and the plugin, after the latter is loaded would then be restricted to calls provided by that interface.
>
> One further library method would likely be necessary to allow for the acquisition of an instance of the plugin, and its attachment to the program.
>
> So, we could have
>
> module ifd;
> interface I
> {
>    string saySomething();
> }
>
> The plugin could then be:
>
> module plugin;
> import ifd;
> import std.stdio;
>
> class Plugin: I
> {
>    this() { writeln("plugin ctor"); }
>
>    string saySomething() { return "I am plugin"; }
> }
>
> I getInstance()
> {
>    return new Plugin();
> }
>
> And our program could be:
>
> module main;
> import core.runtime;
> import std.stdio;
> import ifd;
>
> extern(C) void* dlsym(void*, const char*);
> extern(C) void* dlopen(const char*, int);
>
> alias I function() pfi;
>
> I getPlugin(string name)
> {
>    // Take your pick from these two - makes no odds
>    //void* lib = dlopen("plugin.so\0".ptr, 1);
>    void* lib = Runtime.loadLibrary(name~".so");
>
>    if (lib is null)
>    {
>       writeln("failed to load plugin shared object");
>       return null;
>    }
>
>    void* vp = dlsym(lib, "_D6plugin11getInstanceFZC3ifd1I\0".ptr);
>    if (vp is null)
>    {
>       writeln("plugin creator function not found");
>       return null;
>    }
>    pfi f = cast(pfi) vp;
>    I x = f();
> 	if (x is null)
> 	{
> 		writeln("creation of plugin failed");
> 		return null;
> 	}
> 	return x;
> }
>
> void main()
> {
>    I x = getPlugin("plugin");
>    writeln(x.saySomething());
> }
>
> Unfortunately, the result of running the program is:
>
> steve@steve-desktop:~/scratch/piif$ ./main
> plugin ctor
>
> Segmentation fault (core dumped)
>
> Which suggests that the library was loaded, the symbol found, and an instance of plugin created.
>
> The two pieces were built using dmd2.064 with:
>
> main : ifd.d main.d
> 	dmd -c ifd.d
> 	dmd -c main.d
> 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
>
> plugin : plugin.d
> 	dmd -c -shared -fPIC plugin.d
> 	dmd plugin.o -shared -defaultlib=libphobos2.so -map
>
> clean :
> 	rm *.o
> 	rm main
> 	rm plugin.so
>
> Does anyone have any suggestions as to what might be going wrong here? I have further examples, but I guess I should do them one at a time.
>
> Steve

>> 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.

Hmm. I saw that you are using libphobos2.so. I wasn't using this.

On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
> One of the primary uses of dynamic loading of libraries might well be to provide plugins. By plugins I mean extensions to an existing program that can be added to the program at run-time, and can be written by separate authors who don't necessarily have access to the source code of the program, but who do understand the rules provided by the program documentation as to what capabilities the plugin must have.

It might be purely a matter of historic interest at this point,
but DDL was basically doing plugins in D ages ago:

http://www.dsource.org/projects/ddl

I /think/ this was the dynamic loading scheme h3r3tic used in his
game engine back in the day.  I'm sure someone with a better
memory than me could explain it better.

On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga Cakiroglu wrote:
>>> 	dmd main.o ifd.o -L-ldl -defaultlib=libphobos2.so -L-rpath=.
>
> Hmm. I saw that you are using libphobos2.so. I wasn't using this.

Yes, you have to make the program and the plugin both use the same runtime, presumably so they are using the same memory allocation system, so you link both with libphobos2.so.

It is not just a shut-down artefact, since if I repeat the exercise with an exemplar class instead of an interface, then it will work.

But then if I make the exemplar class an instance of an interface, and call one of the methods that is part of the interface it goes back to crashing, so it does seem that this behaviour is linked to interfaces.

I will get stuck in with GDB and see if I can find out more, but that's not an agreeable experience, so I thought I would ask first.

Steve

On Thursday, 13 March 2014 at 23:44:23 UTC, Sean Kelly wrote:
> It might be purely a matter of historic interest at this point,
> but DDL was basically doing plugins in D ages ago:
>
> http://www.dsource.org/projects/ddl
>
> I /think/ this was the dynamic loading scheme h3r3tic used in his
> game engine back in the day.  I'm sure someone with a better
> memory than me could explain it better.

Sean,

Yes, been there years ago - maybe 2006. I could never get the full edifice working, so I extracted the parts that would enable me to just load a single .o file, and that would work most of the time, but then sometimes it would just crash.

Maybe that would work with today's runtime, but the machine that particular code was on got stolen, so I'd have to start from scratch.

Steve

On Thursday, 13 March 2014 at 13:22:55 UTC, Steve Teale wrote:
> Does anyone have any suggestions as to what might be going wrong here? I have further examples, but I guess I should do them one at a time.
>
> Steve

Copied your files, added: -L-rpath=/home/geod/bin/dmd-master/linux/lib64/
as dmd is not on the system path for me.

It was working. So try to update your compiler and check if that's the reason.

On Friday, 14 March 2014 at 06:54:34 UTC, Steve Teale wrote:
> On Thursday, 13 March 2014 at 21:20:10 UTC, Tolga Cakiroglu

> It is not just a shut-down artefact, since if I repeat the exercise with an exemplar class instead of an interface, then it will work.


OK, so here's an example using an exemplar base class instead of an interface. This includes making the exemplar class implement an interface.

Here's the exemplar class, and another class that references an instance of it:

module bc;
import std.stdio;

interface I
{
   string saySomething();
}

class Bc: I
{
   string saySomething() { return null; };
}

class Other
{
   I target;

   this(I i) { target = i; }

   void invokeWithCast() { writeln((cast(Bc) target).saySomething()); }
   void invoke() { writeln(target.saySomething()); }
}

Then the plugin:

module plugin;
import bc;
import std.stdio;

class Plugin: Bc
{
   this() { writeln("plugin ctor"); }

   override string saySomething() { return "I am plugin"; }
}

Bc getInstance()
{
   return new Plugin();
}

And the program:

module main;
import core.runtime;
import std.stdio;
import bc;

extern(C) void* dlsym(void*, const char*);

alias Bc function() pfi;

Bc getPlugin(string name)
{
   void* lib = Runtime.loadLibrary(name~".so");

   void* vp = dlsym(lib, "_D6plugin11getInstanceFZC2bc2Bc\0".ptr);
   pfi f = cast(pfi) vp;
   Bc x = f();
	return x;
}

void main()
{
   Bc x = getPlugin("plugin");
   writeln(x.saySomething());    // OK
   Other other = new Other(x);
   other.invokeWithCast();       // OK
   other.invoke();               // Crash
}

Built with:

main : bc.d main.d
	dmd -c bc.d
	dmd -c main.d
	dmd main.o bc.o -L-L/usr/local/lib -L-ldl -L-lgtkd-2 -defaultlib=libphobos2.so -L-rpath=.

plugin : plugin.d
	dmd -c -shared -fPIC plugin.d
	dmd plugin.o -shared -defaultlib=libphobos2.so -map

clean :
	rm *.o
	rm main
	rm plugin.so

Output:

steve@steve-desktop:~/scratch/pibc$ ./main
plugin ctor
I am plugin
I am plugin
Segmentation fault (core dumped)

If you comment out the last call via the interface instance with no cast, the program exits cleanly. So the problem does seem to relate to interfaces.

So you can do it, but if the base class conforms to some interface, then you have to special-case calls to the interface functions. That rather destroys the utility of classes like Other which provide services to the installed plugin.

My gut-feeling question is "is the vtable in the program or the plugin, and does this vary when the plugin implements an interface?"

I'm hoping Martin Nowak might join in this discussion.

Steve

On Friday, 14 March 2014 at 07:46:25 UTC, Steve Teale wrote:
> My gut-feeling question is "is the vtable in the program or the plugin, and does this vary when the plugin implements an interface?"
>
> I'm hoping Martin Nowak might join in this discussion.
>
> Steve

geod@Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make plugin
/home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c -shared -fPIC plugin.d
/home/geod/bin/dmd-2.064.2/linux/bin64/dmd plugin.o -shared -defaultlib=libphobos2.so -map
geod@Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ make
/home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c bc.d
/home/geod/bin/dmd-2.064.2/linux/bin64/dmd -c main.d
/home/geod/bin/dmd-2.064.2/linux/bin64/dmd main.o bc.o -L-L/usr/local/lib -L-ldl -defaultlib=libphobos2.so -L-rpath=. -L-rpath=/home/geod/bin/dmd-2.064.2/linux/lib64/
geod@Barsoom:~/Work/BoapNet/Client/Native/SharedTemplate/test$ ./main
plugin ctor
I am plugin
I am plugin
I am plugin


Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?

On Friday, 14 March 2014 at 08:15:00 UTC, Mathias LANG wrote:
> ./main
> plugin ctor
> I am plugin
> I am plugin
> I am plugin
>
>
> Same result with master branch. Long story short: I can't reproduce it on my machine (Debian x86_64). Which platform are you working on ? 32 bits ?

Ubuntu 12.04 32 bit.

I'll reboot into 64 bit and try there.