On Monday, 20 June 2016 at 11:25:04 UTC, moe wrote:

>
> Where I still have a problem is with a plugin system. I would like to write an app that is plugin based. So that I can write a plugin to extend the functionality of the app. I imagine that I could copy the plugin into a folder from the (compiled) app and the app would be able to use the plugin without me having to recompile the app. I have been looking at LoadLibrary (https://dlang.org/library/core/runtime/runtime.load_library.html) and Object.factory (https://dlang.org/library/object/object.factory.html) but I don't quite get how to use them. I did find some info that was several years old. They mentioned that these functions are not yet ready for all platforms. I don't know if that has changed in the meantime. It is not clear from the documentation.
>
> I would like to find some snippets or examples but that sort of resource for D is very hard to find online. Most of the time there are only some partial snippets where it turns out they only work under specific conditions or are restricted to one platform. Does D not supply some infrastructure for this kind of a task? It seams to me that is a very common approach to build an app. I expected there would be some out of the box solution for this in D. I would write the app and the plugins purely in D (no C bindings if possible).
>

The problem is that shared library (DLL) support in D is not at a level where you can load one at runtime and treat it as the rest of your D code (at least on Windows -- I'm not sure how complete support is elsewhere). Until it is, there are a number of steps you can take to work around potential problems.

> I have tried to build a mini app which defines an interface for the plugins. The plugins then implement that interface. But when I want to load the plugin both LoadLibrary() and Object.factory() fail (I am aware that Object.factory() requires the full qualified class names).
>
> Is there a recommended way of writing a simple plugin system that works on windows and linux? Maybe someone has a working example or knows a book that covers how to write a plugin system?

You've got the right idea in defining an interface, but don't bother with Object.factory. Here's a brief rundown of how I wouldn handle it.

Let's assume we'e got an interface like so:

enum PluginError {
   none,
   initFileSys,
   initAudio,
   initSomethingElse
}

class PluginException : Exception { ... }

interface Plugin {
   bool initialize();
   void terminate();
   Throwable getLastException();
   SomeObject getSomeObject();
   void returnSomeObject(SomeObject);
}

Now, I would take the following steps:

* Any free functions in the D code for the DLL would be marked as extern(C). This does not make them "C functions" in that you can't use D with them. It only turns off D symbol decoration so that they look like C functions to the linker and ensures that they have the cdecl calling convention. This makes them much easier to load, as the symbol names will match the function names.

* Any objects that need to be used in the application from the plugin should be allocated in the DLL. Be careful with this. You have to initialize the runtime yourself in your DLL, so you have two different GC instances running: one in the app and one in the DLL. I believe on Linux its possible to use Phobos + DRuntime as a shared library, so you have only one GC, but on Windows it is not. This means you should keep a reference to any objects you allocate in the DLL to make sure they aren't collected while the app is using them. When the app is finished with an instance, it should let the DLL know.

* Don't allocate Exceptions on the DLL side for use on the app side. You can't throw them across boundaries anyway, but you also have the issue of two separate GC instances. If you return an Exception somehow (like, plugin.getLastError or something) it will probably never be collected or, if the DLL terminates before the exception propagates fully, the memory will be invalid. Hence my use of PluginError above.

One possible use of the interface:

=================
// In the DLL code
export extern(C) nothrow Plugin getPluginImpl() {
    import core.memory : GC;
    auto plugin = new PluginImpl();

    // Make sure the GC doesn't collect the instance
    GC.addRoot(cast(void*)plugin);
    return plugin;
}

// In the app code
// You'll use this to load the plugin
alias GetPluginImpl = extern(c) nothrow function();
GetPluginImpl getPluginImpl;

// Using DerelictUtil [1] makes it easy.
// Use derelict-util version ~>2.0.6 as a dub dependency
import derelict.util.sharedlib;

// Keep this around if you need to unload the library manually
// at some point. It does not unload in the destructor, so you
// can let it go out of scope with no problems if you intend to
// keep the library open for the life of the program.
SharedLib lib;
lib.load(["plugin.dll"]);
getPluginImpl = cast(GetPluginImpl)lib.loadSymbol("getPluginImpl");

auto so = plugin.getSomeObject();

// Now get the plugin interface
auto plugin = getPluginImpl();

// And do some work
auto res = plugin.initialize();
if(res != PluginError.none) throw new PluginException(res);
===================================

Anyway, given my current level of knowledge about shared library support in D, this is the approach I would take. I may very well be behind the times, though, so if you discover this isn't all necessary, then great. But I'm pretty sure Linux has the most complete support of any of the platforms right now, while Windows and Mac are rather behind.

[1] https://github.com/DerelictOrg/DerelictUtil/blob/master/source/derelict/util/sharedlib.d

> interface Plugin {
>    bool initialize();
>    void terminate();
>    Throwable getLastException();
>    SomeObject getSomeObject();
>    void returnSomeObject(SomeObject);
> }

Sorry, I forgot a couple of commments. I did explain it in the text, though. It was supposed to read:

interface Plugin {
   bool initialize();
   void terminate();

   // Dont' do this! Allocate exceptions on app side.
   Throwable getLastException();

   // Do do this. Allocate plugin objects on DLL side.
   SomeObject getSomeObject();
   void returnSomeObject(SomeObject);
}

On Monday, 20 June 2016 at 13:51:15 UTC, Mike Parker wrote:
>> interface Plugin {
>>    bool initialize();
>>    void terminate();
>>    Throwable getLastException();
>>    SomeObject getSomeObject();
>>    void returnSomeObject(SomeObject);
>> }
>
> Sorry, I forgot a couple of commments. I did explain it in the text, though. It was supposed to read:
>
> interface Plugin {
>    bool initialize();
>    void terminate();
>
>    // Dont' do this! Allocate exceptions on app side.
>    Throwable getLastException();
>
>    // Do do this. Allocate plugin objects on DLL side.
>    SomeObject getSomeObject();
>    void returnSomeObject(SomeObject);
> }

Wow, absolutely awesome! That's exactly what I have been looking for. I should have some time to try it out later this week. Many thanks for all the info. I very much appreciate that you took the time to explain everything. It helped me a lot!

Also thanks to everybody else for the input!

I had some time to try it out and I finally got it to work. I have only tried in windows so far but there was a pitfall in windows. Your dll need a DllMain entry to compile. This was the only thing that was missing from your information. The rest worked perfectly. This may be obvious to most around here, but I did not know before. So, I thought it might make sense to show my working solution in case someone else stumbles upon the same problem.

I wanted the app and the plugin to be independent projects form one another. So they both need a shared project containing the interface for the plugin. That makes the 3 projects shown below. I can than after compiling the app simply copy the plugin dll into a plugins folder and the app will find it on demand. So that I could later add more plugins if desired. Obviously in that case I would have to get a list of available plugins by reading the filenames in the plugins directory instead of a hard coded path in the app. I tried to reduce it to the bare minimum with a working solution. Even with the DllMain entry (which is necessary for windows) it turns out to be surprisingly compact.

There are 3 distinct projects. I have the following directories (omitting some generated by dub):

- PluginTest

-- PluginContract // all files for the shared plugin interface (separate project)
---- source
------ iplugin.d // the interface for the plugin
---- dub.json // the project file for the shared plugin interface

-- TestApp // all files for the app (separate project)
---- packages
------ DerelictUtil-master // contains the project for derelict
---- source
------ app.d // the app
---- dub.json // the project file for the app

-- TestPlugin // all files for the plugin (separate project)
---- source
------ someplugin.d // the plugin
---- dub.json // the project file for the plugin


Here are the files:

Shared plugin interface (Project 1)
===================================
Note, these are necessary for the linker to find the files:
"targetType": "library"
"targetPath": "lib"

PluginTest/PluginContract/dub.json
----------------------------------
{
	"name": "plugincontract",
	"authors": ["root"],
	"description": "A minimal D application.",
	"copyright": "Copyright © 2016, root",
	"license": "proprietary",
	"platforms": ["windows"],
	"versions": ["DesktopApp"],
	"targetType": "library",
	"configurations": [
	{
		"name": "debug",
		"targetPath": "lib",
		"buildOptions": ["debugMode", "debugInfo"],
	},
	{
		"name": "release",
		"targetPath": "lib",
		"buildOptions": ["releaseMode", "optimize", "inline"],
	}
	]
}


PluginTest/PluginContract/source/iplugin.d
------------------------------------------
module iplugin;

interface IPlugin
{
	void Talk(string msg);
}





TestApp (Project 2)
===================

PluginTest/TestApp/dub.json
---------------------------
{
	"name": "testapp",
	"authors": ["root"],
	"description": "A minimal D application.",
	"copyright": "Copyright © 2016, root",
	"license": "proprietary",
	"platforms": ["windows"],
	"versions": ["DesktopApp"],
	"targetType": "executable",
	"dependencies": {
		"derelict-util":  {"path": "packages/DerelictUtil-master"},
		"plugincontract": {"path": "../PluginContract"}
	},
	"configurations": [
	{
		"name": "debug",
		"targetPath": "bin/debug",
		"buildOptions": ["debugMode", "debugInfo"],
	},
	{
		"name": "release",
		"targetPath": "bin/release",
		"buildOptions": ["releaseMode", "optimize", "inline"],
	}
	]
}


PluginTest/TestApp/source/app.d
-------------------------------
import std.stdio;
import derelict.util.sharedlib;
import iplugin;

alias GetPluginImpl = extern(C) nothrow IPlugin function();
GetPluginImpl getPlugin;

void main()
{
	SharedLib lib;
	lib.load(["plugins/testplugin.dll"]);
	getPlugin = cast(GetPluginImpl)lib.loadSymbol("getPlugin");

	auto plugin = getPlugin();
	plugin.Talk("Hello World.");

	writeln("End of app.");
}




TestPlugin (Project 3)
======================

PluginTest/TestPlugin/dub.json
------------------------------
{
	"name": "testplugin",
	"authors": ["root"],
	"description": "A minimal D application.",
	"copyright": "Copyright © 2016, root",
	"license": "proprietary",
	"platforms": ["windows"],
	"versions": ["DesktopApp"],
	"targetType": "dynamicLibrary",
	"importPaths": ["../PluginContract"],
	"dependencies": {
		"plugincontract": {"path": "../PluginContract"}
	},
	"configurations": [
	{
		"name": "debug",
		"targetPath": "bin/debug",
		"buildOptions": ["debugMode", "debugInfo"],
	},
	{
		"name": "release",
		"targetPath": "bin/release",
		"buildOptions": ["releaseMode", "optimize", "inline"],
	}
	]
}

PluginTest/TestPlugin/source/SomePlugin.d
-----------------------------------------
module someplugin;
import std.stdio;
import iplugin;


export extern(C) nothrow IPlugin getPlugin()
{
	import core.memory : GC;
	auto plugin = new SomePlugin();

	// Make sure the GC doesn't collect the instance
	GC.addRoot(cast(void*)plugin);
	return plugin;
}


class SomePlugin : IPlugin
{
	//	this() {}
	void Talk(string msg)
	{
		writefln("SomePlugin: %s", msg);
	}
}

//shared static this() { printf("plugin shared static this\n"); }
//shared static ~this() { printf("plugin shared static ~this\n"); }


version(Windows)
extern(Windows) bool DllMain(void* hInstance, uint ulReason, void*)
{
	import std.c.windows.windows;
	import core.sys.windows.dll;
	switch (ulReason)
	{
		default: assert(0);
		case DLL_PROCESS_ATTACH:
				 dll_process_attach( hInstance, true );
				 break;

		case DLL_PROCESS_DETACH:
				 dll_process_detach( hInstance, true );
				 break;

		case DLL_THREAD_ATTACH:
				 dll_thread_attach( true, true );
				 break;

		case DLL_THREAD_DETACH:
				 dll_thread_detach( true, true );
				 break;
	}
	return true;
}


Hope it helps someone. And again thanks for the help!

On Tuesday, 21 June 2016 at 23:59:54 UTC, moe wrote:
> I had some time to try it out and I finally got it to work. I have only tried in windows so far but there was a pitfall in windows. Your dll need a DllMain entry to compile. This was the only thing that was missing from your information.

Right, I forgot about that. It's always optional in C and C++, but in D we need to use it to initialize the runtime.


> -- TestApp // all files for the app (separate project)
> ---- packages
> ------ DerelictUtil-master // contains the project for derelict
> ---- source
> ------ app.d // the app
> ---- dub.json // the project file for the app

This is not the way DerelictUtil, or any DUB package, is intended to be used when you use DUB to manage your project. You don't need to download it this way. You chould change your dependency in TestApp:


> 	"dependencies": {
> 		"derelict-util":  "version=~>2.0.6",

2.0.6 is the latest. The '~>' means >=2.0.6 && <2.1.0, so that if there is a 2.0.7 released, you can run 'dub upgrade' on your project and start using it. The same for 2.0.8, 2.0.9, 2.0.n...

Of course, if you have a reason for doing it that way, more power to you :)

On Wednesday, 22 June 2016 at 01:40:47 UTC, Mike Parker wrote:
> On Tuesday, 21 June 2016 at 23:59:54 UTC, moe wrote:
>> I had some time to try it out and I finally got it to work. I have only tried in windows so far but there was a pitfall in windows. Your dll need a DllMain entry to compile. This was the only thing that was missing from your information.
>
> Right, I forgot about that. It's always optional in C and C++, but in D we need to use it to initialize the runtime.
>
>
>> -- TestApp // all files for the app (separate project)
>> ---- packages
>> ------ DerelictUtil-master // contains the project for derelict
>> ---- source
>> ------ app.d // the app
>> ---- dub.json // the project file for the app
>
> This is not the way DerelictUtil, or any DUB package, is intended to be used when you use DUB to manage your project. You don't need to download it this way. You chould change your dependency in TestApp:
>
>
>> 	"dependencies": {
>> 		"derelict-util":  "version=~>2.0.6",
>
> 2.0.6 is the latest. The '~>' means >=2.0.6 && <2.1.0, so that if there is a 2.0.7 released, you can run 'dub upgrade' on your project and start using it. The same for 2.0.8, 2.0.9, 2.0.n...
>
> Of course, if you have a reason for doing it that way, more power to you :)

Yes, I did it intentionally. I wanted to ensure that the packages are self contained and check whether it would work fine like this. Basically I like to have a project that contains everything it needs with the versions originally used to build. It sort of creates a reference for learning.

For testing I would have also liked not to build a dub project for the PluginContract but rather have a simple iplugins.d file in a folder. I could not get it to be imported if it's outside of a dub project.

"importPaths": ["../PluginContract"]

Did not work for some reason. But that's a minor issue.

On Wednesday, 22 June 2016 at 02:38:23 UTC, moe wrote:

>
> Yes, I did it intentionally. I wanted to ensure that the packages are self contained and check whether it would work fine like this. Basically I like to have a project that contains everything it needs with the versions originally used to build. It sort of creates a reference for learning.

Understood. FYI, you can specify a specific version and dub upgrade won't touch it.


>
> For testing I would have also liked not to build a dub project for the PluginContract but rather have a simple iplugins.d file in a folder. I could not get it to be imported if it's outside of a dub project.
>
> "importPaths": ["../PluginContract"]
>
> Did not work for some reason. But that's a minor issue.

Becuase you specified an incomplete path:

../PluginContract/source

The compiler needs the root directory of the root package for any imporyed module.

On Wednesday, 22 June 2016 at 02:54:06 UTC, Mike Parker wrote:
> On Wednesday, 22 June 2016 at 02:38:23 UTC, moe wrote:
>
>>
>> Yes, I did it intentionally. I wanted to ensure that the packages are self contained and check whether it would work fine like this. Basically I like to have a project that contains everything it needs with the versions originally used to build. It sort of creates a reference for learning.
>
> Understood. FYI, you can specify a specific version and dub upgrade won't touch it.
>
>
>>
>> For testing I would have also liked not to build a dub project for the PluginContract but rather have a simple iplugins.d file in a folder. I could not get it to be imported if it's outside of a dub project.
>>
>> "importPaths": ["../PluginContract"]
>>
>> Did not work for some reason. But that's a minor issue.
>
> Becuase you specified an incomplete path:
>
> ../PluginContract/source
>
> The compiler needs the root directory of the root package for any imporyed module.


I meant like this:

- PluginContract // not a dub project, just some folder
-- iplugin.d

- TestApp // all files for the app (separate project)
-- packages
---- DerelictUtil-master // contains the project for derelict
-- source
---- app.d // the app
-- dub.json // the project file for the app

The only dub project would be TestApp. PluginContract would just be some folder completely outside the TestApp dub project. I could not get a relative path to work like this.

On Wednesday, 22 June 2016 at 03:06:29 UTC, moe wrote:

> I meant like this:
>
> - PluginContract // not a dub project, just some folder
> -- iplugin.d
>
> - TestApp // all files for the app (separate project)
> -- packages
> ---- DerelictUtil-master // contains the project for derelict
> -- source
> ---- app.d // the app
> -- dub.json // the project file for the app
>
> The only dub project would be TestApp. PluginContract would just be some folder completely outside the TestApp dub project. I could not get a relative path to work like this.

Just to be clear, are you compiling iplugin.d as well? I assumed you were referring to a compiler error (i.e. missing import), but based on this post I would guess you're getting a linker error. You should probably add this to your dub.json in addition to the importPaths:

"sourceFiles": ["../PluginContract/iplugin.d"]

On Wednesday, 22 June 2016 at 05:34:33 UTC, Mike Parker wrote:
> On Wednesday, 22 June 2016 at 03:06:29 UTC, moe wrote:
>
>> I meant like this:
>>
>> - PluginContract // not a dub project, just some folder
>> -- iplugin.d
>>
>> - TestApp // all files for the app (separate project)
>> -- packages
>> ---- DerelictUtil-master // contains the project for derelict
>> -- source
>> ---- app.d // the app
>> -- dub.json // the project file for the app
>>
>> The only dub project would be TestApp. PluginContract would just be some folder completely outside the TestApp dub project. I could not get a relative path to work like this.
>
> Just to be clear, are you compiling iplugin.d as well? I assumed you were referring to a compiler error (i.e. missing import), but based on this post I would guess you're getting a linker error. You should probably add this to your dub.json in addition to the importPaths:
>
> "sourceFiles": ["../PluginContract/iplugin.d"]



I have added all of these to the dub.json:

"sourcePaths": ["../PluginContract"],
"importPaths": ["../PluginContract"],
"sourceFiles": ["../PluginContract/iplugin.d"],

In my app I use:
import iplugin;

I would expect that both the compiler and the linker finds the needed files. I would also prefer a path to link a folder rather than adding files individually. It seams more error prone when I have to remember to add every file in a bigger project. However, every combination of the above seams to fail. With a linker error.

Linking...
OPTLINK (R) for Win32  Release 8.00.17
Copyright (C) Digital Mars 1989-2013  All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
OPTLINK : Warning 23: No Stack
.dub\build\debug-debug-windows-x86-dmd_2071-0BEC1C92408DC77EE5C50BCF4B1225A9\tes
tapp.obj(testapp)
 Error 42: Symbol Undefined _D18TypeInfo_Interface6__vtblZ
.dub\build\debug-debug-windows-x86-dmd_2071-0BEC1C92408DC77EE5C50BCF4B1225A9\tes
tapp.obj(testapp)
 Error 42: Symbol Undefined _D14TypeInfo_Class6__vtblZ
OPTLINK : Warning 134: No Start Address
--- errorlevel 2
dmd failed with exit code 2.


Without the adjustments in the dub.json I get the following error (But that is expected if dub only searches in the source folder by default):

testplugin ~master: building configuration "debug"...
source\SomePlugin.d(3,8): Error: module iplugin is in file 'iplugin.d' which can
not be read
import path[0] = source
import path[1] = C:\D\dmd2\windows\bin\..\..\src\phobos
import path[2] = C:\D\dmd2\windows\bin\..\..\src\druntime\import
dmd failed with exit code 1.