On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg wrote:
> On 2012-09-27 10:55, Maxim Fomin wrote:
>> On Thursday, 27 September 2012 at 08:26:08 UTC, Jacob Carlborg wrote:
>>> 1. Does this actually run?
>>>
>>
>> If it were non-runnable, I wouldn't posted it.
>>
>>> 2. This is statically linked with druntime and Phobos. What happens
>>> when you create an executable that links with the D dynamic library?
>>
>> Solution depends on a problem. I understood Andrei's post that he wanted
>> a .so file or DLL. I told originally that it is possible to make shared
>> libraries on linux. Now I see there is some misunderstanding. Is the
>> problem in diving D application on executables and shared libraries or
>> making druntime+phobos a shared library or loading a library at runtime?
>> A was speaking about the first.
>
> Actually, I seriously doubt everything is working as expected. For example, what happens when an application loads (via dlopen or similar) a D dynamic library:
>
> * Are exception handlers registered
> * Are module infos properly registered
> * What happens if I call Object.factory, will that find a class in the dynamic library
> * Are GC sections registered
> * What happens when the library is unloaded, will all of the above be unregistered and cleaned up
>
> A quick look in rt.minfo in druntime shows the it uses a single array or linked list for all module infos. It expect the module infos to be in just one place, i.e. in the application. There's no handling when a dynamic library is loaded.
>
> Have a look at this:
>
> https://github.com/D-Programming-Language/druntime/blob/master/src/rt/minfo.d#L184
>
> That symbol points to the start of the linked list containing module infos. That's only a single linked list, no handling of module infos from multiple places.
>
> rt.minfo needs to be changed to use an associative array with the keys pointing to images (executables and loaded dynamic libraries) and the values mapped to module infos of the given image. It also needs to intercept when a library is dynamically loaded, extract the module infos and register it with the runtime.
>
> I would think that the same is true for exception handling tables, TLS and GC sections.

Posted code doesn't load libraries at runtime, it is just linked to shared libraries.

On 9/27/12 3:37 PM, Maxim Fomin wrote:
> Posted code doesn't load libraries at runtime, it is just linked to
> shared libraries.

Exactly! (I can't believe I'm starting to get the hang of this...) But what we ultimately need is true dynamic loading of never-seen modules.

After the initial test I tried to load symbols with dlopen(). Indeed this works:

auto p = dlopen("liblib.so", RTLD_LAZY);

And it does work, all fine. To my dismay, as soon as I removed the flag "-llib" from the linker command line in the makefile, dlopen() didn't want to work anymore.

So I think in order to enable "true" dynamic loading, I'll need to generate PIC for druntime and phobos, and then link liblib.so like this:

dmd -fPIC -c lib.d
gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2

Is that correct?

At that point, of course, a variety of issues will need to be resolved as people pointed out.


Andrei

For me to get C or C++ to run a D function, I had to do the following:

// ====================
// C/C++ library source
// ====================

// sample D function from library
void foo(int i, int j, int k);
// D runtime initialization & shutdown
void init();
void done();

void bar()
{
    foo(6, 7, 8);
}

int main(int argc, char **argv)
{
init();
	bar();
	done();
	return 0;
}


// ================
// D library source
// ================
import std.stdio;
static import core.runtime;

// sample D function for test
extern (C++) //	int foo(int i, int j, int k)
	void foo(int i, int j, int k)
	{
		writefln("i = %s", i);
		writefln("j = %s", j);
		writefln("k = %s", k);
		int t = i + j + k;
		writefln("Total = %s", t);
	}

// Had to initialize and shutdown D system from C/C++.
extern (C++) export void init() { // to be called once after loading shared lib
    core.runtime.Runtime.initialize();
}

extern (C++) export void done() { // to be called before unloading shared lib
    core.runtime.Runtime.terminate();
}


// I had to include main even though this is a library.
int main()
{
   return 0;
}

On Thursday, 27 September 2012 at 07:54:29 UTC, Johannes Pfau wrote:
> Am Thu, 27 Sep 2012 08:26:36 +0200
> schrieb "Daniel Kozak" <kozzi11@gmail.com>:
>
>> Now I try it, and it is not required to build shared variant of druntime and phobos, only rebuild it with -fPIC
>> 
>
> In the end you'll probably need a shared druntime & phobos: Let's say
> your main app doesn't use std.datetime and you statically link
> libphobos. Then the linker might strip std.datetime from your
> executable. If your shared library now needs std.datetime it won't
> work.

I'm using C++ main app that dynamically loads shared libs at runtime, but I cannot load D shared libs because they cannot be linked to the non-fPIC built phobos2/druntime lib. It seems that the only way to get things working is to re-build the C++ code so that it is linked with non-fPIC phobos2, but this is not something I want to be doing.

So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number?

--rt

On Thursday, 27 September 2012 at 19:57:13 UTC, Andrei Alexandrescu wrote:
> So I think in order to enable "true" dynamic loading, I'll need to generate PIC for druntime and phobos, and then link liblib.so like this:
>
> dmd -fPIC -c lib.d
> gcc -shared lib.o -o liblib.so -L/path/to/phobos -lphobos2
>
> Is that correct?

Yes, that is correct.

The other thing missing is a phobos2.so for dynamic runtime linking.

-rt

On 2012-09-28 02:04, Rob T wrote:

> I'm using C++ main app that dynamically loads shared libs at runtime,
> but I cannot load D shared libs because they cannot be linked to the
> non-fPIC built phobos2/druntime lib. It seems that the only way to get
> things working is to re-build the C++ code so that it is linked with
> non-fPIC phobos2, but this is not something I want to be doing.
>
> So if I read this right, I can build phobos/druntime with -fPIC from the
> source code and it will work OK? If so, then why was this not done with
> the latest distrubution package, or is this only possible after a
> certain version number?

No, it will not work ok. Depending on what you do in the D code it might work, but everything will not work. Perhaps if you limit yourself to a C subset. See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg

On 2012-09-28 01:21, Rob T wrote:
> For me to get C or C++ to run a D function, I had to do the following:
>
> // ====================
> // C/C++ library source
> // ====================
>
> // sample D function from library
> void foo(int i, int j, int k);
> // D runtime initialization & shutdown
> void init();
> void done();
>
> void bar()
> {
>      foo(6, 7, 8);
> }
>
> int main(int argc, char **argv)
> {
> init();
>      bar();
>      done();
>      return 0;
> }
>
>
> // ================
> // D library source
> // ================
> import std.stdio;
> static import core.runtime;
>
> // sample D function for test
> extern (C++) //    int foo(int i, int j, int k)
>      void foo(int i, int j, int k)
>      {
>          writefln("i = %s", i);
>          writefln("j = %s", j);
>          writefln("k = %s", k);
>          int t = i + j + k;
>          writefln("Total = %s", t);
>      }
>
> // Had to initialize and shutdown D system from C/C++.
> extern (C++) export void init() { // to be called once after loading
> shared lib
>      core.runtime.Runtime.initialize();
> }
>
> extern (C++) export void done() { // to be called before unloading
> shared lib
>      core.runtime.Runtime.terminate();
> }
>
>
> // I had to include main even though this is a library.
> int main()
> {
>     return 0;
> }

That is not sufficient to have everything work. It might be ok if you limit yourself to a C subset of D. On some platforms, calling Runtime.initialize, will miss initializing some parts.

The implementation of the runtime initialization contains a lot of code duplication and some functions/branches are missing functionality. Example:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L346

The C main function handles the runtime initialization for a regular statically linked executable. If you initialize the runtime via "rt_init", which is called by "Runtime.initialize", it will miss to initialize some parts on some platforms. The C main function should really call "rt_init" to remove the code duplication.

Also see this post why everything will not properly work:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com


-- 
/Jacob Carlborg

Am Fri, 28 Sep 2012 02:04:11 +0200
schrieb "Rob T" <rob@ucora.com>:

> 
> So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number?
> 
> --rt

It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called.

GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings.

The fix for this issue is simple: Do not modify the EBX register, or at least save & restore it.

Old patch, which shows what needs to be done: https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Example asm code which probably doesn't work: https://github.com/D-Programming-Language/druntime/blob/master/src/core/cpuid.d#L432


I just tried building a shared druntime & phobos with gdc:
Druntime seems to be working, Phobos doesn't link here because
something's wrong with the fstat64 symbol. I haven't done any real
testing and we probably need some additional support in the runtime, but
the compiler part seems to be OK with gdc.
http://gdcproject.org/wiki/SharedRuntime
https://github.com/D-Programming-GDC/gdc/pull/32

On 28 September 2012 12:14, Johannes Pfau <nospam@example.com> wrote:
> Am Fri, 28 Sep 2012 02:04:11 +0200
> schrieb "Rob T" <rob@ucora.com>:
>
>>
>> So if I read this right, I can build phobos/druntime with -fPIC from the source code and it will work OK? If so, then why was this not done with the latest distrubution package, or is this only possible after a certain version number?
>>
>> --rt
>
> It shouldn't compile on x86 (32bit) with -fPIC because there's some incompatible asm code in druntime and phobos. If dmd doesn't warn in these cases, this code will fail as soon as it's called.
>
> GDC used to warn about these errors, but as we dropped the D inline assembler that code doesn't affect us anymore and there are no more warnings.
>
> The fix for this issue is simple: Do not modify the EBX register, or at least save & restore it.
>

In GDC D1 phobos, one hack around it was to use db to manually write out the bytecode instruction (thus bypassing GCC 'ebx' clobbered checks).  Which was a fiendish workaround to say the least. :~)


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that even with a successulf PIC build I will still not be able to reliably execute D functions directly from C/C++ code.

> Old patch, which shows what needs to be done:
> https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329

Does anyone know why patches like this are taking so [bleeping] long to get recognized and implemented?

-rt