View mode: basic / threaded / horizontal-split · Log in · Help
September 28, 2012
Re: dynamic library building and loading
On 28 September 2012 17:43, Rob T <rob@ucora.com> wrote:
> I am trying out gdc 4.7 branch with -fPIC, but the info I'm getting is that
> even with a successulf PIC build I will still not be able to reliably
> execute D functions directly from C/C++ code.
>
>
>> Old patch, which shows what needs to be done:
>>
>> https://bitbucket.org/goshawk/gdc/issue/166/add-shared-lib-support#comment-648329
>
>
> Does anyone know why patches like this are taking so [bleeping] long to get
> recognized and implemented?
>
> -rt
>

Because the [bleeping] runtime does not support running from a shared library.

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
September 28, 2012
Re: dynamic library building and loading
On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:
>
> Because the [bleeping] runtime does not support running from a 
> shared library.

I suppose the answer is very complicated, but why can't the 
runtime execute as a shared lib? Is it a design limitation of the 
runtime model? This sort of problem should have been dealt with 
from the ground up, not now. What can be done to fix it, or is it 
a major redesign + re-write effort?

-rt
September 28, 2012
Re: dynamic library building and loading
On Thursday, 27 September 2012 at 17:10:07 UTC, Jacob Carlborg 
wrote:
> Actually, I seriously doubt everything is working as expected.

I tried to check how TLS, EX, etc. (mostly exposed to dll issue) 
are working and here is some kind of test: 
https://github.com/mxfm/sharedtest. Unfortunately scope(exit) 
isn't executed when it is situated in a shared library and which 
calls some throwing function from another shared library. 
Unittests aren't working either. Regarding other parts - they 
seem to work.
September 28, 2012
Re: dynamic library building and loading
On 28 September 2012 18:54, Rob T <rob@ucora.com> wrote:
> On Friday, 28 September 2012 at 17:07:16 UTC, Iain Buclaw wrote:
>>
>>
>> Because the [bleeping] runtime does not support running from a shared
>> library.
>
>
> I suppose the answer is very complicated, but why can't the runtime execute
> as a shared lib? Is it a design limitation of the runtime model? This sort
> of problem should have been dealt with from the ground up, not now. What can
> be done to fix it, or is it a major redesign + re-write effort?
>
> -rt

The big implementation issue is that the runtime only passes two data
ranges to the GC to scan. Global data segment and TLS data segment of
the resultant binary/executable.  So any static data stored in shared
libraries aren't scanned / recognised as data we want to keep. So the
use of them may cause sudden unwanted collections.

The way I intend to address it is to have each module handle it's own
gshared/thread data and pass the information to the D runtime during
the module construction stage (.ctor) - there is already something
done this way for _Dmodule_ref - so it may end up being that two new
fields will be tacked onto it; void[] tlsdata, void[] gsharedata;


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
September 29, 2012
Re: dynamic library building and loading
On 2012-09-28 20:25, Maxim Fomin wrote:

> I tried to check how TLS, EX, etc. (mostly exposed to dll issue) are
> working and here is some kind of test:
> https://github.com/mxfm/sharedtest. Unfortunately scope(exit) isn't
> executed when it is situated in a shared library and which calls some
> throwing function from another shared library. Unittests aren't working
> either. Regarding other parts - they seem to work.

That's a fairly uninteresting test. You are linking to the dynamic 
library. What's interesting is loading a dynamic library using dlopen, 
or similar. What's the point of using dynamic libraries if you're 
linking with them?

-- 
/Jacob Carlborg
September 29, 2012
Re: dynamic library building and loading
On 2012-09-28 19:54, Rob T wrote:

> I suppose the answer is very complicated, but why can't the runtime
> execute as a shared lib? Is it a design limitation of the runtime model?
> This sort of problem should have been dealt with from the ground up, not
> now. What can be done to fix it, or is it a major redesign + re-write
> effort?

See one of my other posts:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg
September 29, 2012
Re: dynamic library building and loading
On Saturday, 29 September 2012 at 13:19:01 UTC, Jacob Carlborg 
wrote:
> That's a fairly uninteresting test.

I am not a D developer which means I have no incentive in blindly 
portraying D as a language having shared libraries support when 
in fact it has some issues. I am a D user which has incentive to 
gladly report which part of D does work (or seems to work) and 
which doesn't when making shared libraries.

> You are linking to the dynamic library. What's interesting is 
> loading a dynamic library using dlopen, or similar. What's the 
> point of using dynamic libraries if you're linking with them?

I was not taking about dynamic loading, but about dynamic 
linking. If I understand topic right, the issue is the former, 
not the latter.

BTW, in majority cases dynamic loading gives no advantages over 
dynamic linking (just the opposite: doing unnecessary job which 
can be done by linker and loader). In most cases, when an app is 
being written, it is known which functional from which 
third-party libraries would be used. The only exceptions I can 
name are pluggin support and hacker's binary ELF/PE tools. I can 
count few apps in windows and linux which are used often, which 
dynamically load something, and if they do, they likely load 
pluggins. This was made for cases when app developers by 
definition cannot know beforehand full list of used shared 
libraries. So, the question is just opposite: "What's the point 
of using dynamic loading if you know beforehand which libraries 
you use, which happens almost in all cases?".

But I certainly agree that dynamic loading is essential for some 
applications, if not irreplaceable, and support of it also needed 
to be inspected and improved.
September 29, 2012
Re: dynamic library building and loading
On 2012-09-29 17:40, Maxim Fomin wrote:

> I was not taking about dynamic loading, but about dynamic linking. If I
> understand topic right, the issue is the former, not the latter.

The title of the thread says "... and loading".

> BTW, in majority cases dynamic loading gives no advantages over dynamic
> linking (just the opposite: doing unnecessary job which can be done by
> linker and loader). In most cases, when an app is being written, it is
> known which functional from which third-party libraries would be used.

I agree, in these cases their no point in using dynamic loading.

> The only exceptions I can name are pluggin support and hacker's binary
> ELF/PE tools. I can count few apps in windows and linux which are used
> often, which dynamically load something, and if they do, they likely
> load pluggins. This was made for cases when app developers by definition
> cannot know beforehand full list of used shared libraries.

Exactly, pluggins are one of the main reasons for using dynamic loading. 
An other reason is license issues. There's a D library called Derelict 
which uses dynamic loading to load C libraries (OpenGL, SDL, OpenAL and 
others). One of the many reason for Derelict using dynamic loading is 
due to license issues/complications with these libraries.

http://dsource.org/projects/derelict/

> So, the question is just opposite: "What's the point of using dynamic loading if
> you know beforehand which libraries you use, which happens almost in all
> cases?".

There's no point in doing that. But what I'm saying is if you know 
beforehand the libraries you will use you can get quite far with static 
libraries.

-- 
/Jacob Carlborg
September 30, 2012
Re: dynamic library building and loading
On Saturday, 29 September 2012 at 16:09:14 UTC, Jacob Carlborg 
wrote:
> There's no point in doing that. But what I'm saying is if you 
> know beforehand the libraries you will use you can get quite 
> far with static libraries.

There are plenty of cases where you have to use a dynamically 
loaded lib even if you know before hand what will be loaded. I 
think you understand this if I read your posts correctly.

In my case I have a pre-existing C++ app that is designed to load 
user defined C++ plugins. I wanted to interface D to one of my 
own C++ plugins, but I apparently cannot do it reliably because 
of initialization issues with the GC and possibly some other 
obscure items.

If I can figure out what needs to be done to resolve the problem 
in enough detail, then maybe I can hack the runtime source and 
roll out a solution.

The GC always seems to pop up as a source of problems. For long 
term solution, the GC should be made 100% optional (in practice 
as opposed to in theory), the standard library ought to be made 
to work with or wothout a GC (or simply without), and the GC 
itself should be easily replaceable with alternate versions. I 
think this idea has already been discussed elsewhere, and is on 
the TODO list (I hope!).

--rt
September 30, 2012
Re: dynamic library building and loading
On Friday, 28 September 2012 at 21:26:47 UTC, Iain Buclaw wrote:
> The way I intend to address it is to have each module handle 
> it's own
> gshared/thread data and pass the information to the D runtime 
> during
> the module construction stage (.ctor) - there is already 
> something
> done this way for _Dmodule_ref - so it may end up being that 
> two new
> fields will be tacked onto it; void[] tlsdata, void[] 
> gsharedata;

I re-built libgphobos and libgdruntime with -fPIC and I can now 
successfully create dynamically loaded D libs. I have 
successfully linked a dynamic D lib to a C++ app, executing some 
test code successfully. I have not yet tried to dlopen a dynamic 
D lib from C++, but I will try maybe tomorrow.

My simple dynamic lib test seems to run fine, but I understand 
that there may be problems, such as the GC failing to work 
properly, and perhaps more can/will go wrong as per Jacob's post
http://forum.dlang.org/post/k4219f$uft$1@digitalmars.com

Any idea when/if you will get around to implementing a fix? Wish 
I could help but I've only just started looking at the source 
code, so whatever I try to fix will probably cause more harm than 
good for a while (but it's a start).

--rt
1 2 3 4 5 6 7
Top | Discussion index | About this forum | D home