View mode: basic / threaded / horizontal-split · Log in · Help
August 09, 2004
Libraries, users, DLLs
Currently, D libraries are predominantly linked in a static fashion. I have
a fairly substantial OO library that I'd really like to distribute as a
dll/shared-lib, but there are apparently a number of issues that stop one
doing so. I may have all of this completely wrong, and I'm really looking
for guidance more that anything else:

1) each DLL has its own GC. Crossing DLL boundaries is a problem for memory
allocations, especially for OO libraries (new doesn't work; some methods
allocate within the DLL but the reference is held by the client instead;
etc).

2) as I understand it, Exceptions do not cross DLL boundaries correctly

One of the goals of providing a library is "ease of use" for the end user. I
don't wish to make anyone jump through hoops just to use an OO class from a
DLL, yet the library really needs to be shareable (there's potentially a
very large number of small bound clients, so one would hope to avoid
static-binding where possible). Can someone point me in the right direction?
Or can we get some support from the compiler, if it's necessary? After all,
a new language is a tough sell without good, easy to use libraries ...
right?
August 09, 2004
Re: Libraries, users, DLLs
antiAlias wrote:

> Currently, D libraries are predominantly linked in a static fashion. I have
> a fairly substantial OO library that I'd really like to distribute as a
> dll/shared-lib, but there are apparently a number of issues that stop one
> doing so. I may have all of this completely wrong, and I'm really looking
> for guidance more that anything else:
> 
> 1) each DLL has its own GC. Crossing DLL boundaries is a problem for memory
> allocations, especially for OO libraries (new doesn't work; some methods
> allocate within the DLL but the reference is held by the client instead;
> etc).
> 
> 2) as I understand it, Exceptions do not cross DLL boundaries correctly
> 
> One of the goals of providing a library is "ease of use" for the end user. I
> don't wish to make anyone jump through hoops just to use an OO class from a
> DLL, yet the library really needs to be shareable (there's potentially a
> very large number of small bound clients, so one would hope to avoid
> static-binding where possible). Can someone point me in the right direction?
> Or can we get some support from the compiler, if it's necessary? After all,
> a new language is a tough sell without good, easy to use libraries ...
> right?
> 
> 

I have to say that I agree. DLL's are supposed to be the glorious way 
forward (at least it is what is preached outside the D world), but being 
rather difficult to use correctly and safely seems to keep most people 
from actually creating any.

At the very least, strategies for how to overcome the problems mentioned 
above, should be collected (at the Wiki, I suppose). Maybe it's possible 
to make tools that can create the factories, stubs, etc automatically, 
but it sure would be nice to be able to use them the easy way.

I would like to come with suggestions for how to 'fix' the DLL's, but I 
don't think I really have the know-how to even suggest. I *do* know 
though, that I would like to not have to use reference-counting.

Lars Ivar Igesund
August 09, 2004
Re: Libraries, users, DLLs
"Lars Ivar Igesund" <larsivar@igesund.net> wrote in message news:cf73n3$b1l$2@digitaldaemon.com...
> antiAlias wrote:
>
> > Currently, D libraries are predominantly linked in a static fashion. I have
> > a fairly substantial OO library that I'd really like to distribute as a
> > dll/shared-lib, but there are apparently a number of issues that stop one
> > doing so. I may have all of this completely wrong, and I'm really looking
> > for guidance more that anything else:
> >
> > 1) each DLL has its own GC. Crossing DLL boundaries is a problem for memory
> > allocations, especially for OO libraries (new doesn't work; some methods
> > allocate within the DLL but the reference is held by the client instead;
> > etc).
> >
> > 2) as I understand it, Exceptions do not cross DLL boundaries correctly
> >
> > One of the goals of providing a library is "ease of use" for the end user. I
> > don't wish to make anyone jump through hoops just to use an OO class from a
> > DLL, yet the library really needs to be shareable (there's potentially a
> > very large number of small bound clients, so one would hope to avoid
> > static-binding where possible). Can someone point me in the right direction?
> > Or can we get some support from the compiler, if it's necessary? After all,
> > a new language is a tough sell without good, easy to use libraries ...
> > right?
> >
> >
>
> I have to say that I agree. DLL's are supposed to be the glorious way
> forward (at least it is what is preached outside the D world), but being
> rather difficult to use correctly and safely seems to keep most people
> from actually creating any.
>
> At the very least, strategies for how to overcome the problems mentioned
> above, should be collected (at the Wiki, I suppose). Maybe it's possible
> to make tools that can create the factories, stubs, etc automatically,
> but it sure would be nice to be able to use them the easy way.
>
> I would like to come with suggestions for how to 'fix' the DLL's, but I
> don't think I really have the know-how to even suggest. I *do* know
> though, that I would like to not have to use reference-counting.
>
> Lars Ivar Igesund

If someone could right a brief - 2 page? - treatise on the current state, and their perceived problems, I'd be happy to
take a look at start to digest the problem.
August 09, 2004
Re: Libraries, users, DLLs
In article <cf73n3$b1l$2@digitaldaemon.com>, Lars Ivar Igesund says...
>
>antiAlias wrote:
>
>> Currently, D libraries are predominantly linked in a static fashion. I have
>> a fairly substantial OO library that I'd really like to distribute as a
>> dll/shared-lib, but there are apparently a number of issues that stop one
>> doing so. I may have all of this completely wrong, and I'm really looking
>> for guidance more that anything else:
>> 
>> 1) each DLL has its own GC. Crossing DLL boundaries is a problem for memory
>> allocations, especially for OO libraries (new doesn't work; some methods
>> allocate within the DLL but the reference is held by the client instead;
>> etc).
>> 
>> 2) as I understand it, Exceptions do not cross DLL boundaries correctly
>> 
>> One of the goals of providing a library is "ease of use" for the end user. I
>> don't wish to make anyone jump through hoops just to use an OO class from a
>> DLL, yet the library really needs to be shareable (there's potentially a
>> very large number of small bound clients, so one would hope to avoid
>> static-binding where possible). Can someone point me in the right direction?
>> Or can we get some support from the compiler, if it's necessary? After all,
>> a new language is a tough sell without good, easy to use libraries ...
>> right?
>> 
>> 
>
>I have to say that I agree. DLL's are supposed to be the glorious way 
>forward (at least it is what is preached outside the D world), but being 
>rather difficult to use correctly and safely seems to keep most people 
>from actually creating any.
>
>At the very least, strategies for how to overcome the problems mentioned 
>above, should be collected (at the Wiki, I suppose). Maybe it's possible 
>to make tools that can create the factories, stubs, etc automatically, 
>but it sure would be nice to be able to use them the easy way.
>
>I would like to come with suggestions for how to 'fix' the DLL's, but I 
>don't think I really have the know-how to even suggest. I *do* know 
>though, that I would like to not have to use reference-counting.
>
>Lars Ivar Igesund

Honestly, I've already surmounted a huge number of issues regarding D DLL's in
my time writing DSP.  I've had to devise some workarounds including, but not
limited to: object proxies, pointer lists and event messaging to signal dll
unloads.  I like your suggestion that solutions should be up on the Wiki, and I
may publish my findings there. :)  Like everyone in this thread, I want more
from D with regard to how libraries are handled.

Here's my $0.02:

What is really missing is giving the DLL's a larger role in the eyes of the
garbage collector.  Having a separate GC on each library works well when having
D inter-operate with different languages, but within a 100% D project it can
become a hinderance.  Without close and careful management of where a given D
object "lives", releasing a DLL mid-process can be disasterous.  However, this
isn't any different from what we've already come to know about C and C++, its
just something we would all rather live without (what with D already making
strides to overcome other problems in C++).

Two years ago, I put down development of my own GC-based runtime library for C++
 I came across this issue and devised some solutions for it, but was unable to
implement it as my GC engine had some impossible-to-debug quirks with it.  For
all that effort, I did learn some rather invaluable lessons about the impact a
collector has on a language.

Hooking a GC into another dll isn't that hard to do, as long as one is willing
to take on the notion of D-platform-dependent libraries.  Ultimately, I realized
that the problem regarding Dll's and GC engines is easy provided you can answer
one question:

Does a given pointer represent a pointer to code on a given DLL, or to a valid
root in a given DLL?

From this we can deduce the rest of the information we need to do everything
else dynamically.  If a DLL has no vtables or delegates pointing to it's code,
and no pointers elsewhere to any other roots within the DLL, then there are no
dependencies on that DLL and it may be dismissed.  This leads up to the notion
that DLL's themselves may be managed by utilizing the gc.

A DLL can be treated as a collectable object, provided that its root pointers
and _entire code space_ are set as roots to be scanned by the hosting process's
garbage collector.  Such A DLL can be unloaded from the process in a manner
analagous to collection of a typical object (i.e. lazy destruction and release
of resources).

The result is something that is almost totally transparent to the language,
provided that phobos is given some additional support (outside of std.loader)
for DLLs.  Careful object management and prodding the GC should be enough to get
a library to release/unload/close whenever needed.

By constrast an explicit and immediate unload of a DLL requires far more
plumbing and participation on behalf of the library developer and library user
alike.  Collectable DLL's, IMO are the way forward for D.

- Pragma
August 20, 2004
Re: Libraries, users, DLLs
What you suggest sounds really promising Eric, and it would certainly help
tremendously with what I'm doing.

So, what am I doing, specifically? Well, Mango has a clustering package,
used for the caching, queuing, and execution of D classes across a LAN.
Naturally, the classes passed around at runtime must be known at
compile-time.

Within a cluster, the server (storage) nodes get around this problem by
maintaining such classes in serialized form only. That is, they deliberately
avoid instantiating an instance (otherwise they'd need to be compiled with
every single possible class that might be stored).

The cluster-clients, on the other hand, do need to know about each class ~
since they will receive and operate upon concrete class instances. These
classes must be known at compile time. For certain applications this is not
a restriction of any kind. In fact, it may actually be viewed as security
benefit.

However, most software systems evolve rather dynamically, and rebooting an
entire operational network to get some new class-instances configured into
the cluster is just not practical.

That's where DLL's (or shared libs) step in. In particular, the clustering
package would benefit greatly if the mobile classes involved were packaged
as discrete dll's. That way, it would be easy to ship them around the
cluster too (as files), and they'd 'attach' to client nodes as appropriate.
This is very similar in approach to how servlets are loaded dynamically into
a running server (and reloaded when they are seen to change).

The other factor involved in such an approach is this: these mobile classes
often need to use resources outside of their own codebase (think Mango
classes, or a database, or whatever). One does not want to bloat the mobile
dll's with such artifacts ~ they should be bound when said dll is loaded
instead. That is, the Mango library itself should be a dll.

Being able to do this cleanly, elegantly, efficiently, and robustly is
important. I think something needs to be done to bring D up-to-date in this
respect.

How about it?




"pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in
message news:cf80ak$m3v$1@digitaldaemon.com...
> In article <cf73n3$b1l$2@digitaldaemon.com>, Lars Ivar Igesund says...
> >
> >antiAlias wrote:
> >
> >> Currently, D libraries are predominantly linked in a static fashion. I
have
> >> a fairly substantial OO library that I'd really like to distribute as a
> >> dll/shared-lib, but there are apparently a number of issues that stop
one
> >> doing so. I may have all of this completely wrong, and I'm really
looking
> >> for guidance more that anything else:
> >>
> >> 1) each DLL has its own GC. Crossing DLL boundaries is a problem for
memory
> >> allocations, especially for OO libraries (new doesn't work; some
methods
> >> allocate within the DLL but the reference is held by the client
instead;
> >> etc).
> >>
> >> 2) as I understand it, Exceptions do not cross DLL boundaries correctly
> >>
> >> One of the goals of providing a library is "ease of use" for the end
user. I
> >> don't wish to make anyone jump through hoops just to use an OO class
from a
> >> DLL, yet the library really needs to be shareable (there's potentially
a
> >> very large number of small bound clients, so one would hope to avoid
> >> static-binding where possible). Can someone point me in the right
direction?
> >> Or can we get some support from the compiler, if it's necessary? After
all,
> >> a new language is a tough sell without good, easy to use libraries ...
> >> right?
> >>
> >>
> >
> >I have to say that I agree. DLL's are supposed to be the glorious way
> >forward (at least it is what is preached outside the D world), but being
> >rather difficult to use correctly and safely seems to keep most people
> >from actually creating any.
> >
> >At the very least, strategies for how to overcome the problems mentioned
> >above, should be collected (at the Wiki, I suppose). Maybe it's possible
> >to make tools that can create the factories, stubs, etc automatically,
> >but it sure would be nice to be able to use them the easy way.
> >
> >I would like to come with suggestions for how to 'fix' the DLL's, but I
> >don't think I really have the know-how to even suggest. I *do* know
> >though, that I would like to not have to use reference-counting.
> >
> >Lars Ivar Igesund
>
> Honestly, I've already surmounted a huge number of issues regarding D
DLL's in
> my time writing DSP.  I've had to devise some workarounds including, but
not
> limited to: object proxies, pointer lists and event messaging to signal
dll
> unloads.  I like your suggestion that solutions should be up on the Wiki,
and I
> may publish my findings there. :)  Like everyone in this thread, I want
more
> from D with regard to how libraries are handled.
>
> Here's my $0.02:
>
> What is really missing is giving the DLL's a larger role in the eyes of
the
> garbage collector.  Having a separate GC on each library works well when
having
> D inter-operate with different languages, but within a 100% D project it
can
> become a hinderance.  Without close and careful management of where a
given D
> object "lives", releasing a DLL mid-process can be disasterous.  However,
this
> isn't any different from what we've already come to know about C and C++,
its
> just something we would all rather live without (what with D already
making
> strides to overcome other problems in C++).
>
> Two years ago, I put down development of my own GC-based runtime library
for C++
>   I came across this issue and devised some solutions for it, but was
unable to
> implement it as my GC engine had some impossible-to-debug quirks with it.
For
> all that effort, I did learn some rather invaluable lessons about the
impact a
> collector has on a language.
>
> Hooking a GC into another dll isn't that hard to do, as long as one is
willing
> to take on the notion of D-platform-dependent libraries.  Ultimately, I
realized
> that the problem regarding Dll's and GC engines is easy provided you can
answer
> one question:
>
> Does a given pointer represent a pointer to code on a given DLL, or to a
valid
> root in a given DLL?
>
> From this we can deduce the rest of the information we need to do
everything
> else dynamically.  If a DLL has no vtables or delegates pointing to it's
code,
> and no pointers elsewhere to any other roots within the DLL, then there
are no
> dependencies on that DLL and it may be dismissed.  This leads up to the
notion
> that DLL's themselves may be managed by utilizing the gc.
>
> A DLL can be treated as a collectable object, provided that its root
pointers
> and _entire code space_ are set as roots to be scanned by the hosting
process's
> garbage collector.  Such A DLL can be unloaded from the process in a
manner
> analagous to collection of a typical object (i.e. lazy destruction and
release
> of resources).
>
> The result is something that is almost totally transparent to the
language,
> provided that phobos is given some additional support (outside of
std.loader)
> for DLLs.  Careful object management and prodding the GC should be enough
to get
> a library to release/unload/close whenever needed.
>
> By constrast an explicit and immediate unload of a DLL requires far more
> plumbing and participation on behalf of the library developer and library
user
> alike.  Collectable DLL's, IMO are the way forward for D.
>
> - Pragma
>
>
August 20, 2004
How should D work with DLLs at the "class" level. (Was Re: Libraries, users, DLLs)
Kris,

... cough ... splutter ...

sorry.

Mr Alias

From your description of your requirements I would say that it's _possible_ that you can implement what you need by
using interfaces, and being _very_ careful with your memory management, but I don't fancy your (or anyone else's)
chances of making such an infrastructure simply, robustly and flexibly. For my money, I don't think D's got any kind of
acceptable answer to this issue atm.

Nomenclatural definition: I use the term "link unit" a lot. I'm sure you'll grok the full extent of the concept when I
say that an exes and dynamic libs are link-units, and static libs are not. Another way to think of it is link units
define a complete symbol name space. (For more on this and other fascinating issues, please feel free to preorder a copy
of "Imperfect C++" from http://www.amazon.com/exec/obidos/tg/detail/-/0321228774/ <g>)

I see four issues with DLLs and D. I'm interested to hear whether I've missed any and, especially, what ideas people
have for solving these issues in a "Spirit of D" way (i.e. without all the clodhopping nonsense that is the Java/.NET
infrastructure):

1. Sharing/managing GC - I think this is the most important in the short/medium term
2. Being able to talk _classes_ not just _interfaces_ between link units. We can get a long way with interfaces,
methinks, once GC sharing is handled, but still it'd be nice to work with classes as well, if simply achievable.
3. Versioning, i.e. avoiding DLL Hell
4. Security. Is there something simple and efficient that nonetheless addresses those handled by the heaving leviathan
that is .NET?

Ancilary issues:

1. Serialisation as a part of the language??


(Germane) thoughts everyone?

Cheers



-- 
Matthew Wilson

Author: "Imperfect C++", Addison-Wesley, 2004
   (http://www.imperfectcplusplus.com)
Contributing editor, C/C++ Users Journal
   (http://www.synesis.com.au/articles.html#columns)
Director, Synesis Software
   (www.synesis.com.au)
STLSoft moderator
   (http://www.stlsoft.org)

"Youth ages, Immaturity is outgrown, Ignorance can be educated, drunkeness sobered. But stupid lasts forever",
Aristophenes

-----------------------------------------------------





"antiAlias" <fu@bar.com> wrote in message news:cg3pmt$2q32$1@digitaldaemon.com...
> What you suggest sounds really promising Eric, and it would certainly help
> tremendously with what I'm doing.
>
> So, what am I doing, specifically? Well, Mango has a clustering package,
> used for the caching, queuing, and execution of D classes across a LAN.
> Naturally, the classes passed around at runtime must be known at
> compile-time.
>
> Within a cluster, the server (storage) nodes get around this problem by
> maintaining such classes in serialized form only. That is, they deliberately
> avoid instantiating an instance (otherwise they'd need to be compiled with
> every single possible class that might be stored).
>
> The cluster-clients, on the other hand, do need to know about each class ~
> since they will receive and operate upon concrete class instances. These
> classes must be known at compile time. For certain applications this is not
> a restriction of any kind. In fact, it may actually be viewed as security
> benefit.
>
> However, most software systems evolve rather dynamically, and rebooting an
> entire operational network to get some new class-instances configured into
> the cluster is just not practical.
>
> That's where DLL's (or shared libs) step in. In particular, the clustering
> package would benefit greatly if the mobile classes involved were packaged
> as discrete dll's. That way, it would be easy to ship them around the
> cluster too (as files), and they'd 'attach' to client nodes as appropriate.
> This is very similar in approach to how servlets are loaded dynamically into
> a running server (and reloaded when they are seen to change).
>
> The other factor involved in such an approach is this: these mobile classes
> often need to use resources outside of their own codebase (think Mango
> classes, or a database, or whatever). One does not want to bloat the mobile
> dll's with such artifacts ~ they should be bound when said dll is loaded
> instead. That is, the Mango library itself should be a dll.
>
> Being able to do this cleanly, elegantly, efficiently, and robustly is
> important. I think something needs to be done to bring D up-to-date in this
> respect.
>
> How about it?
>
>
>
>
> "pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in
> message news:cf80ak$m3v$1@digitaldaemon.com...
> > In article <cf73n3$b1l$2@digitaldaemon.com>, Lars Ivar Igesund says...
> > >
> > >antiAlias wrote:
> > >
> > >> Currently, D libraries are predominantly linked in a static fashion. I
> have
> > >> a fairly substantial OO library that I'd really like to distribute as a
> > >> dll/shared-lib, but there are apparently a number of issues that stop
> one
> > >> doing so. I may have all of this completely wrong, and I'm really
> looking
> > >> for guidance more that anything else:
> > >>
> > >> 1) each DLL has its own GC. Crossing DLL boundaries is a problem for
> memory
> > >> allocations, especially for OO libraries (new doesn't work; some
> methods
> > >> allocate within the DLL but the reference is held by the client
> instead;
> > >> etc).
> > >>
> > >> 2) as I understand it, Exceptions do not cross DLL boundaries correctly
> > >>
> > >> One of the goals of providing a library is "ease of use" for the end
> user. I
> > >> don't wish to make anyone jump through hoops just to use an OO class
> from a
> > >> DLL, yet the library really needs to be shareable (there's potentially
> a
> > >> very large number of small bound clients, so one would hope to avoid
> > >> static-binding where possible). Can someone point me in the right
> direction?
> > >> Or can we get some support from the compiler, if it's necessary? After
> all,
> > >> a new language is a tough sell without good, easy to use libraries ...
> > >> right?
> > >>
> > >>
> > >
> > >I have to say that I agree. DLL's are supposed to be the glorious way
> > >forward (at least it is what is preached outside the D world), but being
> > >rather difficult to use correctly and safely seems to keep most people
> > >from actually creating any.
> > >
> > >At the very least, strategies for how to overcome the problems mentioned
> > >above, should be collected (at the Wiki, I suppose). Maybe it's possible
> > >to make tools that can create the factories, stubs, etc automatically,
> > >but it sure would be nice to be able to use them the easy way.
> > >
> > >I would like to come with suggestions for how to 'fix' the DLL's, but I
> > >don't think I really have the know-how to even suggest. I *do* know
> > >though, that I would like to not have to use reference-counting.
> > >
> > >Lars Ivar Igesund
> >
> > Honestly, I've already surmounted a huge number of issues regarding D
> DLL's in
> > my time writing DSP.  I've had to devise some workarounds including, but
> not
> > limited to: object proxies, pointer lists and event messaging to signal
> dll
> > unloads.  I like your suggestion that solutions should be up on the Wiki,
> and I
> > may publish my findings there. :)  Like everyone in this thread, I want
> more
> > from D with regard to how libraries are handled.
> >
> > Here's my $0.02:
> >
> > What is really missing is giving the DLL's a larger role in the eyes of
> the
> > garbage collector.  Having a separate GC on each library works well when
> having
> > D inter-operate with different languages, but within a 100% D project it
> can
> > become a hinderance.  Without close and careful management of where a
> given D
> > object "lives", releasing a DLL mid-process can be disasterous.  However,
> this
> > isn't any different from what we've already come to know about C and C++,
> its
> > just something we would all rather live without (what with D already
> making
> > strides to overcome other problems in C++).
> >
> > Two years ago, I put down development of my own GC-based runtime library
> for C++
> >   I came across this issue and devised some solutions for it, but was
> unable to
> > implement it as my GC engine had some impossible-to-debug quirks with it.
> For
> > all that effort, I did learn some rather invaluable lessons about the
> impact a
> > collector has on a language.
> >
> > Hooking a GC into another dll isn't that hard to do, as long as one is
> willing
> > to take on the notion of D-platform-dependent libraries.  Ultimately, I
> realized
> > that the problem regarding Dll's and GC engines is easy provided you can
> answer
> > one question:
> >
> > Does a given pointer represent a pointer to code on a given DLL, or to a
> valid
> > root in a given DLL?
> >
> > From this we can deduce the rest of the information we need to do
> everything
> > else dynamically.  If a DLL has no vtables or delegates pointing to it's
> code,
> > and no pointers elsewhere to any other roots within the DLL, then there
> are no
> > dependencies on that DLL and it may be dismissed.  This leads up to the
> notion
> > that DLL's themselves may be managed by utilizing the gc.
> >
> > A DLL can be treated as a collectable object, provided that its root
> pointers
> > and _entire code space_ are set as roots to be scanned by the hosting
> process's
> > garbage collector.  Such A DLL can be unloaded from the process in a
> manner
> > analagous to collection of a typical object (i.e. lazy destruction and
> release
> > of resources).
> >
> > The result is something that is almost totally transparent to the
> language,
> > provided that phobos is given some additional support (outside of
> std.loader)
> > for DLLs.  Careful object management and prodding the GC should be enough
> to get
> > a library to release/unload/close whenever needed.
> >
> > By constrast an explicit and immediate unload of a DLL requires far more
> > plumbing and participation on behalf of the library developer and library
> user
> > alike.  Collectable DLL's, IMO are the way forward for D.
> >
> > - Pragma
> >
> >
>
>
August 20, 2004
Re: How should D work with DLLs at the "class" level. (Was Re: Libraries, users, DLLs)
I agree that #1 (the shared GC) is a primary issue. On the face of it, what
Eric describes (below) sounds really attractive.

Other than that, what about the exception-boundary issue?



"Matthew" <admin.hat@stlsoft.dot.org> wrote in message
news:cg3ul4$2sq0$1@digitaldaemon.com...
> Kris,
>
> ... cough ... splutter ...
>
> sorry.
>
> Mr Alias
>
> From your description of your requirements I would say that it's
_possible_ that you can implement what you need by
> using interfaces, and being _very_ careful with your memory management,
but I don't fancy your (or anyone else's)
> chances of making such an infrastructure simply, robustly and flexibly.
For my money, I don't think D's got any kind of
> acceptable answer to this issue atm.
>
> Nomenclatural definition: I use the term "link unit" a lot. I'm sure
you'll grok the full extent of the concept when I
> say that an exes and dynamic libs are link-units, and static libs are not.
Another way to think of it is link units
> define a complete symbol name space. (For more on this and other
fascinating issues, please feel free to preorder a copy
> of "Imperfect C++" from
http://www.amazon.com/exec/obidos/tg/detail/-/0321228774/ <g>)
>
> I see four issues with DLLs and D. I'm interested to hear whether I've
missed any and, especially, what ideas people
> have for solving these issues in a "Spirit of D" way (i.e. without all the
clodhopping nonsense that is the Java/.NET
> infrastructure):
>
> 1. Sharing/managing GC - I think this is the most important in the
short/medium term
> 2. Being able to talk _classes_ not just _interfaces_ between link units.
We can get a long way with interfaces,
> methinks, once GC sharing is handled, but still it'd be nice to work with
classes as well, if simply achievable.
> 3. Versioning, i.e. avoiding DLL Hell
> 4. Security. Is there something simple and efficient that nonetheless
addresses those handled by the heaving leviathan
> that is .NET?
>
> Ancilary issues:
>
> 1. Serialisation as a part of the language??
>
>
> (Germane) thoughts everyone?
>
> Cheers
>
>
>
> --
> Matthew Wilson
>
> Author: "Imperfect C++", Addison-Wesley, 2004
>     (http://www.imperfectcplusplus.com)
> Contributing editor, C/C++ Users Journal
>     (http://www.synesis.com.au/articles.html#columns)
> Director, Synesis Software
>     (www.synesis.com.au)
> STLSoft moderator
>     (http://www.stlsoft.org)
>
> "Youth ages, Immaturity is outgrown, Ignorance can be educated, drunkeness
sobered. But stupid lasts forever",
> Aristophenes
>
> -----------------------------------------------------
>
>
>
>
>
> "antiAlias" <fu@bar.com> wrote in message
news:cg3pmt$2q32$1@digitaldaemon.com...
> > What you suggest sounds really promising Eric, and it would certainly
help
> > tremendously with what I'm doing.
> >
> > So, what am I doing, specifically? Well, Mango has a clustering package,
> > used for the caching, queuing, and execution of D classes across a LAN.
> > Naturally, the classes passed around at runtime must be known at
> > compile-time.
> >
> > Within a cluster, the server (storage) nodes get around this problem by
> > maintaining such classes in serialized form only. That is, they
deliberately
> > avoid instantiating an instance (otherwise they'd need to be compiled
with
> > every single possible class that might be stored).
> >
> > The cluster-clients, on the other hand, do need to know about each class
~
> > since they will receive and operate upon concrete class instances. These
> > classes must be known at compile time. For certain applications this is
not
> > a restriction of any kind. In fact, it may actually be viewed as
security
> > benefit.
> >
> > However, most software systems evolve rather dynamically, and rebooting
an
> > entire operational network to get some new class-instances configured
into
> > the cluster is just not practical.
> >
> > That's where DLL's (or shared libs) step in. In particular, the
clustering
> > package would benefit greatly if the mobile classes involved were
packaged
> > as discrete dll's. That way, it would be easy to ship them around the
> > cluster too (as files), and they'd 'attach' to client nodes as
appropriate.
> > This is very similar in approach to how servlets are loaded dynamically
into
> > a running server (and reloaded when they are seen to change).
> >
> > The other factor involved in such an approach is this: these mobile
classes
> > often need to use resources outside of their own codebase (think Mango
> > classes, or a database, or whatever). One does not want to bloat the
mobile
> > dll's with such artifacts ~ they should be bound when said dll is loaded
> > instead. That is, the Mango library itself should be a dll.
> >
> > Being able to do this cleanly, elegantly, efficiently, and robustly is
> > important. I think something needs to be done to bring D up-to-date in
this
> > respect.
> >
> > How about it?
> >
> >
> >
> >
> > "pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote
in
> > message news:cf80ak$m3v$1@digitaldaemon.com...
> > > In article <cf73n3$b1l$2@digitaldaemon.com>, Lars Ivar Igesund says...
> > > >
> > > >antiAlias wrote:
> > > >
> > > >> Currently, D libraries are predominantly linked in a static
fashion. I
> > have
> > > >> a fairly substantial OO library that I'd really like to distribute
as a
> > > >> dll/shared-lib, but there are apparently a number of issues that
stop
> > one
> > > >> doing so. I may have all of this completely wrong, and I'm really
> > looking
> > > >> for guidance more that anything else:
> > > >>
> > > >> 1) each DLL has its own GC. Crossing DLL boundaries is a problem
for
> > memory
> > > >> allocations, especially for OO libraries (new doesn't work; some
> > methods
> > > >> allocate within the DLL but the reference is held by the client
> > instead;
> > > >> etc).
> > > >>
> > > >> 2) as I understand it, Exceptions do not cross DLL boundaries
correctly
> > > >>
> > > >> One of the goals of providing a library is "ease of use" for the
end
> > user. I
> > > >> don't wish to make anyone jump through hoops just to use an OO
class
> > from a
> > > >> DLL, yet the library really needs to be shareable (there's
potentially
> > a
> > > >> very large number of small bound clients, so one would hope to
avoid
> > > >> static-binding where possible). Can someone point me in the right
> > direction?
> > > >> Or can we get some support from the compiler, if it's necessary?
After
> > all,
> > > >> a new language is a tough sell without good, easy to use libraries
...
> > > >> right?
> > > >>
> > > >>
> > > >
> > > >I have to say that I agree. DLL's are supposed to be the glorious way
> > > >forward (at least it is what is preached outside the D world), but
being
> > > >rather difficult to use correctly and safely seems to keep most
people
> > > >from actually creating any.
> > > >
> > > >At the very least, strategies for how to overcome the problems
mentioned
> > > >above, should be collected (at the Wiki, I suppose). Maybe it's
possible
> > > >to make tools that can create the factories, stubs, etc
automatically,
> > > >but it sure would be nice to be able to use them the easy way.
> > > >
> > > >I would like to come with suggestions for how to 'fix' the DLL's, but
I
> > > >don't think I really have the know-how to even suggest. I *do* know
> > > >though, that I would like to not have to use reference-counting.
> > > >
> > > >Lars Ivar Igesund
> > >
> > > Honestly, I've already surmounted a huge number of issues regarding D
> > DLL's in
> > > my time writing DSP.  I've had to devise some workarounds including,
but
> > not
> > > limited to: object proxies, pointer lists and event messaging to
signal
> > dll
> > > unloads.  I like your suggestion that solutions should be up on the
Wiki,
> > and I
> > > may publish my findings there. :)  Like everyone in this thread, I
want
> > more
> > > from D with regard to how libraries are handled.
> > >
> > > Here's my $0.02:
> > >
> > > What is really missing is giving the DLL's a larger role in the eyes
of
> > the
> > > garbage collector.  Having a separate GC on each library works well
when
> > having
> > > D inter-operate with different languages, but within a 100% D project
it
> > can
> > > become a hinderance.  Without close and careful management of where a
> > given D
> > > object "lives", releasing a DLL mid-process can be disasterous.
However,
> > this
> > > isn't any different from what we've already come to know about C and
C++,
> > its
> > > just something we would all rather live without (what with D already
> > making
> > > strides to overcome other problems in C++).
> > >
> > > Two years ago, I put down development of my own GC-based runtime
library
> > for C++
> > >   I came across this issue and devised some solutions for it, but was
> > unable to
> > > implement it as my GC engine had some impossible-to-debug quirks with
it.
> > For
> > > all that effort, I did learn some rather invaluable lessons about the
> > impact a
> > > collector has on a language.
> > >
> > > Hooking a GC into another dll isn't that hard to do, as long as one is
> > willing
> > > to take on the notion of D-platform-dependent libraries.  Ultimately,
I
> > realized
> > > that the problem regarding Dll's and GC engines is easy provided you
can
> > answer
> > > one question:
> > >
> > > Does a given pointer represent a pointer to code on a given DLL, or to
a
> > valid
> > > root in a given DLL?
> > >
> > > From this we can deduce the rest of the information we need to do
> > everything
> > > else dynamically.  If a DLL has no vtables or delegates pointing to
it's
> > code,
> > > and no pointers elsewhere to any other roots within the DLL, then
there
> > are no
> > > dependencies on that DLL and it may be dismissed.  This leads up to
the
> > notion
> > > that DLL's themselves may be managed by utilizing the gc.
> > >
> > > A DLL can be treated as a collectable object, provided that its root
> > pointers
> > > and _entire code space_ are set as roots to be scanned by the hosting
> > process's
> > > garbage collector.  Such A DLL can be unloaded from the process in a
> > manner
> > > analagous to collection of a typical object (i.e. lazy destruction and
> > release
> > > of resources).
> > >
> > > The result is something that is almost totally transparent to the
> > language,
> > > provided that phobos is given some additional support (outside of
> > std.loader)
> > > for DLLs.  Careful object management and prodding the GC should be
enough
> > to get
> > > a library to release/unload/close whenever needed.
> > >
> > > By constrast an explicit and immediate unload of a DLL requires far
more
> > > plumbing and participation on behalf of the library developer and
library
> > user
> > > alike.  Collectable DLL's, IMO are the way forward for D.
> > >
> > > - Pragma
> > >
> > >
> >
> >
>
>
August 20, 2004
Re: How should D work with DLLs at the "class" level. (Was Re: Libraries, users, DLLs)
"antiAlias" <fu@bar.com> wrote in message news:cg408a$2tge$1@digitaldaemon.com...
> I agree that #1 (the shared GC) is a primary issue. On the face of it, what
> Eric describes (below) sounds really attractive.

It sounds promising. I'd like to hear more fine details, though.

I'd also like to hear if his ideas would require changes to std.loader (as I'm itching for an excuse to rewrite that
stillborn obscenity).

I'd also like to hear Walter's opinions on the matter. I think this issue is of sufficient import to be a 1.0
prerequesite, don't you?

> Other than that, what about the exception-boundary issue?

Well, D's going to have an ABI, and since all exceptions are going to co-exist memorywise (when the GC's
linearised/homogenised), what's the prob?

> "Matthew" <admin.hat@stlsoft.dot.org> wrote in message
> news:cg3ul4$2sq0$1@digitaldaemon.com...
> > Kris,
> >
> > ... cough ... splutter ...
> >
> > sorry.
> >
> > Mr Alias
> >
> > From your description of your requirements I would say that it's
> _possible_ that you can implement what you need by
> > using interfaces, and being _very_ careful with your memory management,
> but I don't fancy your (or anyone else's)
> > chances of making such an infrastructure simply, robustly and flexibly.
> For my money, I don't think D's got any kind of
> > acceptable answer to this issue atm.
> >
> > Nomenclatural definition: I use the term "link unit" a lot. I'm sure
> you'll grok the full extent of the concept when I
> > say that an exes and dynamic libs are link-units, and static libs are not.
> Another way to think of it is link units
> > define a complete symbol name space. (For more on this and other
> fascinating issues, please feel free to preorder a copy
> > of "Imperfect C++" from
> http://www.amazon.com/exec/obidos/tg/detail/-/0321228774/ <g>)
> >
> > I see four issues with DLLs and D. I'm interested to hear whether I've
> missed any and, especially, what ideas people
> > have for solving these issues in a "Spirit of D" way (i.e. without all the
> clodhopping nonsense that is the Java/.NET
> > infrastructure):
> >
> > 1. Sharing/managing GC - I think this is the most important in the
> short/medium term
> > 2. Being able to talk _classes_ not just _interfaces_ between link units.
> We can get a long way with interfaces,
> > methinks, once GC sharing is handled, but still it'd be nice to work with
> classes as well, if simply achievable.
> > 3. Versioning, i.e. avoiding DLL Hell
> > 4. Security. Is there something simple and efficient that nonetheless
> addresses those handled by the heaving leviathan
> > that is .NET?
> >
> > Ancilary issues:
> >
> > 1. Serialisation as a part of the language??
> >
> >
> > (Germane) thoughts everyone?
> >
> > Cheers
> >
> >
> >
> > --
> > Matthew Wilson
> >
> > Author: "Imperfect C++", Addison-Wesley, 2004
> >     (http://www.imperfectcplusplus.com)
> > Contributing editor, C/C++ Users Journal
> >     (http://www.synesis.com.au/articles.html#columns)
> > Director, Synesis Software
> >     (www.synesis.com.au)
> > STLSoft moderator
> >     (http://www.stlsoft.org)
> >
> > "Youth ages, Immaturity is outgrown, Ignorance can be educated, drunkeness
> sobered. But stupid lasts forever",
> > Aristophenes
> >
> > -----------------------------------------------------
> >
> >
> >
> >
> >
> > "antiAlias" <fu@bar.com> wrote in message
> news:cg3pmt$2q32$1@digitaldaemon.com...
> > > What you suggest sounds really promising Eric, and it would certainly
> help
> > > tremendously with what I'm doing.
> > >
> > > So, what am I doing, specifically? Well, Mango has a clustering package,
> > > used for the caching, queuing, and execution of D classes across a LAN.
> > > Naturally, the classes passed around at runtime must be known at
> > > compile-time.
> > >
> > > Within a cluster, the server (storage) nodes get around this problem by
> > > maintaining such classes in serialized form only. That is, they
> deliberately
> > > avoid instantiating an instance (otherwise they'd need to be compiled
> with
> > > every single possible class that might be stored).
> > >
> > > The cluster-clients, on the other hand, do need to know about each class
> ~
> > > since they will receive and operate upon concrete class instances. These
> > > classes must be known at compile time. For certain applications this is
> not
> > > a restriction of any kind. In fact, it may actually be viewed as
> security
> > > benefit.
> > >
> > > However, most software systems evolve rather dynamically, and rebooting
> an
> > > entire operational network to get some new class-instances configured
> into
> > > the cluster is just not practical.
> > >
> > > That's where DLL's (or shared libs) step in. In particular, the
> clustering
> > > package would benefit greatly if the mobile classes involved were
> packaged
> > > as discrete dll's. That way, it would be easy to ship them around the
> > > cluster too (as files), and they'd 'attach' to client nodes as
> appropriate.
> > > This is very similar in approach to how servlets are loaded dynamically
> into
> > > a running server (and reloaded when they are seen to change).
> > >
> > > The other factor involved in such an approach is this: these mobile
> classes
> > > often need to use resources outside of their own codebase (think Mango
> > > classes, or a database, or whatever). One does not want to bloat the
> mobile
> > > dll's with such artifacts ~ they should be bound when said dll is loaded
> > > instead. That is, the Mango library itself should be a dll.
> > >
> > > Being able to do this cleanly, elegantly, efficiently, and robustly is
> > > important. I think something needs to be done to bring D up-to-date in
> this
> > > respect.
> > >
> > > How about it?
> > >
> > >
> > >
> > >
> > > "pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote
> in
> > > message news:cf80ak$m3v$1@digitaldaemon.com...
> > > > In article <cf73n3$b1l$2@digitaldaemon.com>, Lars Ivar Igesund says...
> > > > >
> > > > >antiAlias wrote:
> > > > >
> > > > >> Currently, D libraries are predominantly linked in a static
> fashion. I
> > > have
> > > > >> a fairly substantial OO library that I'd really like to distribute
> as a
> > > > >> dll/shared-lib, but there are apparently a number of issues that
> stop
> > > one
> > > > >> doing so. I may have all of this completely wrong, and I'm really
> > > looking
> > > > >> for guidance more that anything else:
> > > > >>
> > > > >> 1) each DLL has its own GC. Crossing DLL boundaries is a problem
> for
> > > memory
> > > > >> allocations, especially for OO libraries (new doesn't work; some
> > > methods
> > > > >> allocate within the DLL but the reference is held by the client
> > > instead;
> > > > >> etc).
> > > > >>
> > > > >> 2) as I understand it, Exceptions do not cross DLL boundaries
> correctly
> > > > >>
> > > > >> One of the goals of providing a library is "ease of use" for the
> end
> > > user. I
> > > > >> don't wish to make anyone jump through hoops just to use an OO
> class
> > > from a
> > > > >> DLL, yet the library really needs to be shareable (there's
> potentially
> > > a
> > > > >> very large number of small bound clients, so one would hope to
> avoid
> > > > >> static-binding where possible). Can someone point me in the right
> > > direction?
> > > > >> Or can we get some support from the compiler, if it's necessary?
> After
> > > all,
> > > > >> a new language is a tough sell without good, easy to use libraries
> ...
> > > > >> right?
> > > > >>
> > > > >>
> > > > >
> > > > >I have to say that I agree. DLL's are supposed to be the glorious way
> > > > >forward (at least it is what is preached outside the D world), but
> being
> > > > >rather difficult to use correctly and safely seems to keep most
> people
> > > > >from actually creating any.
> > > > >
> > > > >At the very least, strategies for how to overcome the problems
> mentioned
> > > > >above, should be collected (at the Wiki, I suppose). Maybe it's
> possible
> > > > >to make tools that can create the factories, stubs, etc
> automatically,
> > > > >but it sure would be nice to be able to use them the easy way.
> > > > >
> > > > >I would like to come with suggestions for how to 'fix' the DLL's, but
> I
> > > > >don't think I really have the know-how to even suggest. I *do* know
> > > > >though, that I would like to not have to use reference-counting.
> > > > >
> > > > >Lars Ivar Igesund
> > > >
> > > > Honestly, I've already surmounted a huge number of issues regarding D
> > > DLL's in
> > > > my time writing DSP.  I've had to devise some workarounds including,
> but
> > > not
> > > > limited to: object proxies, pointer lists and event messaging to
> signal
> > > dll
> > > > unloads.  I like your suggestion that solutions should be up on the
> Wiki,
> > > and I
> > > > may publish my findings there. :)  Like everyone in this thread, I
> want
> > > more
> > > > from D with regard to how libraries are handled.
> > > >
> > > > Here's my $0.02:
> > > >
> > > > What is really missing is giving the DLL's a larger role in the eyes
> of
> > > the
> > > > garbage collector.  Having a separate GC on each library works well
> when
> > > having
> > > > D inter-operate with different languages, but within a 100% D project
> it
> > > can
> > > > become a hinderance.  Without close and careful management of where a
> > > given D
> > > > object "lives", releasing a DLL mid-process can be disasterous.
> However,
> > > this
> > > > isn't any different from what we've already come to know about C and
> C++,
> > > its
> > > > just something we would all rather live without (what with D already
> > > making
> > > > strides to overcome other problems in C++).
> > > >
> > > > Two years ago, I put down development of my own GC-based runtime
> library
> > > for C++
> > > >   I came across this issue and devised some solutions for it, but was
> > > unable to
> > > > implement it as my GC engine had some impossible-to-debug quirks with
> it.
> > > For
> > > > all that effort, I did learn some rather invaluable lessons about the
> > > impact a
> > > > collector has on a language.
> > > >
> > > > Hooking a GC into another dll isn't that hard to do, as long as one is
> > > willing
> > > > to take on the notion of D-platform-dependent libraries.  Ultimately,
> I
> > > realized
> > > > that the problem regarding Dll's and GC engines is easy provided you
> can
> > > answer
> > > > one question:
> > > >
> > > > Does a given pointer represent a pointer to code on a given DLL, or to
> a
> > > valid
> > > > root in a given DLL?
> > > >
> > > > From this we can deduce the rest of the information we need to do
> > > everything
> > > > else dynamically.  If a DLL has no vtables or delegates pointing to
> it's
> > > code,
> > > > and no pointers elsewhere to any other roots within the DLL, then
> there
> > > are no
> > > > dependencies on that DLL and it may be dismissed.  This leads up to
> the
> > > notion
> > > > that DLL's themselves may be managed by utilizing the gc.
> > > >
> > > > A DLL can be treated as a collectable object, provided that its root
> > > pointers
> > > > and _entire code space_ are set as roots to be scanned by the hosting
> > > process's
> > > > garbage collector.  Such A DLL can be unloaded from the process in a
> > > manner
> > > > analagous to collection of a typical object (i.e. lazy destruction and
> > > release
> > > > of resources).
> > > >
> > > > The result is something that is almost totally transparent to the
> > > language,
> > > > provided that phobos is given some additional support (outside of
> > > std.loader)
> > > > for DLLs.  Careful object management and prodding the GC should be
> enough
> > > to get
> > > > a library to release/unload/close whenever needed.
> > > >
> > > > By constrast an explicit and immediate unload of a DLL requires far
> more
> > > > plumbing and participation on behalf of the library developer and
> library
> > > user
> > > > alike.  Collectable DLL's, IMO are the way forward for D.
> > > >
> > > > - Pragma
> > > >
> > > >
> > >
> > >
> >
> >
>
>
August 20, 2004
Re: How should D work with DLLs at the
FYI: this is a long post, please read.  Thank you. :)

In article <cg41tm$2u6a$1@digitaldaemon.com>, Matthew says...
>
>
>"antiAlias" <fu@bar.com> wrote in message news:cg408a$2tge$1@digitaldaemon.com...
>> I agree that #1 (the shared GC) is a primary issue. On the face of it, what
>> Eric describes (below) sounds really attractive.
>
>It sounds promising. I'd like to hear more fine details, though.

I'll work on a more detailed treatment as soon as time permits.  I'll have to
first go back through my old source/notes and make sure that there's nothing
that I've managed to miss.

One thing I will say though: No matter how many times I revisit the problem, the
businss of explicitly unloading a library at runtime has ugly consequences
regardless of how the GC is hooked.  Also, I blame languages like Java and C#
for raising our expectations of how this should be handled.  I don't think the
problem can be completely solved for in-memory objects, without resorting to
proxies and/or notification lists.

>I'd also like to hear if his ideas would require changes to std.loader (as I'm itching for an excuse to rewrite that
>stillborn obscenity).

I think my first reaction to reading that source was: "This compiled?" (j/k)

I wouldn't anticipate need to change std.loader drastically.  Outside of that,
I'd love to see the thread-local storage bits broken out into another library or
as an extension to std.thread. ;) 

One question though: is all the extra ref-counting code needed for linux
libraries, or does it supply some extra value to both Win32 and Linux
environments?

>
>I'd also like to hear Walter's opinions on the matter. I think this issue is of sufficient import to be a 1.0
>prerequesite, don't you?

I couldn't agree more.  I too, would like to hear what direction things will
take with respect to dll's. 
>
>> Other than that, what about the exception-boundary issue?
>
>Well, D's going to have an ABI, and since all exceptions are going to co-exist memorywise (when the GC's
>linearised/homogenised), what's the prob?

With respect to Kris' concern about handling exceptions, its really obnoxious
how an uncaught exception within a library kills your app.  Aside from making
all of my libraries behave like class-factories, I've taken to using methods
that do this:

> int someMethod(int foo){ /* impl */ }

..to be called and exported by a dll stub function:

> export(Windows) Exception someMethod(int foo,out _returnval){
>   Exception _exception;
>   try{ _returnval = someMethod(foo); }
>   catch(Exception e){ _exception = e; }
>   return(_exception);
> }

You can use a similar stub on the calling side that wraps the function pointer
that you grab from the library.

Meanwhile (on a slightly OT note), I'm still working on some of the more tricky
aspects of dll's in D.  In particular, I've tracked one really nasty bug down to
(what looks like) the digitalmars-C backend:

> // deceptively simple example.
> import std.c.windows.windows;
> void main(){
>   HMODULE mod = LoadLibrary("mydll.dll");
>   printf("freeing...");
>   FreeLibrary(mod);
>   printf("You will never see this print.\n");
> }

(Matthew, Kris, have you had this problem on your Windows machines?  I've done
this on two separate boxes with the same result.  If you two or anyone else can
confirm this, I'll go ahead and post it to the bugs NG)

Basically, stdout and stdin are "automagically" set to EOF (or some similar
state) after the call to FreeLibrary(): nothing goes in and nothing comes out.
Everything else runs fine, no exceptions are thrown, and GetLastError()
faithfully returns 0.

I've been able to recreate the above in DMC, with a 100% C-built Dll using the
same .def file as the mydll sample.  The results are also the same when calling
that dll from a host program written in MSVC6, DMD and DMC.  Unless I'm horribly
mistaken on how to do compose dll's using Digitalmars' technology, this is what
you get the instant you unload any DMC/DMD built dll.  

The problem seems to only manifest when a proper DllMain() method is exported
from the library.  If none is provided, or if the given implementation can be
optimized away, the error does not ocurr.

This caused me hours upon hours of anguish trying to figure out why in my
DSP/Mango server kept quietly halting on me: it was waiting for console input to
terminate, but Mango's Stdin throws on EOF.  :(

- Pragma
August 21, 2004
Re: How should D work with DLLs at the
I've read it once, but want to take some time to think before I respond to some of the points.

Let me pose my question in a more direct manner:

AFAICS, one of the biggest problems is the ability to be able to use DLLs with D in *all* of the following guises:

1. D exe, C dll. Communication by plain C-API function calls. All resources returned whence they came.
2. C exe, D dll. Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). All
resources returned whence they came
3. D exe, D dll, Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). All
resources returned whence they came
4. D exe, D dll, Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). Memory
allocated hither and used safely thither
5. D exe, D dll, Communication via extern(C) calls and/or extern(D) calls and/or interfaces with well-known vtable
layout (e.g. COM). Memory allocated hither and used safely thither
6. D exe, D dll, Communication via 5. + classes. (Not sure of these details.)

Note, for each of the above, you can permute by replacing exe with D client code inside a D dll in any of cases 2-6!

Since D has, putatively at least, an ABI, all 6 (or 11) of these should, in principle be realisable. Further,
commercially speaking, D *must* provide all of these, to not be abandoned by developers more influenced by deadline and
pragmatism than idealism and performance - in other words, most of them.

To my mind, we have two choices. One option is that the GC and supporting infrastructure is *always* in some DCore.dll -
i.e. there're no more statically linked programs - in the same vein as .NET. The other option is that we come up with a
mechanism whereby each link-unit is able to detect whether the process in which it is loaded already has a GC and, if
so, links to that, otherwise starting its own. The complication to this is that if the first such link-unit is not the
process' exe, we have the issue whereby the GC might be prematurely unloaded while other link-units are still depending
on it. This could, hopefully(!), be resolved be having dependent link-units hold an independent dynamic-load reference
on the GC's owning DLL.

Thoughts?

Walter, I'd be very interested in your thoughts on this, both on my comments on the need for this commercially, and also
on possible implementation mechanisms.




"pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in message
news:cg5184$c7u$1@digitaldaemon.com...
> FYI: this is a long post, please read.  Thank you. :)
>
> In article <cg41tm$2u6a$1@digitaldaemon.com>, Matthew says...
> >
> >
> >"antiAlias" <fu@bar.com> wrote in message news:cg408a$2tge$1@digitaldaemon.com...
> >> I agree that #1 (the shared GC) is a primary issue. On the face of it, what
> >> Eric describes (below) sounds really attractive.
> >
> >It sounds promising. I'd like to hear more fine details, though.
>
> I'll work on a more detailed treatment as soon as time permits.  I'll have to
> first go back through my old source/notes and make sure that there's nothing
> that I've managed to miss.
>
> One thing I will say though: No matter how many times I revisit the problem, the
> businss of explicitly unloading a library at runtime has ugly consequences
> regardless of how the GC is hooked.  Also, I blame languages like Java and C#
> for raising our expectations of how this should be handled.  I don't think the
> problem can be completely solved for in-memory objects, without resorting to
> proxies and/or notification lists.
>
> >I'd also like to hear if his ideas would require changes to std.loader (as I'm itching for an excuse to rewrite that
> >stillborn obscenity).
>
> I think my first reaction to reading that source was: "This compiled?" (j/k)
>
> I wouldn't anticipate need to change std.loader drastically.  Outside of that,
> I'd love to see the thread-local storage bits broken out into another library or
> as an extension to std.thread. ;)
>
> One question though: is all the extra ref-counting code needed for linux
> libraries, or does it supply some extra value to both Win32 and Linux
> environments?
>
> >
> >I'd also like to hear Walter's opinions on the matter. I think this issue is of sufficient import to be a 1.0
> >prerequesite, don't you?
>
> I couldn't agree more.  I too, would like to hear what direction things will
> take with respect to dll's.
> >
> >> Other than that, what about the exception-boundary issue?
> >
> >Well, D's going to have an ABI, and since all exceptions are going to co-exist memorywise (when the GC's
> >linearised/homogenised), what's the prob?
>
> With respect to Kris' concern about handling exceptions, its really obnoxious
> how an uncaught exception within a library kills your app.  Aside from making
> all of my libraries behave like class-factories, I've taken to using methods
> that do this:
>
> > int someMethod(int foo){ /* impl */ }
>
> ..to be called and exported by a dll stub function:
>
> > export(Windows) Exception someMethod(int foo,out _returnval){
> >   Exception _exception;
> >   try{ _returnval = someMethod(foo); }
> >   catch(Exception e){ _exception = e; }
> >   return(_exception);
> > }
>
> You can use a similar stub on the calling side that wraps the function pointer
> that you grab from the library.
>
> Meanwhile (on a slightly OT note), I'm still working on some of the more tricky
> aspects of dll's in D.  In particular, I've tracked one really nasty bug down to
> (what looks like) the digitalmars-C backend:
>
> > // deceptively simple example.
> > import std.c.windows.windows;
> > void main(){
> >   HMODULE mod = LoadLibrary("mydll.dll");
> >   printf("freeing...");
> >   FreeLibrary(mod);
> >   printf("You will never see this print.\n");
> > }
>
> (Matthew, Kris, have you had this problem on your Windows machines?  I've done
> this on two separate boxes with the same result.  If you two or anyone else can
> confirm this, I'll go ahead and post it to the bugs NG)
>
> Basically, stdout and stdin are "automagically" set to EOF (or some similar
> state) after the call to FreeLibrary(): nothing goes in and nothing comes out.
> Everything else runs fine, no exceptions are thrown, and GetLastError()
> faithfully returns 0.
>
> I've been able to recreate the above in DMC, with a 100% C-built Dll using the
> same .def file as the mydll sample.  The results are also the same when calling
> that dll from a host program written in MSVC6, DMD and DMC.  Unless I'm horribly
> mistaken on how to do compose dll's using Digitalmars' technology, this is what
> you get the instant you unload any DMC/DMD built dll.
>
> The problem seems to only manifest when a proper DllMain() method is exported
> from the library.  If none is provided, or if the given implementation can be
> optimized away, the error does not ocurr.
>
> This caused me hours upon hours of anguish trying to figure out why in my
> DSP/Mango server kept quietly halting on me: it was waiting for console input to
> terminate, but Mango's Stdin throws on EOF.  :(
>
> - Pragma
>
>
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home