August 21, 2004
I've read it once, but want to take some time to think before I respond to some of the points.

Let me pose my question in a more direct manner:

AFAICS, one of the biggest problems is the ability to be able to use DLLs with D in *all* of the following guises:

1. D exe, C dll. Communication by plain C-API function calls. All resources returned whence they came.
2. C exe, D dll. Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). All
resources returned whence they came
3. D exe, D dll, Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). All
resources returned whence they came
4. D exe, D dll, Communication via extern(C) calls and/or interfaces with well-known vtable layout (e.g. COM). Memory
allocated hither and used safely thither
5. D exe, D dll, Communication via extern(C) calls and/or extern(D) calls and/or interfaces with well-known vtable
layout (e.g. COM). Memory allocated hither and used safely thither
6. D exe, D dll, Communication via 5. + classes. (Not sure of these details.)

Note, for each of the above, you can permute by replacing exe with D client code inside a D dll in any of cases 2-6!

Since D has, putatively at least, an ABI, all 6 (or 11) of these should, in principle be realisable. Further, commercially speaking, D *must* provide all of these, to not be abandoned by developers more influenced by deadline and pragmatism than idealism and performance - in other words, most of them.

To my mind, we have two choices. One option is that the GC and supporting infrastructure is *always* in some DCore.dll - i.e. there're no more statically linked programs - in the same vein as .NET. The other option is that we come up with a mechanism whereby each link-unit is able to detect whether the process in which it is loaded already has a GC and, if so, links to that, otherwise starting its own. The complication to this is that if the first such link-unit is not the process' exe, we have the issue whereby the GC might be prematurely unloaded while other link-units are still depending on it. This could, hopefully(!), be resolved be having dependent link-units hold an independent dynamic-load reference on the GC's owning DLL.

Thoughts?

Walter, I'd be very interested in your thoughts on this, both on my comments on the need for this commercially, and also on possible implementation mechanisms.




"pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in message news:cg5184$c7u$1@digitaldaemon.com...
> FYI: this is a long post, please read.  Thank you. :)
>
> In article <cg41tm$2u6a$1@digitaldaemon.com>, Matthew says...
> >
> >
> >"antiAlias" <fu@bar.com> wrote in message news:cg408a$2tge$1@digitaldaemon.com...
> >> I agree that #1 (the shared GC) is a primary issue. On the face of it, what
> >> Eric describes (below) sounds really attractive.
> >
> >It sounds promising. I'd like to hear more fine details, though.
>
> I'll work on a more detailed treatment as soon as time permits.  I'll have to first go back through my old source/notes and make sure that there's nothing that I've managed to miss.
>
> One thing I will say though: No matter how many times I revisit the problem, the businss of explicitly unloading a library at runtime has ugly consequences regardless of how the GC is hooked.  Also, I blame languages like Java and C# for raising our expectations of how this should be handled.  I don't think the problem can be completely solved for in-memory objects, without resorting to proxies and/or notification lists.
>
> >I'd also like to hear if his ideas would require changes to std.loader (as I'm itching for an excuse to rewrite that stillborn obscenity).
>
> I think my first reaction to reading that source was: "This compiled?" (j/k)
>
> I wouldn't anticipate need to change std.loader drastically.  Outside of that, I'd love to see the thread-local storage bits broken out into another library or as an extension to std.thread. ;)
>
> One question though: is all the extra ref-counting code needed for linux libraries, or does it supply some extra value to both Win32 and Linux environments?
>
> >
> >I'd also like to hear Walter's opinions on the matter. I think this issue is of sufficient import to be a 1.0 prerequesite, don't you?
>
> I couldn't agree more.  I too, would like to hear what direction things will take with respect to dll's.
> >
> >> Other than that, what about the exception-boundary issue?
> >
> >Well, D's going to have an ABI, and since all exceptions are going to co-exist memorywise (when the GC's linearised/homogenised), what's the prob?
>
> With respect to Kris' concern about handling exceptions, its really obnoxious how an uncaught exception within a library kills your app.  Aside from making all of my libraries behave like class-factories, I've taken to using methods that do this:
>
> > int someMethod(int foo){ /* impl */ }
>
> ..to be called and exported by a dll stub function:
>
> > export(Windows) Exception someMethod(int foo,out _returnval){
> >   Exception _exception;
> >   try{ _returnval = someMethod(foo); }
> >   catch(Exception e){ _exception = e; }
> >   return(_exception);
> > }
>
> You can use a similar stub on the calling side that wraps the function pointer that you grab from the library.
>
> Meanwhile (on a slightly OT note), I'm still working on some of the more tricky aspects of dll's in D.  In particular, I've tracked one really nasty bug down to (what looks like) the digitalmars-C backend:
>
> > // deceptively simple example.
> > import std.c.windows.windows;
> > void main(){
> >   HMODULE mod = LoadLibrary("mydll.dll");
> >   printf("freeing...");
> >   FreeLibrary(mod);
> >   printf("You will never see this print.\n");
> > }
>
> (Matthew, Kris, have you had this problem on your Windows machines?  I've done this on two separate boxes with the same result.  If you two or anyone else can confirm this, I'll go ahead and post it to the bugs NG)
>
> Basically, stdout and stdin are "automagically" set to EOF (or some similar
> state) after the call to FreeLibrary(): nothing goes in and nothing comes out.
> Everything else runs fine, no exceptions are thrown, and GetLastError()
> faithfully returns 0.
>
> I've been able to recreate the above in DMC, with a 100% C-built Dll using the same .def file as the mydll sample.  The results are also the same when calling that dll from a host program written in MSVC6, DMD and DMC.  Unless I'm horribly mistaken on how to do compose dll's using Digitalmars' technology, this is what you get the instant you unload any DMC/DMD built dll.
>
> The problem seems to only manifest when a proper DllMain() method is exported from the library.  If none is provided, or if the given implementation can be optimized away, the error does not ocurr.
>
> This caused me hours upon hours of anguish trying to figure out why in my DSP/Mango server kept quietly halting on me: it was waiting for console input to terminate, but Mango's Stdin throws on EOF.  :(
>
> - Pragma
>
>



August 22, 2004
"pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in message news:cg5184$c7u$1@digitaldaemon.com...
> FYI: this is a long post, please read.  Thank you. :)
>
> In article <cg41tm$2u6a$1@digitaldaemon.com>, Matthew says...
> >
> >
> >"antiAlias" <fu@bar.com> wrote in message news:cg408a$2tge$1@digitaldaemon.com...
> >> I agree that #1 (the shared GC) is a primary issue. On the face of it, what
> >> Eric describes (below) sounds really attractive.
> >
> >It sounds promising. I'd like to hear more fine details, though.
>
> I'll work on a more detailed treatment as soon as time permits.  I'll have to first go back through my old source/notes and make sure that there's nothing that I've managed to miss.

Ok. This is an issue that will run and run, I think. At least for a month or two, or until it's sorted, at least.

> One thing I will say though: No matter how many times I revisit the problem, the businss of explicitly unloading a library at runtime has ugly consequences regardless of how the GC is hooked.

Can you give some more detailed examples. I'm an example kinda guy. :)

>  Also, I blame languages like Java and C#
> for raising our expectations of how this should be handled.  I don't think the
> problem can be completely solved for in-memory objects, without resorting to
> proxies and/or notification lists.
>
> >I'd also like to hear if his ideas would require changes to std.loader (as I'm itching for an excuse to rewrite that stillborn obscenity).
>
> I think my first reaction to reading that source was: "This compiled?" (j/k)

Can you be specific?

I'm troubled by its implementations for certain reasons, but I'd like to hear your unalloyed criticisms.

> I wouldn't anticipate need to change std.loader drastically.  Outside of that, I'd love to see the thread-local storage bits broken out into another library or as an extension to std.thread. ;)

Do you think we'll need to provide any additional mechanisms that will need to be standardised via std.loader? For example, GC-using D dlls might be required to export a specific entry point to facilitate GC-hooking, which would be built into std.loader.

> One question though: is all the extra ref-counting code needed for linux libraries, or does it supply some extra value to both Win32 and Linux environments?

Probably not for Linux, but I'm very cautious/cynical on this issue, as I've previously been bitten on some UNIXen that did not do internal ref-counting on dynamic libs. Hence, I think it's safest to do it ourselves.


> >I'd also like to hear Walter's opinions on the matter. I think this issue is of sufficient import to be a 1.0 prerequesite, don't you?
>
> I couldn't agree more.  I too, would like to hear what direction things will take with respect to dll's.

I've posted another, more detailed, response on this in this thread. I'd be keen to hear your ideas on that.

> >> Other than that, what about the exception-boundary issue?
> >
> >Well, D's going to have an ABI, and since all exceptions are going to co-exist memorywise (when the GC's linearised/homogenised), what's the prob?
>
> With respect to Kris' concern about handling exceptions, its really obnoxious how an uncaught exception within a library kills your app.  Aside from making all of my libraries behave like class-factories, I've taken to using methods that do this:

Yes, exceptions also need sorting from a multiple library point of view. As I say in the other post, I'm beginning to think that D link-units need to search each other out at runtime, and coordinate their activities somehow. We've just got to hope that it turns out to be less messy than COM or, heaven forefend, .NET!

> > int someMethod(int foo){ /* impl */ }
>
> ..to be called and exported by a dll stub function:
>
> > export(Windows) Exception someMethod(int foo,out _returnval){
> >   Exception _exception;
> >   try{ _returnval = someMethod(foo); }
> >   catch(Exception e){ _exception = e; }
> >   return(_exception);
> > }
>
> You can use a similar stub on the calling side that wraps the function pointer that you grab from the library.
>
> Meanwhile (on a slightly OT note), I'm still working on some of the more tricky aspects of dll's in D.  In particular, I've tracked one really nasty bug down to (what looks like) the digitalmars-C backend:
>
> > // deceptively simple example.
> > import std.c.windows.windows;
> > void main(){
> >   HMODULE mod = LoadLibrary("mydll.dll");
> >   printf("freeing...");
> >   FreeLibrary(mod);
> >   printf("You will never see this print.\n");
> > }
>
> (Matthew, Kris, have you had this problem on your Windows machines?  I've done this on two separate boxes with the same result.  If you two or anyone else can confirm this, I'll go ahead and post it to the bugs NG)

I confess I've done ~0 dynamic libs with D. :-)

> Basically, stdout and stdin are "automagically" set to EOF (or some similar
> state) after the call to FreeLibrary(): nothing goes in and nothing comes out.
> Everything else runs fine, no exceptions are thrown, and GetLastError()
> faithfully returns 0.
>
> I've been able to recreate the above in DMC, with a 100% C-built Dll using the same .def file as the mydll sample.  The results are also the same when calling that dll from a host program written in MSVC6, DMD and DMC.  Unless I'm horribly mistaken on how to do compose dll's using Digitalmars' technology, this is what you get the instant you unload any DMC/DMD built dll.
>
> The problem seems to only manifest when a proper DllMain() method is exported from the library.  If none is provided, or if the given implementation can be optimized away, the error does not ocurr.
>
> This caused me hours upon hours of anguish trying to figure out why in my DSP/Mango server kept quietly halting on me: it was waiting for console input to terminate, but Mango's Stdin throws on EOF.  :(
>
> - Pragma
>
>


August 22, 2004
Matthew;

As far as the GC is concerned, it would be wonderful to stuff that into a DLL ... that would solve a major problem right off the bat. Putting Phobos in a DLL would be a good idea also (no duplication of code across DLL instances).

If subsequent DLL's were loaded as collectable entities (wrapped by a class; or perhaps /as/ a class) then the GC could handle all the murky ref-counting in its own inimitable fashion (invoking the destructor when the DLL is no longer referenced). That would eliminate another sticky wicket; as Eric pointed out.

Those two get my immediate support; they would help tremendously with both Mango clustering and with the servlet-engine.

As to how one gains access to a class within the DLL ... I'd be happy with a simple and convenient means of accessing a factory method from the DLL, and going from there. Of course, what you are describing would be infinitely superior; but I'd be happy to wait for it if the primary issues were taken care of (shareable GC; DLLs as collectible entities).



"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:cg8kuk$2il8$1@digitaldaemon.com...
> I've read it once, but want to take some time to think before I respond to
some of the points.
>
> Let me pose my question in a more direct manner:
>
> AFAICS, one of the biggest problems is the ability to be able to use DLLs
with D in *all* of the following guises:
>
> 1. D exe, C dll. Communication by plain C-API function calls. All
resources returned whence they came.
> 2. C exe, D dll. Communication via extern(C) calls and/or interfaces with
well-known vtable layout (e.g. COM). All
> resources returned whence they came
> 3. D exe, D dll, Communication via extern(C) calls and/or interfaces with
well-known vtable layout (e.g. COM). All
> resources returned whence they came
> 4. D exe, D dll, Communication via extern(C) calls and/or interfaces with
well-known vtable layout (e.g. COM). Memory
> allocated hither and used safely thither
> 5. D exe, D dll, Communication via extern(C) calls and/or extern(D) calls
and/or interfaces with well-known vtable
> layout (e.g. COM). Memory allocated hither and used safely thither
> 6. D exe, D dll, Communication via 5. + classes. (Not sure of these
details.)
>
> Note, for each of the above, you can permute by replacing exe with D
client code inside a D dll in any of cases 2-6!
>
> Since D has, putatively at least, an ABI, all 6 (or 11) of these should,
in principle be realisable. Further,
> commercially speaking, D *must* provide all of these, to not be abandoned
by developers more influenced by deadline and
> pragmatism than idealism and performance - in other words, most of them.
>
> To my mind, we have two choices. One option is that the GC and supporting
infrastructure is *always* in some DCore.dll -
> i.e. there're no more statically linked programs - in the same vein as
.NET. The other option is that we come up with a
> mechanism whereby each link-unit is able to detect whether the process in
which it is loaded already has a GC and, if
> so, links to that, otherwise starting its own. The complication to this is
that if the first such link-unit is not the
> process' exe, we have the issue whereby the GC might be prematurely
unloaded while other link-units are still depending
> on it. This could, hopefully(!), be resolved be having dependent
link-units hold an independent dynamic-load reference
> on the GC's owning DLL.
>
> Thoughts?
>
> Walter, I'd be very interested in your thoughts on this, both on my
comments on the need for this commercially, and also
> on possible implementation mechanisms.
>
>
>
>
> "pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote
in message
> news:cg5184$c7u$1@digitaldaemon.com...
> > FYI: this is a long post, please read.  Thank you. :)
> >
> > In article <cg41tm$2u6a$1@digitaldaemon.com>, Matthew says...
> > >
> > >
> > >"antiAlias" <fu@bar.com> wrote in message
news:cg408a$2tge$1@digitaldaemon.com...
> > >> I agree that #1 (the shared GC) is a primary issue. On the face of
it, what
> > >> Eric describes (below) sounds really attractive.
> > >
> > >It sounds promising. I'd like to hear more fine details, though.
> >
> > I'll work on a more detailed treatment as soon as time permits.  I'll
have to
> > first go back through my old source/notes and make sure that there's
nothing
> > that I've managed to miss.
> >
> > One thing I will say though: No matter how many times I revisit the
problem, the
> > businss of explicitly unloading a library at runtime has ugly
consequences
> > regardless of how the GC is hooked.  Also, I blame languages like Java
and C#
> > for raising our expectations of how this should be handled.  I don't
think the
> > problem can be completely solved for in-memory objects, without
resorting to
> > proxies and/or notification lists.
> >
> > >I'd also like to hear if his ideas would require changes to std.loader
(as I'm itching for an excuse to rewrite that
> > >stillborn obscenity).
> >
> > I think my first reaction to reading that source was: "This compiled?"
(j/k)
> >
> > I wouldn't anticipate need to change std.loader drastically.  Outside of
that,
> > I'd love to see the thread-local storage bits broken out into another
library or
> > as an extension to std.thread. ;)
> >
> > One question though: is all the extra ref-counting code needed for linux libraries, or does it supply some extra value to both Win32 and Linux environments?
> >
> > >
> > >I'd also like to hear Walter's opinions on the matter. I think this
issue is of sufficient import to be a 1.0
> > >prerequesite, don't you?
> >
> > I couldn't agree more.  I too, would like to hear what direction things
will
> > take with respect to dll's.
> > >
> > >> Other than that, what about the exception-boundary issue?
> > >
> > >Well, D's going to have an ABI, and since all exceptions are going to
co-exist memorywise (when the GC's
> > >linearised/homogenised), what's the prob?
> >
> > With respect to Kris' concern about handling exceptions, its really
obnoxious
> > how an uncaught exception within a library kills your app.  Aside from
making
> > all of my libraries behave like class-factories, I've taken to using
methods
> > that do this:
> >
> > > int someMethod(int foo){ /* impl */ }
> >
> > ..to be called and exported by a dll stub function:
> >
> > > export(Windows) Exception someMethod(int foo,out _returnval){
> > >   Exception _exception;
> > >   try{ _returnval = someMethod(foo); }
> > >   catch(Exception e){ _exception = e; }
> > >   return(_exception);
> > > }
> >
> > You can use a similar stub on the calling side that wraps the function
pointer
> > that you grab from the library.
> >
> > Meanwhile (on a slightly OT note), I'm still working on some of the more
tricky
> > aspects of dll's in D.  In particular, I've tracked one really nasty bug
down to
> > (what looks like) the digitalmars-C backend:
> >
> > > // deceptively simple example.
> > > import std.c.windows.windows;
> > > void main(){
> > >   HMODULE mod = LoadLibrary("mydll.dll");
> > >   printf("freeing...");
> > >   FreeLibrary(mod);
> > >   printf("You will never see this print.\n");
> > > }
> >
> > (Matthew, Kris, have you had this problem on your Windows machines?
I've done
> > this on two separate boxes with the same result.  If you two or anyone
else can
> > confirm this, I'll go ahead and post it to the bugs NG)
> >
> > Basically, stdout and stdin are "automagically" set to EOF (or some
similar
> > state) after the call to FreeLibrary(): nothing goes in and nothing
comes out.
> > Everything else runs fine, no exceptions are thrown, and GetLastError()
> > faithfully returns 0.
> >
> > I've been able to recreate the above in DMC, with a 100% C-built Dll
using the
> > same .def file as the mydll sample.  The results are also the same when
calling
> > that dll from a host program written in MSVC6, DMD and DMC.  Unless I'm
horribly
> > mistaken on how to do compose dll's using Digitalmars' technology, this
is what
> > you get the instant you unload any DMC/DMD built dll.
> >
> > The problem seems to only manifest when a proper DllMain() method is
exported
> > from the library.  If none is provided, or if the given implementation
can be
> > optimized away, the error does not ocurr.
> >
> > This caused me hours upon hours of anguish trying to figure out why in
my
> > DSP/Mango server kept quietly halting on me: it was waiting for console
input to
> > terminate, but Mango's Stdin throws on EOF.  :(
> >
> > - Pragma
> >
> >
>
>


August 22, 2004
In article <cg8png$2ltl$1@digitaldaemon.com>, Matthew says...
>
>"pragma" <EricAnderton at yahoo dot compragma_member@pathlink.com> wrote in
>> One thing I will say though: No matter how many times I revisit the problem, the businss of explicitly unloading a library at runtime has ugly consequences regardless of how the GC is hooked.
>
>Can you give some more detailed examples. I'm an example kinda guy. :)

I'll take an oversimplified example from my D-Servlet Pages project.

void foo(Request req,Response res){
// ServletLibrary proxies all the exports from the dll
//   and exposes some extra housekeeping functions
ServletLibrary library = getServletLibrary("mylib.dll");

// library is queried for a servlet.  The servlet is created inside the dll DSPServlet servlet = library.getServlet();

// Do some work (generate HTML, and so forth)
servlet.service(req,res);

// This frees the library.  It is now no longer in memory
library.unload();

// "servlet" is now invalid (see below)
}

No matter which side has the responsibility for performing GC operations and memory allocations, the V-table for any object always lives where it was created.  This application in particular rebuilds dlls on demand, so references to servlets can be invalidated at a drop of a hat, across multiple threads.

>> I think my first reaction to reading that source was: "This compiled?" (j/k)
>
>Can you be specific?

Honestly, it comes down to a matter of coding style: I'm just not used to reading your code. :)

You say poh-tay-toh I say po-tot-oh...

But I understand why its organized the way it is, and I feel its a very solid lib.  The only thing I'd have done differently would be to roll HXModule into a non-auto class that handles all the refcounting, while wrapping the linux functions with Win32-like functions (or the other way around), so there's less code in there.

>
>I'm troubled by its implementations for certain reasons, but I'd like to hear your unalloyed criticisms.
>
>> I wouldn't anticipate need to change std.loader drastically.  Outside of that, I'd love to see the thread-local storage bits broken out into another library or as an extension to std.thread. ;)
>
>Do you think we'll need to provide any additional mechanisms that will need to be standardised via std.loader? For example, GC-using D dlls might be required to export a specific entry point to facilitate GC-hooking, which would be built into std.loader.

If we're going to go with hooking rather than a GCCore.dll-type solution (from your other post, I like that idea), then yea, that would be best served internally by loader.

A non-auto class (DllModule maybe?) would be a nice touch.  Also an 'event' (I
guess a delegate[] in D is close enough) that is called when a dll is finally
unloaded (refcount goes to zero), would be a *very* nice addition.

>Yes, exceptions also need sorting from a multiple library point of view. As I say in the other post, I'm beginning to think that D link-units need to search each other out at runtime, and coordinate their activities somehow. We've just got to hope that it turns out to be less messy than COM or, heaven forefend, .NET!

>I confess I've done ~0 dynamic libs with D. :-)

Hehe.. we'll fix that. ;)

Walter has made it easy as sin to create .dll's in D.  What's missing is a D-standard stub that is /implicitly/ added to a library along with classinfo, exception piping, GC hooking, etc etc... one becomes painfully aware of all this only *after* you start working with dll's heavily. Its even worse coming from using Java's classloader or .NET assemblies; D has a very long way to go in this department.

- Pragma


August 22, 2004
In article <cg8kut$2ilc$1@digitaldaemon.com>, Matthew says...
>
>To my mind, we have two choices. One option is that the GC and supporting infrastructure is *always* in some DCore.dll - i.e. there're no more statically linked programs - in the same vein as .NET. The other option is that we come up with a mechanism whereby each link-unit is able to detect whether the process in which it is loaded already has a GC and, if so, links to that, otherwise starting its own. The complication to this is that if the first such link-unit is not the process' exe, we have the issue whereby the GC might be prematurely unloaded while other link-units are still depending on it. This could, hopefully(!), be resolved be having dependent link-units hold an independent dynamic-load reference on the GC's owning DLL.
>
>Thoughts?

I like the idea, but it represents quite a paradigm shift for D.  Also, any time you introduce a platform library, you have the messy consequences of library versioning and which library instance (if multiple are present) is used on the search path.

- Pragma


1 2
Next ›   Last »