"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:csse2a$1jk2$1@digitaldaemon.com...
> > Why not use D interfaces?
>
> But how do we manage the memory that may be exchanged between link units?
Which GC owns it? I confess I little
> understand this, but it seems pretty clear that if GCs are link
unit-specific, then there're going to be problems.

GC's own memory that is allocated from the pool it controls. A gc will not free a pointer unless it is pointing into its own pool. Next, a gc looks for pointers by examining its "roots". The roots include the stack and the static data owned by the dll. Other roots can be manually added to the gc's scan list through the gc API.

The easy answer is maintain a pointer to the allocated memory in the DLL that allocated it, in one of the roots that are scanned by its gc.

"Kris" <Kris_member@pathlink.com> wrote in message news:csse93$1jq1$1@digitaldaemon.com...
> >Why not use D interfaces?
>
> It would be great to use interfaces. I presume you mean "in conjunction
with a
> Factory Pattern" ?
>
> Surely the obvious problem is this, Walter:
>
> 1) the factory (within the DLL) would create the implementing class, and
hand it
> back to the caller
> 2) the caller now hangs onto the returned interface handle
> 3) the DLL GC thinks there are no more references to the constructed
class, and
> reaps it

How the gc works is it scans regions for roots. If any those roots point into the gc's own memory pool, the associated allocated memory is not deleted. So the solution here is to add the ranges of the caller where the handle may reside to the DLL's gc. The stack is already there, because the DLL and the caller share the same stack.

Another way is to use the COM style AddRef and Release.

> 4) the other GC (from the caller) may try to reap the interface handle,
which
> does not belong within its pools

This won't happen because the gc won't reap memory which is not in its own pool.

> There are many more, non-trivial, examples.
>
> Since we're talking about the factory pattern, one might consider adding a version number as an argument? So the client could specify some kind of
expected
> version? Perhaps not ideal, but it might address some of your concerns?

COM uses the guid approach for this. It hasn't worked out as well in practice as was initially hoped.

"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:cssdt2$1jea$1@digitaldaemon.com...
> But I agree with your implication of the importance of this issue to D. As
I've said three times in the last 24hrs,
> without an effective and elegant (i.e. totally ignorable to 98% of users)
solution, D will be stillborn. Of that I have
> no doubt. Hence, I agree with Kris that Walter needs to consider this to
be a serious issue, and a mandatory pre-1.0
> accomplishment.

Memory management is never totally ignorable, in any language. And I know of no DLLs in any language where one can totally ignore memory management and not get into nasty trouble.

"Matthew" <Matthew_member@pathlink.com> wrote in message news:csv2tk$234l$1@digitaldaemon.com...
> In article <csuu5g$1uc1$1@digitaldaemon.com>, Walter says...
> >I guess I'm trying to point out that C doesn't solve the problem -
dealing
> >with it requires an understanding about how memory allocation works. The same goes for D. In D, a gc will only recover a pointer *that it can find the root for* AND *that it has allocated*. All pretty much follows from these two key points. If you're passing gc allocated memory between DLLs, you'll need to think about if and where a root for that pointer exists,
and
> >which gc will be looking at that root.
>
> My point is that languages/runtime environments such as Java and .NET -
*which
> use garbage collection* - provide exactly this guarantee.

I don't know about .net, but if you're hooking DLLs up to Java, you have to understand how Java's gc works and adapt to it. You cannot ignore the issue. The "guarantee" is provided by the implementor of the DLL conforming to the rules of Java's gc, which are non-trivial.

> Since GC memory is a
> big feature of D, then it's eminently reasonable for users who are
attracted to
> D for this feature will expect that cross-module memory issues will not
exist,
> i.e. 'D' memory will be like it is in Java and .NET.

Those cross module memory issues do exist in Java. I've written a Java gc, and I've written DLLs to hook into Java.

> Conversely, the situation in C and C++ is that one *knows* to be careful
with
> *all* memory, and so one is. But because we'll be used to GC nonchalence
inside
> the (D) implementation of our DLLs, it's very likely that will carry over
to our
> use/implementation of the module boundaries.
>
> In my opinion, this facet of D is a strong disadvantage and, if left as it
is,
> will cause large-scale dismissal of D as a serious language for anything
other
> than statically linked (and therefore trivial) applications.
>
> D must provide memory transparency like that of Java/.NET, at least when
used
> between different D link-units.

Perhaps Java has changed in the last few years, but I don't recall Java behaving that way with DLLs. The Java vm itself doesn't hook up to other Java code using DLLs. It hooks up to DLLs using the "java native interface" which has a carefully described protocol on how to handle pointers and memory allocation - and that interface is a C interface.

It's a lot easier to do a DLL to hook into D than one to hook into Java.

"Walter" <newshound@digitalmars.com> wrote in message news:ct029h$5s8$1@digitaldaemon.com...
>
> "Matthew" <Matthew_member@pathlink.com> wrote in message news:csv2tk$234l$1@digitaldaemon.com...
>> In article <csuu5g$1uc1$1@digitaldaemon.com>, Walter says...
>> >I guess I'm trying to point out that C doesn't solve the problem -
> dealing
>> >with it requires an understanding about how memory allocation works. The same goes for D. In D, a gc will only recover a pointer *that it can find the root for* AND *that it has allocated*. All pretty much follows from these two key points. If you're passing gc allocated memory between DLLs, you'll need to think about if and where a root for that pointer exists,
> and
>> >which gc will be looking at that root.
>>
>> My point is that languages/runtime environments such as Java and .NET -
> *which
>> use garbage collection* - provide exactly this guarantee.
>
> I don't know about .net, but if you're hooking DLLs up to Java, you have to understand how Java's gc works and adapt to it. You cannot ignore the issue. The "guarantee" is provided by the implementor of the DLL conforming to the rules of Java's gc, which are non-trivial.
>
>> Since GC memory is a
>> big feature of D, then it's eminently reasonable for users who are
> attracted to
>> D for this feature will expect that cross-module memory issues will not
> exist,
>> i.e. 'D' memory will be like it is in Java and .NET.
>
> Those cross module memory issues do exist in Java. I've written a Java gc, and I've written DLLs to hook into Java.
>
>> Conversely, the situation in C and C++ is that one *knows* to be careful
> with
>> *all* memory, and so one is. But because we'll be used to GC nonchalence
> inside
>> the (D) implementation of our DLLs, it's very likely that will carry over
> to our
>> use/implementation of the module boundaries.
>>
>> In my opinion, this facet of D is a strong disadvantage and, if left as it
> is,
>> will cause large-scale dismissal of D as a serious language for anything
> other
>> than statically linked (and therefore trivial) applications.
>>
>> D must provide memory transparency like that of Java/.NET, at least when
> used
>> between different D link-units.
>
> Perhaps Java has changed in the last few years, but I don't recall Java behaving that way with DLLs. The Java vm itself doesn't hook up to other Java code using DLLs. It hooks up to DLLs using the "java native interface" which has a carefully described protocol on how to handle pointers and memory allocation - and that interface is a C interface.


Come on. Clearly I'm not talking about Windows .DLL DLLs. I'm talking about dynamically loaded executable objects, which, for D will likely be in Windows .DLLs and UNIX .sos.

In Java, such a thing is a jar and/or class file. Are you telling me that I cannot pass objects created in one class file to another? Do all classes in Java communicate only by fundamental types? I don't think so.

In .NET, such a thing is an assembly. Are you telling me that I cannot pass objects created in one assembly to another? Do all classes in .NET communicate only by fundamental types? I'm pretty sure they do not.

> It's a lot easier to do a DLL to hook into D than one to hook into Java.

That's just not true (since I wasn't talking about JNI/native DLLs). And it worries me that you think it's reasonable to say so. Do you really not care to engage the massive (and, with modernity, growing) sub-set of software development/engineers that need to do this kind of engineering? I cannot fathom the attitude.

Matthew wrote:
> Come on. Clearly I'm not talking about Windows .DLL DLLs. I'm talking about dynamically loaded executable objects, which, for D will likely be in Windows .DLLs and UNIX .sos.
> 
> In Java, such a thing is a jar and/or class file. Are you telling me that I cannot pass objects created in one class file to another? Do all classes in Java communicate only by fundamental types? I don't think so.
> 
> In .NET, such a thing is an assembly. Are you telling me that I cannot pass objects created in one assembly to another? Do all classes in .NET communicate only by fundamental types? I'm pretty sure they do not.
> 
>> It's a lot easier to do a DLL to hook into D than one to hook into Java.
> 
> That's just not true (since I wasn't talking about JNI/native DLLs). And it worries me that you think it's reasonable to say so. Do you really not care to engage the massive (and, with modernity, growing) sub-set of software development/engineers that need to do this kind of engineering? I cannot fathom the attitude.

Now, are you talking about mixing languages in dynamically linked libraries or about staying within one language?

In the first case, the problem is mostly language independent: linking D code with C code and passing around objects gives you about the same kind of troubles as linking a native library into a Java program or vice versa.

Mixing a bunch of Java libraries is unproblematic, as is mixing various D libraries (as long as you don't explicitely fiddle with the memory allocation by replacing the allocator in one of the libraries.)

(Of course, with Java, you can also use other languages targeting the same virtual machine, but that would be similar of using the D libraries for memory management of foreign code.)

In any case, I still don't see where the problem is intrinsically linked with the issue of dynamic libraries. Statically linked libraries collide in the same manner when they use different mechanisms for memory management.

Matthew wrote:
> Sounding more and more normal. ;)
> 
> But I agree with your implication of the importance of this issue to D. As I've said three times in the last 24hrs, without an effective and elegant (i.e. totally ignorable to 98% of users) solution, D will be stillborn. Of that I have no doubt. Hence, I agree with Kris that Walter needs to consider this to be a serious issue, and a mandatory pre-1.0 accomplishment.

If D-made DLLs were only used from D-made programs, then we'd have a lot more freedom.

Until the OS does provide gc, the dll has to manage its own gc, at least when called from a non-D program.

Besides, DLLs written in non-gc languages manage their own memory, too.

(OT: Gee, I wish there was a "standalone gc" OSS project, having a well designed api, and robust, well known implementations for the most used OSs! A Pipe Dream, I know.)

I am afraid fixing this issue may be so hard that D1.0 can't wait for it. Maybe D1.5 or something, but please, not 1.0.

With the unfathomable experience Walter has, an accurate intuition follows. That's why he might not be able to refute exactly every attack with fact-by-fact analysis. Not to mention the time it would take away from actually developing D.

One solution would be a non-standard, for D-use-only DDL (dynamic D library  :-) that uses the main program's gc, and solves whatever other issues on the side. But this would not be much better than the ancient CP/M Program Overlays.

I still don't understand: where is the problem with a dynamically linked gc like the Boehm GC? Every D-created DLL would just link to that one and use it for its memory management.

As long as you link together only D libraries, each one will link to the same GC, so there exists only one GC handling everything.

If you mix D libraries with foreign libraries, the foreign libraries could either use the same GC (which should have a C interface like every DLL) or they ignore it, doing their own memory management.

In the last case, danger only arises when you pass around pointers between GC and non-GC code. That case, of course, is always tricky, but that does not change at all whether you have dynamic or static linking.

Ciao,
Norbert


Georg Wrede wrote:

> Matthew wrote:
>> Sounding more and more normal. ;)
>> 
>> But I agree with your implication of the importance of this issue to D. As I've said three times in the last 24hrs, without an effective and elegant (i.e. totally ignorable to 98% of users) solution, D will be stillborn. Of that I have no doubt. Hence, I agree with Kris that Walter needs to consider this to be a serious issue, and a mandatory pre-1.0 accomplishment.
> 
> If D-made DLLs were only used from D-made programs, then we'd have a lot more freedom.
> 
> Until the OS does provide gc, the dll has to manage its own gc, at least when called from a non-D program.
> 
> Besides, DLLs written in non-gc languages manage their own memory, too.
> 
> (OT: Gee, I wish there was a "standalone gc" OSS project, having a well designed api, and robust, well known implementations for the most used OSs! A Pipe Dream, I know.)
> 
> I am afraid fixing this issue may be so hard that D1.0 can't wait for it. Maybe D1.5 or something, but please, not 1.0.
> 
> With the unfathomable experience Walter has, an accurate intuition follows. That's why he might not be able to refute exactly every attack with fact-by-fact analysis. Not to mention the time it would take away from actually developing D.
> 
> One solution would be a non-standard, for D-use-only DDL (dynamic D library  :-) that uses the main program's gc, and solves whatever other issues on the side. But this would not be much better than the ancient CP/M Program Overlays.

Norbert Nemec wrote:
> I still don't understand: where is the problem with a dynamically linked gc
> like the Boehm GC? Every D-created DLL would just link to that one and use
> it for its memory management.
> 
> As long as you link together only D libraries, each one will link to the
> same GC, so there exists only one GC handling everything.
> 
> If you mix D libraries with foreign libraries, the foreign libraries could
> either use the same GC (which should have a C interface like every DLL) or
> they ignore it, doing their own memory management.
> 
> In the last case, danger only arises when you pass around pointers between
> GC and non-GC code. That case, of course, is always tricky, but that does
> not change at all whether you have dynamic or static linking.
> 
> Ciao,
> Norbert

Exactly, Norbert.

Unfortunately, D internalizes specific kinds of memory allocations: adjusting the length of an array, for example.

What we're effectively asking for is for D to expose all such behavior, such that the GC can be treated as a third-, and shareable, party. Just as you suggest.

Additionally, the static-data extents of each shared-lib need to be registered with the GC as potential roots (in the same manner as statically-linked code). The latter is really a job for the compiler, although it could perhaps be done manually?

Walter is apparently concerned that exposing the GC as an externalized entity (a shared-lib) will lead to all kinds of versioning conflict. Yet the upside is that MM would become trivial, robust, comprehensive, and comprehendable for component-based designs (for D code) -- one could immediately build sophisticated, dynamic systems.

The alternative (what we have now with multiple GC instances) leads to incredibly pedantic management of multiple heaps, by the programmer. This is actually much worse than malloc/free, since the multiple GC instances effectively fight against you. And that's without considering the cognitive model-change, switching from a managed-memory environment (statically-linked D) to one that could be termed "disruptive memory management".

Pragma ran into some truly hideous problems with his DSP project, doing just this kind of thing. I've held off doing large scale grid-style processing (via mango.cluster) for many months, waiting for this to be resolved. Such issues are truly holding D back. Sean has taken the initiative, and has managed to pry the GC away from DMD such that it /can/ actually be located in a shared-lib. All we need is for the compiler to generate static-data roots for DLLs and so on ...

Theoretically speaking, Walter is correct in saying the programmer can manage all this by themselves. However, that argument is no different that one calling for the removal of GC altogether. Therein lies the conflict: to do what Walter proposes would actually be considerably simpler, and more robust, if the GC did not exist at all, and we all went back to explicitly managing heap allocations.

Again; the only reason for the pushback on this is Walter's concern over what he terms "DLL hell". Yes, that is perhaps of some concern. Yet, let's put that into perspective: exactly how often would this happen with a GC shared-lib? And, couldn't the "DLL hell" be managed for this one specific case? I think it can. And when you consider the downside of not doing it, one simply cannot fathom the pushback.

Walter wrote:
> "Matthew" <admin.hat@stlsoft.dot.org> wrote in message
> news:cssdt2$1jea$1@digitaldaemon.com...
> 
>>But I agree with your implication of the importance of this issue to D. As
> 
> I've said three times in the last 24hrs,
> 
>>without an effective and elegant (i.e. totally ignorable to 98% of users)
> 
> solution, D will be stillborn. Of that I have
> 
>>no doubt. Hence, I agree with Kris that Walter needs to consider this to
> 
> be a serious issue, and a mandatory pre-1.0
> 
>>accomplishment.
> 
> 
> Memory management is never totally ignorable, in any language. And I know of
> no DLLs in any language where one can totally ignore memory management and
> not get into nasty trouble.
> 
> 
With respect, Walter, that kind of rebuke is somewhat pedantic and seemingly irrational.

I'd like to ask you to consider the field here:

1) dynamically (non-statically) linked systems are a reality, and they are important. Please put aside the specifics of MS DLLs, and think about what you can do with such systems: grid-computing, agent-based systems, the lofty goals of ubiquitous computing, even a lowly web-server.

2) D, like other moderns languages, deliberately removes the need to explicitly manage heap allocation. Yet to build said dynamic systems in D, one must revert to C-style heap management and, at the same time, both understand and compensate for every little nuance of the particular GC itself. There is a pre-requisite for uber-expertise before embarking on such a trail, and the maintainence thereof. These are surely blatant contradictions for a language such as D?

3) allowing for one, per-process, GC cleanly and elegantly resolves the problems of (2) and thereby opens the doors of (1) to D.

4) one approach to implement a per-process GC is to place it into its own shared-lib. You have noted some concerns about "DLL hell" for such an implementation.


Three questions arise:

a) do you feel there's not much value in anything but statically-linked systems? If yes, then I will understand you position much more clearly.

b) assuming "no" to (a), is there another means of implementing a single per-process GC, without placing it into a shared-lib?

c) please consider the serious difficulties exposed to the programmer when dealing with (and maintaining) the fallout from multpile GC instances, and balance that against the likelyhood of "DLL hell" specifically for a D GC shared-lib instance. Please also consider if there's a means of implementing that specific shared-lib such that any potential for the latter would be further minimized, or completely eliminated.

Thanks;