September 13, 2005
I've a long (10 years) experience in C++ programming, but now I'm at the point "fed up": I can't take it any more. In my experience C++ is nice for self contained, monolithic programs, but when it comes to write large scale, modularized, extensible software there are so many things to deal with, that one is programming more infrastructure than actual worker code.

I was looking for some other languages that would produce native code and don't rely on special libraries. Among others e.g. Objective-C I also ran over D a couple of months ago. Now it's time to put the code snippets together I've developed in the last years to form a 3D engine. And here come my questions:

=== Binary Modules ===
I played around with D a bit, but so long it just created
standard .o files (no problem there). However for my engine I
created a component system, that already can glue various
scripting languages together, but I failed with exporting C++;
it's just to complicated. One special feature of the component
system is, that it has it's own dynamic loader, that uses it's
own binary format ECM (EVEN Component Module, EVEN is the name
of my engine), being ELF with some add ons, e.g. the custom
format can store not only relocation data and identifiers, but
also metadata and type dependencies. E.g. you you've got a class
foo in module foomod.ecm derived from class bar on barmod.ecs in
foomod.ecs there is stored a dependency on a class bar with a
certain signature. It may also store a link to the module file,
but the default behaviour of ECS is to index all avaliable
modules on startup and load an module with an apropriate class
on demand. If you've deriving classes naturally the exact class
must be matched, but it is also possible to instanciate by
interface. The main reason I'm having my own dynamic linker is,
that I want to use the same binary with different operating
system platforms (of course not CPU), so that only plattform
dependent modules must be provided redundant. As a speciality
ECS allows tagging multimodules, which means a module can
contain multiple implementations, and which one is loaded
depends on the tags it carries, e.g. on which operating system
it is executed.

Well, that's the theory and it works quite well with the pure procedural exports I did from C and a small subset of C++ by providing a separate IDL, but I'm in huge trouble with templates, multiple inheritance. You know what I mean. The problem is, that a lot of C++ standard types are based around templates.

In how far is D better suited to create such binary modules? To create a basic module my module builder/linker needs information about the exported classes, interfaces, and functions plus their Metadata.

=== Metadata ===
An important aspect of my component system is, that it chooses
class implementations, interfaces and function on thir
relationship and metadata. Metadata is of which types a value is
of (it also supports generic variant types) and what values an
object carries or a function takes and returns. Can D provide
this metadata on
- runtime
- compile time
?

=== Memory Management ===
As I read D has it's own memory management. Is it possible, that
I replace the builtin memory management with my own?

Please be patient with me, I installed D on my development system only a few hours ago and need some orientation. Usually I can need only 2 weeks to learn a new programming language, but learning requires asking - sometimes dumb - questions.

-- 
Wolfgang Draxinger

September 14, 2005
Wolfgang Draxinger wrote:
> I've a long (10 years) experience in C++ programming, but now I'm
> at the point "fed up": I can't take it any more. In my
> experience C++ is nice for self contained, monolithic programs,
> but when it comes to write large scale, modularized, extensible
> software there are so many things to deal with, that one is
> programming more infrastructure than actual worker code.
> 
> I was looking for some other languages that would produce native
> code and don't rely on special libraries. Among others e.g.
> Objective-C I also ran over D a couple of months ago. Now it's
> time to put the code snippets together I've developed in the
> last years to form a 3D engine. And here come my questions:
> 
> === Binary Modules ===
> I played around with D a bit, but so long it just created
> standard .o files (no problem there). However for my engine I
> created a component system, that already can glue various
> scripting languages together, but I failed with exporting C++;
> it's just to complicated. One special feature of the component
> system is, that it has it's own dynamic loader, that uses it's
> own binary format ECM (EVEN Component Module, EVEN is the name
> of my engine), being ELF with some add ons, e.g. the custom
> format can store not only relocation data and identifiers, but
> also metadata and type dependencies. E.g. you you've got a class
> foo in module foomod.ecm derived from class bar on barmod.ecs in
> foomod.ecs there is stored a dependency on a class bar with a
> certain signature. It may also store a link to the module file,
> but the default behaviour of ECS is to index all avaliable
> modules on startup and load an module with an apropriate class
> on demand. If you've deriving classes naturally the exact class
> must be matched, but it is also possible to instanciate by
> interface. The main reason I'm having my own dynamic linker is,
> that I want to use the same binary with different operating
> system platforms (of course not CPU), so that only plattform
> dependent modules must be provided redundant. As a speciality
> ECS allows tagging multimodules, which means a module can
> contain multiple implementations, and which one is loaded
> depends on the tags it carries, e.g. on which operating system
> it is executed.
> 
> Well, that's the theory and it works quite well with the pure
> procedural exports I did from C and a small subset of C++ by
> providing a separate IDL, but I'm in huge trouble with
> templates, multiple inheritance. You know what I mean. The
> problem is, that a lot of C++ standard types are based around
> templates.
> 
> In how far is D better suited to create such binary modules? To
> create a basic module my module builder/linker needs information
> about the exported classes, interfaces, and functions plus their
> Metadata.

Wow.  You certainly know how to get a developer's attention. :)

I'm presently working on a runtime loader and linker architecture for D.

http://www.dsource.org/projects/ddl
http://www.dsource.org/forums/http://www.dsource.org/forums/viewforum.php?f=70

The theory and goals behind DDL are *frighteningly* close to what you've done with EVEN.  Intermediate files in D (.obj files under windows for example) contain the needed symbolic information to determine module-level dependencies.  Since modules map one-to-one with D intermediate files, it suits itself very well to runtime loading and linking.

DDL presently digests OMF formatted .lib and .obj files.  COFF and ELF support are planned for the very near future.  DDL also supports its own module type, which is really a header/wrapper for any other supported type; its intended to help make module searches and linking a bit faster as well as provide a hook for versioning and metadata.  AFAIK, this project won't have any hangups with resolving templates and the rest since we're merely emulating how the compile-time linker works.

Please take a look through the forums for the project and ask me any questions you like.  Also, if there's any features of ECN that you think would work well for DDL, please don't hesitate to post about it or contact me personally via email. :)

... but until this project is complete, D is stuck with monolithic builds, shared libraries and dll files.

> 
> === Metadata ===
> An important aspect of my component system is, that it chooses
> class implementations, interfaces and function on thir
> relationship and metadata. Metadata is of which types a value is
> of (it also supports generic variant types) and what values an
> object carries or a function takes and returns. Can D provide
> this metadata on
> - runtime
> - compile time
> ?

Presently, only at the module level.  D symbols are encoded with namespace and type information intact:

For example, the method "char[] foobar(uint[] a)" is exported as the symbol "_D6foobarFAkZAa".  Currently, DDL can translate the name-mangling into the readable form, but it does not generate reflection information... yet.

D lacks a reflection interface, but here are movements about to bring to the table a solid reccomendation for Walter to implement in the final specification.

Also, while we're on the topic, there is no runtime emit interface either (not that it can't be done of course, just that nobody's written it yet).

> 
> === Memory Management ===
> As I read D has it's own memory management. Is it possible, that
> I replace the builtin memory management with my own?

Yes.  The garbage collector is implemented in the platform library (phobos) itself.  Its quite easy to implement changes, provided that a) you know what you're doing and b) you maintain the needed hooks to satisfy the current ABI.  However, copying and generational GC's will not work with the current incarnation of D, due to its design.

> Please be patient with me, I installed D on my development system
> only a few hours ago and need some orientation. Usually I can
> need only 2 weeks to learn a new programming language, but
> learning requires asking - sometimes dumb - questions.

Welcome to the fold.

-- 
- EricAnderton at yahoo
September 19, 2005
pragma wrote:
> ...
> However, copying and generational GC's will not work with the current incarnation of D, due to its design.
> 
Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.

-- 
Bruno Medeiros
Computer Science/Engineering student
September 22, 2005
Bruno Medeiros wrote:
> pragma wrote:
> 
>> ...
>> However, copying and generational GC's will not work with the current incarnation of D, due to its design.
>>
> Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.
> 

Its true.  The problem is that with any GC that implements moving of memory blocks (generational, copying or compacting), it needs to be able to work around the fact that D program code currently works in raw memory addresses.  Simply moving a block of memory would be disasterous as you'd have to tackle any number of outstanding pointers as well.

Now, you could hack this in by modifying the GC to scan everything and rewrite pointers in all managed memory as needed, but there are tons of corner-cases in this strategy.  Plus, the GC doesn't really keep track of type information, so there's no way to know if a given dword really contains a pointer or an integer.  D is modestly coupled to the conservative nature of mark-sweep, so this normally isn't a problem; so the language would need some way to make feeding this info to the GC transparent.

Alternately, one could redefine references as abstract handles to managed memory, which makes the GC an intermediary for just about everything; a strategy that requires additional engineering at the compiler level to work in to D.  This would also mean complicating interop with legacy C code, again something that requires more programming (probably just in phobos) to get working.

There are probably other approaches out there, but I'd wager that they would all benefit from tweaking D this way or that to work seamlessly.

-- 
- EricAnderton at yahoo
September 27, 2005
pragma wrote:
> Bruno Medeiros wrote:
> 
>> pragma wrote:
>>
>>> ...
>>> However, copying and generational GC's will not work with the current incarnation of D, due to its design.
>>>
>> Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.
>>
> 
Hum, are you sure you understood my question? Or maybe I misunderstood your original statement.
You said (I think) that with D's design one could not implement a copying/moving (whether compacting, generational, whatever) garbage collector. If this is what you were saying, then I don't understand your reply. The language spec of D, namely the "Garbage Collection" section ( http://www.digitalmars.com/d/garbage.html ), specifies restrictions on pointer usage to allow "maximum flexibility in garbage collector design", in particular copying collectors. It even explicitly mentions them in that page ( "A compacting garbage collector may change this value" , "A copying garbage collector can arbitrarily move objects around in memory, thus ..." ) .
Yes, to actually implement one such collector one would need type information, and would also need to change pointer values, but were is the unfeasability in that?

> Its true.  The problem is that with any GC that implements moving of memory blocks (generational, copying or compacting), it needs to be able to work around the fact that D program code currently works in raw memory addresses.  Simply moving a block of memory would be disasterous as you'd have to tackle any number of outstanding pointers as well.
> 
Any copying/moving collector (of any supporting language) has that issue, and they deal with it. Why is it a problem for D? What do you mean by "D program code currently works in raw memory addresses." ?

> Now, you could hack this in by modifying the GC to scan everything and rewrite pointers in all managed memory as needed, but there are tons of corner-cases in this strategy.  Plus, the GC doesn't really keep track of type information, so there's no way to know if a given dword really contains a pointer or an integer.  D is modestly coupled to the conservative nature of mark-sweep, so this normally isn't a problem; so the language would need some way to make feeding this info to the GC transparent.
> 
The current GC doesn't keep track of type information, but there is no design issue that would impossibilitate (or difficultate) that, much to the contrary. Or is it?..

-- 
Bruno Medeiros
Computer Science/Engineering student
September 27, 2005
In article <dhbvq9$ma8$1@digitaldaemon.com>, Bruno Medeiros says...
>
>pragma wrote:
>> Bruno Medeiros wrote:
>> 
>>> pragma wrote:
>>>
>>>> ...
>>>> However, copying and generational GC's will not work with the current
>>>> incarnation of D, due to its design.
>>>>
>>> Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.
>>>
>> 
>Hum, are you sure you understood my question? Or maybe I misunderstood your original statement.

Okay, I may need to clarify myself a little bit here.  :)

D's "design" is actually enshrined within the reference implementation (DMD) in addition to what is on the website.  Having used D through literally over 50 revisions myself, I'm pretty accustomed to confusing the two.  Earlier in this thread, I used the word 'design' in a few spots where 'implementation' (or "D's implementation as DMD") may have been more appropriate.  Sorry for the confusion.

You're right, it's not unfeasable for *D* (as a language) to use a copying collector, but the current design/implementation simply won't work 100% as well as it does against the current GC due to how code is generated.  That was really the main thrust of the point I tried to make earlier.

>You said (I think) that with D's design one could not implement a copying/moving (whether compacting, generational, whatever) garbage collector. If this is what you were saying, then I don't understand your reply. The language spec of D, namely the "Garbage Collection" section ( http://www.digitalmars.com/d/garbage.html ), specifies restrictions on pointer usage to allow "maximum flexibility in garbage collector design", in particular copying collectors. It even explicitly mentions them in that page ( "A compacting garbage collector may change this value" , "A copying garbage collector can arbitrarily move objects around in memory, thus ..." ) .

Again, my apologies for the design/implementation thing.

As far as I understand, Walter has made many steps to keep the door open for copying/moving GC types; hence the references to copying GC's in the documentation.

It has been mentioned in this newsgroup before, it's not 100% ready for a copying/moving GC yet.  For example, the toHash() method of Object is presently implemented as using the memory location for the object; it would need to be changed.

>Yes, to actually implement one such collector one would need type information, and would also need to change pointer values, but were is the unfeasability in that?

Its not unfeasable at all, its just that DMD isn't doing this yet, nor is this anywhere in the specification.  Again, this is just one of the potential changes to D's design I was referring to. :)

>> Its true.  The problem is that with any GC that implements moving of memory blocks (generational, copying or compacting), it needs to be able to work around the fact that D program code currently works in raw memory addresses.  Simply moving a block of memory would be disasterous as you'd have to tackle any number of outstanding pointers as well.
>> 
>Any copying/moving collector (of any supporting language) has that issue, and they deal with it. Why is it a problem for D? What do you mean by "D program code currently works in raw memory addresses." ?

Well, its probably old hat, but new() returns the actual address in memory on
the heap just like malloc() does.  So, right now, one can expect the following
to work:

Object foo = cast(Object)cast(void*)(new Object);

Nice and easy: new returns a pointer which can be casted about.  Regardless of type, that pointer is really the same value.

Now, imagine you've changed the underlying implementation of a reference to be a GC handle, rather than a heap memory address like it is now.  Now you have a problem: the cast to and from void* no longer has such a simple behavior. Regardless of what path you take from here, you are going to change the behavior/design of the language.

>> Now, you could hack this in by modifying the GC to scan everything and rewrite pointers in all managed memory as needed, but there are tons of corner-cases in this strategy.  Plus, the GC doesn't really keep track of type information, so there's no way to know if a given dword really contains a pointer or an integer.  D is modestly coupled to the conservative nature of mark-sweep, so this normally isn't a problem; so the language would need some way to make feeding this info to the GC transparent.
>> 
>The current GC doesn't keep track of type information, but there is no design issue that would impossibilitate (or difficultate) that, much to the contrary. Or is it?..

You're right, there isn't anything that would prevent that.  In fact, someone already submitted to the newsgroup here a prototype mark/sweep GC that uses typeinfo hints to increase efficency (basically prevents scans of uint[] arrays and things of that nature).  But like everything that would make an efficent and useful copying/moving GC, it's not here yet.

I hope this clears up my position. :)  Again, I'd don't think a moving/copying GC is impossible for D, but any such solution approached without some changes to the compiler/spec/design is going to be very far removed from optimal, if at all useful.

- EricAnderton at yahoo
September 27, 2005
In article <dhc4la$15ee$1@digitaldaemon.com>, pragma says...
>
>Now, imagine you've changed the underlying implementation of a reference to be a GC handle, rather than a heap memory address like it is now.  Now you have a problem: the cast to and from void* no longer has such a simple behavior. Regardless of what path you take from here, you are going to change the behavior/design of the language.

This is one feature I don't expect to ever change, but that the syntax allows it still worries me.  Why do class references omit the pointer qualifer if they are clearly pointers?

>I hope this clears up my position. :)  Again, I'd don't think a moving/copying GC is impossible for D, but any such solution approached without some changes to the compiler/spec/design is going to be very far removed from optimal, if at all useful.

I think one issue is that it isn't always clear where to draw the line between established practice and unfinished code when using DMD as a reference.  Though that D has an ABI that seems intended to be a part of the standard says a lot--it means fundamental things like whether object references are pointers or handles simply cannot change from implementation to implementation.  I think moving GCs are an eventual goal, as they tend to be quite fast, but that's pretty far down the list of priorities.


Sean


September 30, 2005
pragma wrote:

> Bruno Medeiros wrote:
>> pragma wrote:
>> 
>>> ...
>>> However, copying and generational GC's will not work with the
>>> current incarnation of D, due to its design.
>>>
>> Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.
>> 
> 
> Its true.  The problem is that with any GC that implements
> moving of memory blocks (generational, copying or compacting),
> it needs to be able to work around the fact that D program code
> currently works in raw
> memory addresses.  Simply moving a block of memory would be
> disasterous as you'd have to tackle any number of outstanding
> pointers as well.

> ...

> Alternately, one could redefine references as abstract handles
> to managed memory, which makes the GC an intermediary for just
> about everything; a strategy that requires additional
> engineering at the
> compiler level to work in to D.  This would also mean
> complicating interop with legacy C code, again something that
> requires more programming (probably just in phobos) to get
> working.

The memory management of my component system was designed with a relocation GC in mind, but it can also work with raw memory and offers an interface to smart memory handles, that operate on a address range, separated from the raw heap.

But thinking of how this could be put in D.

foo a = new foo();
foo c = new foo();
// "a" and "c" are smart handles, not just raw references

{
foo *b; // "b" is an ordinary pointer to foo in a nested scope

b = &a;
// By assigning the pointer of "a" to "b", the smart handle "a"
// gets locked and can't be relocated anymore, in this scope

b = &c;
// Now the same for c
}

// We left the scope of be, so we can unlock the smart handles // "a" and "c"

The idea is, that pointers are only valid within their scope. Any object, that get's referenced by a pointer will by locked upon entering the scope in which the pointer resies. After "b" gets out of scope any instance it pointed to gets unlocked. The same goes for annonymous pointers, such as in a

void bar(foo *b)
{
}

void some_function()
{
 bar(&a);
}

In this example a annonymous pointer exists in the scope of "some_function", thus "a" gets locked by entering "some_function" and unlocked upon return.

Well, that is just an idea, on how to solve it. The nice thing is, that it won't require much logic, since scope determination is simple and it would only require lock and unlock calls, No reference counting required.

Well, just my 2 cents.

-- 
Wolfgang Draxinger

September 30, 2005
pragma wrote:
> In article <dhbvq9$ma8$1@digitaldaemon.com>, Bruno Medeiros says...
> 
>>pragma wrote:
>>
>>>Bruno Medeiros wrote:
>>>
>>>
>>>>pragma wrote:
>>>>
>>>>
>>>>>...
>>>>>However, copying and generational GC's will not work with the current incarnation of D, due to its design.
>>>>>
>>>>
>>>>Huh, what is this you say, is it true? If so why? I was under the impression of pretty much the opposite.
>>>>
>>>
>>Hum, are you sure you understood my question? Or maybe I misunderstood your original statement.
> 
> 
> Okay, I may need to clarify myself a little bit here.  :)
> 
> D's "design" is actually enshrined within the reference implementation (DMD) in
> addition to what is on the website.  Having used D through literally over 50
> revisions myself, I'm pretty accustomed to confusing the two.  Earlier in this
> thread, I used the word 'design' in a few spots where 'implementation' (or "D's
> implementation as DMD") may have been more appropriate.  Sorry for the
> confusion.
> 
> You're right, it's not unfeasable for *D* (as a language) to use a copying
> collector, but the current design/implementation simply won't work 100% as well
> as it does against the current GC due to how code is generated.  That was really
> the main thrust of the point I tried to make earlier.
> 
> 
>>You said (I think) that with D's design one could not implement a copying/moving (whether compacting, generational, whatever) garbage collector. If this is what you were saying, then I don't understand your reply. The language spec of D, namely the "Garbage Collection" section ( http://www.digitalmars.com/d/garbage.html ), specifies restrictions on pointer usage to allow "maximum flexibility in garbage collector design", in particular copying collectors. It even explicitly mentions them in that page ( "A compacting garbage collector may change this value" , "A copying garbage collector can arbitrarily move objects around in memory, thus ..." ) .
> 
> 
> Again, my apologies for the design/implementation thing.
> 
> As far as I understand, Walter has made many steps to keep the door open for
> copying/moving GC types; hence the references to copying GC's in the
> documentation.
> 
> It has been mentioned in this newsgroup before, it's not 100% ready for a
> copying/moving GC yet.  For example, the toHash() method of Object is presently
> implemented as using the memory location for the object; it would need to be
> changed.
> 
> 
>>Yes, to actually implement one such collector one would need type information, and would also need to change pointer values, but were is the unfeasability in that?
> 
> 
> Its not unfeasable at all, its just that DMD isn't doing this yet, nor is this
> anywhere in the specification.  Again, this is just one of the potential changes
> to D's design I was referring to. :)
> 
> 
>>>Its true.  The problem is that with any GC that implements moving of memory blocks (generational, copying or compacting), it needs to be able to work around the fact that D program code currently works in raw memory addresses.  Simply moving a block of memory would be disasterous as you'd have to tackle any number of outstanding pointers as well.
>>>
>>
>>Any copying/moving collector (of any supporting language) has that issue, and they deal with it. Why is it a problem for D? What do you mean by "D program code currently works in raw memory addresses." ?
> 
> 
> Well, its probably old hat, but new() returns the actual address in memory on
> the heap just like malloc() does.  So, right now, one can expect the following
> to work:
> 
> Object foo = cast(Object)cast(void*)(new Object);
> 
> Nice and easy: new returns a pointer which can be casted about.  Regardless of
> type, that pointer is really the same value.
> 
> Now, imagine you've changed the underlying implementation of a reference to be a
> GC handle, rather than a heap memory address like it is now.  Now you have a
> problem: the cast to and from void* no longer has such a simple behavior.
> Regardless of what path you take from here, you are going to change the
> behavior/design of the language.
> 
> 
>>>Now, you could hack this in by modifying the GC to scan everything and rewrite pointers in all managed memory as needed, but there are tons of corner-cases in this strategy.  Plus, the GC doesn't really keep track of type information, so there's no way to know if a given dword really contains a pointer or an integer.  D is modestly coupled to the conservative nature of mark-sweep, so this normally isn't a problem; so the language would need some way to make feeding this info to the GC transparent.
>>>
>>
>>The current GC doesn't keep track of type information, but there is no design issue that would impossibilitate (or difficultate) that, much to the contrary. Or is it?..
> 
> 
> You're right, there isn't anything that would prevent that.  In fact, someone
> already submitted to the newsgroup here a prototype mark/sweep GC that uses
> typeinfo hints to increase efficency (basically prevents scans of uint[] arrays
> and things of that nature).  But like everything that would make an efficent and
> useful copying/moving GC, it's not here yet.
> 
> I hope this clears up my position. :)  Again, I'd don't think a moving/copying
> GC is impossible for D, but any such solution approached without some changes to
> the compiler/spec/design is going to be very far removed from optimal, if at all
> useful.
> 
> - EricAnderton at yahoo
Yes, I understand more clearly now. I would say then that I agree there might be some blurring between the reference implementation and the D language design/spec when the implementation does some things not specified in the D spec. However when the implementation goes against the D spec in some aspect, then there is a bug in either of the two, and I would say that the spec takes precedence by default in terms of correctness.
The toHash() issue you mentioned seems such an example of a bug, however (I presume) it's still there because it has yet no ill effects and there are bigger priorities.

-- 
Bruno Medeiros - CS/E student
"Certain aspects of D are a pathway to many abilities some consider to be... unnatural."
September 30, 2005
In article <dhi5v8$fl7$1@digitaldaemon.com>, Wolfgang Draxinger says...
>
>The memory management of my component system was designed with a relocation GC in mind, but it can also work with raw memory and offers an interface to smart memory handles, that operate on a address range, separated from the raw heap.
>
>But thinking of how this could be put in D.
[snip]

Frankly, I think it must be one way or the other in D.  I don't want to have to deal with pointers in some instances and handles in another.  That just seems like needless complexity for the most part.  And it would also be a tad odd to have classes referenced by handles but all other dynamic types referenced by pointers.    Using dynamic type info would allow a moving GC to work with pointers as easily as handles anyway.  The sticking point for me (as I've mentioned before) is that class references omit the pointer specifier, which I find completely misleading.  Even if we never had stack-based classes, it would be nice if the '*' were required for class references, assuming they are guaranteed to be pointers.


Sean


« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home