Hello all, I read on the realtime type information how-to page that there are
some runtime type information facilities in D.
(http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)

I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation of the Framework for Integrated Test (http://fit.c2.com).  It was extremely painful to implement portable reflection capabilities into C++ to mimic what goes on in Java.  Basically I used every function pointer, template, macro, and static linker trick I knew to pull it off (and it's still fugly.)

If I were to implement fit for D (DeeFIT?) it would be extremely helpful to have Class.forName() functionality out of the box as well as the ability to get pointers-to-function and references to fields by name as well for a given object.

Would it be possible for the D compiler to generate some table of symbols or synthetically generated reflection module and provide that for linkage into the final binary?  Maybe the compiler could employ the same sorts template specialization and static initialization techniques I used in CeeFIT to generate such artifacts.  Clearly such a table could double the size of the resulting .exe, but not like I care.

Cheers,
Dave Woldrich

Dave Woldrich wrote:
> Hello all, I read on the realtime type information how-to page that there are
> some runtime type information facilities in D.
> (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)
> 
> I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
> of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
> painful to implement portable reflection capabilities into C++ to mimic what
> goes on in Java.  Basically I used every function pointer, template, macro,
> and static linker trick I knew to pull it off (and it's still fugly.)
> 

D's reflection support is certainly better than C++'s. However, it is not quite as powerful as you are wishing it is. :-)

> If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
> have Class.forName() functionality out of the box as well as the ability to
> get pointers-to-function and references to fields by name as well for a given
> object.
> 

These are not possible directly. Some other things are:

* The reverse operation, getting the name of a class at runtime, is possible. Every Object instance has a "classinfo" property, which in turn has a "name" property. (It also have a "vtbl" property, of type void*[], which might prove useful for this kind of trickery.) Or stop listening to me, and read the docs: :-)
http://digitalmars.com/d/phobos/object.html

* D is remarkably easy to parse, especially compared to C++. If you have to, parsing user code to generate specialized code can be done in a much more robust way than C++ can dream of.

* D's /compile-time/ reflection abilities kick the pants off C++. Search around for Don Clugston's "Nameof" module. It allows you to get the name of just about any symbol at compile time. (The reverse operation of getting a symbol from its name is still not possible, however.)

> Would it be possible for the D compiler to generate some table of symbols or
> synthetically generated reflection module and provide that for linkage into
> the final binary?  Maybe the compiler could employ the same sorts template
> specialization and static initialization techniques I used in CeeFIT to
> generate such artifacts.  Clearly such a table could double the size of the
> resulting .exe, but not like I care.
> 

As I said, parsing this information out yourself is almost /easy/ compared to C++. It might also be worth pointing out that strings are allowed as template parameters in D. Also, I once again suggest you look at Don Clugston's Nameof (and Demangle). It's a fine bit of work, which appears to do half of this problem already.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://pyd.dsource.org

Dave Woldrich wrote:
> Hello all, I read on the realtime type information how-to page that there are
> some runtime type information facilities in D.
> (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation)
> 
> I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
> of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
> painful to implement portable reflection capabilities into C++ to mimic what
> goes on in Java.  Basically I used every function pointer, template, macro,
> and static linker trick I knew to pull it off (and it's still fugly.)
> 
> If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
> have Class.forName() functionality out of the box as well as the ability to
> get pointers-to-function and references to fields by name as well for a given
> object.
> 
> Would it be possible for the D compiler to generate some table of symbols or
> synthetically generated reflection module and provide that for linkage into
> the final binary?  Maybe the compiler could employ the same sorts template
> specialization and static initialization techniques I used in CeeFIT to
> generate such artifacts.  Clearly such a table could double the size of the
> resulting .exe, but not like I care.

Howdy Dave.

To my knowledge, there are two approaches one can use to put together a reflection interface on top of D, w/o hacking the compiler directly. Neither is very pretty.

1) Use the DMD front-end to compose a utility that pre-processes .d files and generates the needed reflection support code.  I think there are other D-ers who are doing this, but I can't recall who at the moment (hasn't been much word about it lately).  I honestly don't see why this can't be done today, and made to work everywhere D already does.

The facts that the front-end will furnish you with a 100% spec-compliant AST, D supports the #line directive, and the spec allows for custom pragma() statements, only make this approach more appealing.


2) Parse the .map file.  D's mangling specification allows for far more useful information than what any C++ compiler spits out.  The result is that you can extract entire call-signatures and name-spaces directly from the symbol table.  I'm presently using this technique in DDL to bootstrap dynamic linking with object files:

http://www.dsource.org/projects/ddl

The downside to this approach is that it leaves out just about everything that can resolve to an offset or bitfield: enums, struct members, class members, local vars, etc.  Otherwise, everything with a vtable, call signature, typeInfo or moduleInfo is preserved and easily accessed at runtime.


Both approaches leave the actual reflection interface (beyond TypeInfo) up for grabs.  There have been partial implementations, but nothing as complete as say what .NET or Java offers.  My $0.02 on the matter is that if you're first past the post on this, whatever you come up with is likely to become adopted in one way or another. ;)

I hope this helps.

-- 
- EricAnderton at yahoo

Pragma wrote:
> Dave Woldrich wrote:
>> Hello all, I read on the realtime type information how-to page that there are
>> some runtime type information facilities in D.
>> (http://www.prowiki.org/wiki4d/wiki.cgi?action=browse&id=HowTo/RealtimeTypeInformation) 
>>
>>
>> I am the author of CeeFIT (http://ceefit.woldrich.com), the C++ implementation
>> of the Framework for Integrated Test (http://fit.c2.com).  It was extremely
>> painful to implement portable reflection capabilities into C++ to mimic what
>> goes on in Java.  Basically I used every function pointer, template, macro,
>> and static linker trick I knew to pull it off (and it's still fugly.)
>>
>> If I were to implement fit for D (DeeFIT?) it would be extremely helpful to
>> have Class.forName() functionality out of the box as well as the ability to
>> get pointers-to-function and references to fields by name as well for a given
>> object.
>>
>> Would it be possible for the D compiler to generate some table of symbols or
>> synthetically generated reflection module and provide that for linkage into
>> the final binary?  Maybe the compiler could employ the same sorts template
>> specialization and static initialization techniques I used in CeeFIT to
>> generate such artifacts.  Clearly such a table could double the size of the
>> resulting .exe, but not like I care.
> 
> Howdy Dave.
> 
> To my knowledge, there are two approaches one can use to put together a reflection interface on top of D, w/o hacking the compiler directly. Neither is very pretty.
> 
> 1) Use the DMD front-end to compose a utility that pre-processes .d files and generates the needed reflection support code.  I think there are other D-ers who are doing this, but I can't recall who at the moment (hasn't been much word about it lately).  I honestly don't see why this can't be done today, and made to work everywhere D already does.
> 
> The facts that the front-end will furnish you with a 100% spec-compliant AST, D supports the #line directive, and the spec allows for custom pragma() statements, only make this approach more appealing.
> 
> 
> 2) Parse the .map file.  D's mangling specification allows for far more useful information than what any C++ compiler spits out.  The result is that you can extract entire call-signatures and name-spaces directly from the symbol table.  I'm presently using this technique in DDL to bootstrap dynamic linking with object files:
> 
> http://www.dsource.org/projects/ddl
> 
> The downside to this approach is that it leaves out just about everything that can resolve to an offset or bitfield: enums, struct members, class members, local vars, etc.  Otherwise, everything with a vtable, call signature, typeInfo or moduleInfo is preserved and easily accessed at runtime.
> 
> 
> Both approaches leave the actual reflection interface (beyond TypeInfo) up for grabs.  There have been partial implementations, but nothing as complete as say what .NET or Java offers.  My $0.02 on the matter is that if you're first past the post on this, whatever you come up with is likely to become adopted in one way or another. ;)
> 
> I hope this helps.
> 

Addendum:  Kirk is right on the money about the Don's Meta library.  You might be able to put something together that works via explicit registration of classes like so:

DeeFitRegister!(MyClass);

That said, Meta is a work of Template Ninjitsu that is hard to top. Tuples and the rest of D's meta programming capabilities should help you decompose classes and structs from there.

But if you need to answer questions like Class.nameof("Foobar"), then you've already out-stripped what can be done at compile-time. :)

-- 
- EricAnderton at yahoo

== Quote from Pragma (ericanderton@yahoo.removeme.com)'s article

> Neither is very pretty.
> 1) Use the DMD front-end to compose a utility that pre-processes .d
> files and generates the needed reflection support code.  I think there
> are other D-ers who are doing this, but I can't recall who at the moment
> (hasn't been much word about it lately).  I honestly don't see why this
> can't be done today, and made to work everywhere D already does.
> The facts that the front-end will furnish you with a 100% spec-compliant
> AST, D supports the #line directive, and the spec allows for custom
> pragma() statements, only make this approach more appealing.
> 2) Parse the .map file.  D's mangling specification allows for far more
> useful information than what any C++ compiler spits out.  The result is
> that you can extract entire call-signatures and name-spaces directly
> from the symbol table.  I'm presently using this technique in DDL to
> bootstrap dynamic linking with object files:
> http://www.dsource.org/projects/ddl

Excellent, thanks for this response, I will look into it.  I think hacking the compiler wouldn't be a bad deal and would probably produce the fastest executing solution.  My thoughts based on what you are saying is that static construction of compiler-generated type metadata classes would be possible and could be enabled with a compiler switch.

CeeFIT uses a two tier approach to registering types.  Within classes that need reflection, CeeFIT has macros that register fields and functions for reflection for each class instance that is created.  Macros place a "FITFIELD<T>" in the class for every field that needs to be found by name where T is the type of the field.  They place a "FITTEST<T, U>" for every zero-arg function that needs to be found by name where T is the type of the containing class and U is the type of the return type.  There are variants of FITTEST for one-arg functions as well.  Each FITFIELD and FITTEST is initialized with a string name and a pointer to the member function or field.  Templates take care of the rest.

As each FITFIELD and FITTEST member placed by the macros initializes during object construction, they add themselves to a singly-linked list hosted by a base class that facilitates the reflection.  The drawback of this is the additional memory and cpu cycles burned whenever a reflectable object is instantiated.  I did this out of necessity since the C++ limitations had me hamstrung.  Wherever the macros cant or won't work, I have a less efficient manual registration technique that users can use, but that's not really relevant to what I'm discussing here.

I figure a D compiler extension that generates these field and function metadata artifacts could generate them to some class outside the classes being described and initialize them only once at startup.  Leading to function and field metadata in a hashmap keyed by the string name representation.  Slap a simple reflection API on top of that hashmap and you've got it.

The second tier of the reflection facilities in CeeFIT is to register the classes themselves by name for construction.  To do this, macros place a static REGISTERFIXTURECLASS<T> object after the class definition is closed where T is the class type.  The linker ensures that the REGISTERFIXTURECLASS<T> is initialized at program start.   REGISTERFIXTURECLASS<T> provides the mapping between class string name and the class's zero-arg default constructor.  Upon initialization, the REGISTERFIXTURECLASS<T> adds itself to a global singly-linked list which can be searched at runtime to pull off the Class.forName() functionality I needed.

Of course, searching singly-linked lists are going to perform badly for any decent sized application, and a D version of this code could add the statically initialized REGISTERFIXTURECLASS<T> to a hashmap or whatnot instead.  I don't know what kind of DLL support there is in D, but dynamically loading and unloading classes and magically keeping this hashmap up to date would be so cherry.

I am only just now learning about D, so I don't know what the limitations of the compiler are and whether such extensions would be feasible.  As you were saying Eric, the map file could be sufficient to generate all of this type metadata code as a post-processing step, I just figure the compiler has all the information while it is running and therefore would be the best place to put the logic.  Plus, to have reflection facilities be a first class feature of the language/compiler/runtime library would be very slick.

If I had reflection in a systems-programming language like D, the tools that could be written to work with D would be incredible.  I'm sure Eclipse for D would be a dream to use and would benefit from all the refactoring and type information that Java gets.  I am already thinking of porting my videogame code to D (since I'm using a replacement GC heap in my C++ code anyhow, not like D is going to perform any slower.)  I will probably make some ant tasks for D if I do and there aren't any already.

I am just really liking the D language, thank you Walter and whoever else is responsible for this!  :)

Cheers,
Dave Woldrich

> Addendum:  Kirk is right on the money about the Don's Meta library.  You
> might be able to put something together that works via explicit
> registration of classes like so:
> DeeFitRegister!(MyClass);
> That said, Meta is a work of Template Ninjitsu that is hard to top.
> Tuples and the rest of D's meta programming capabilities should help you
> decompose classes and structs from there.
> But if you need to answer questions like Class.nameof("Foobar"), then
> you've already out-stripped what can be done at compile-time. :)

Yeah, for fit implementations, I need something like Java's "Object
Class.forName(String clazz)", "Field Class.getField(String name)", and "Method
Class.getMethod(String name, Class[] parameterTypes)"  :(  So maybe I'm asking for
too much, but I hope not since I can mostly get what I want in C++ and that's
nearly a full letter less than D.  :)

Thanks,
Dave Woldrich

Dave Woldrich schrieb am 2007-01-03:
>> Addendum:  Kirk is right on the money about the Don's Meta library.  You
>> might be able to put something together that works via explicit
>> registration of classes like so:
>> DeeFitRegister!(MyClass);
>> That said, Meta is a work of Template Ninjitsu that is hard to top.
>> Tuples and the rest of D's meta programming capabilities should help you
>> decompose classes and structs from there.
>> But if you need to answer questions like Class.nameof("Foobar"), then
>> you've already out-stripped what can be done at compile-time. :)
>
> Yeah, for fit implementations, I need something like Java's "Object
> Class.forName(String clazz)", "Field Class.getField(String name)", and "Method
> Class.getMethod(String name, Class[] parameterTypes)"  :(  So maybe I'm asking for
> too much, but I hope not since I can mostly get what I want in C++ and that's
> nearly a full letter less than D.  :)

Flectioned should be able to solve your problems, it is however currently rudimental and only available for linux.

http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip

Class.forName:
	cast(Class) cn.kuehne.flectioned.types["fully.qualified.name"]

Class.getField:
	that is impossible without template foo

Class.getMethod
	should be available at the end of the week

Unpack and have a try:
1) gcc -m32 -c elf.c
2) dmd flectioned.d sample.d elf.o -ofsample
3) ./sample

Thomas

== Quote from Thomas Kuehne (thomas-dloop@kuehne.cn)'s article
> Flectioned should be able to solve your problems, it is however currently
> rudimental and only available for linux.
> http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip
> Class.forName:
> 	cast(Class) cn.kuehne.flectioned.types["fully.qualified.name"]
> Class.getField:
> 	that is impossible without template foo
> Class.getMethod
> 	should be available at the end of the week
> Unpack and have a try:
> 1) gcc -m32 -c elf.c
> 2) dmd flectioned.d sample.d elf.o -ofsample
> 3) ./sample
> Thomas

Thomas,

Wow, your library is so impressive.  You scan the running process for symbols, wicked!  I am fascinated by how low level D programs can go, although I see you using a little C there to pull it off.  And you made a Class class, yay!  The Class.newInstance() method was very impressive.

Anyhow, there's many ways to skin a cat, and I see you're an expert cat skinner. I look forward to seeing what you come up with next.  I still think if the compiler did this work and generated type metadata code the reflection would execute faster, but your solution is working today, so I guess I shouldn't complain.  :)

Cheers,
Dave Woldrich

> 
> Flectioned should be able to solve your problems, it is however currently
> rudimental and only available for linux.
> 
> http://svn.dsource.org/projects/flectioned/downloads/flectioned.zip
> 
> Thomas
> 
> 
I updated the wiki to include this file.  It would be nice to have some of the other comments made up there too however I'm either too lazy to don't have the time to reformat what was said.

http://www.prowiki.org/wiki4d/wiki.cgi?HowTo/RealtimeTypeInformation

Dave Woldrich schrieb am 2007-01-04:
>== Quote from Thomas Kuehne (thomas-dloop@kuehne.cn)'s article

> Wow, your library is so impressive.  You scan the running process for symbols, wicked!  I am fascinated by how low level D programs can go, although I see you using a little C there to pull it off.

The C part is only there because I reused existing code and haven't bothered to translate it to D.

Thomas