Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.

If we have an instance of a class and we want to it's metadata via a reflection database, we should be able to query the reflection database using a hash table lookup on the name of the class.

However, we could get the metadata more quickly if it was available via the TypeInfo class.  This could be made possible very easily if the TypeInfo class contained a reference to an IReflection interface.  This reference could be set by the reflection database at program startup.  This would provide a direct link to meta data for each class, and would make reflection a first class citizen in D.

-Craig

Craig Black wrote:
> Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.

Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.

> 
> If we have an instance of a class and we want to it's metadata via a reflection database, we should be able to query the reflection database using a hash table lookup on the name of the class.
> 
> However, we could get the metadata more quickly if it was available via the TypeInfo class.  This could be made possible very easily if the TypeInfo class contained a reference to an IReflection interface.  This reference could be set by the reflection database at program startup.  This would provide a direct link to meta data for each class, and would make reflection a first class citizen in D.

The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably.  This is kind of contrary to D's goal as a systems programming language.  Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.

So I'd caution the addition of adding such a suite to D as something optional if at all possible.  Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in.

Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your .map file as a runtime dependency for the program.  While this works well, and is such a dead-simple way of addressing the problem, it's also easy to get out of sync IMO.

[PSA]

Craig, if you're at all interested in hammering out a reflection interface (for DDL?), the DDL Community wiki there is open for your, and everyone else's abuse.

http://www.dsource.org/projects/ddl/wiki/Community

I have some reflection concepts, and a few other ideas, outlined under the DevNotes section as well:

http://www.dsource.org/projects/ddl/wiki/DevNotes

Enjoy!

-- 
- EricAnderton at yahoo

"Pragma" <ericanderton@yahoo.removeme.com> wrote in message news:eeclc4$2206$1@digitaldaemon.com...
> Craig Black wrote:
>> Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.
>
> Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.

All in due time I'm sure.  At this point, I'd be happy with just the basics.

> The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably.  This is kind of contrary to D's goal as a systems programming language.  Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.
>
> So I'd caution the addition of adding such a suite to D as something optional if at all possible.  Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in.

Yes, optional of coarse.  DDL is a library, not a part of the D compiler. However, an extra reference per TypeInfo instance shouldn't hurt too bad. If not using reflection, the reference would simply be null.

> Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your .map file as a runtime dependency for the program.  While this works well, and is such a dead-simple way of addressing the problem, it's also easy to get out of sync IMO.
>
> [PSA]
>
> Craig, if you're at all interested in hammering out a reflection interface (for DDL?), the DDL Community wiki there is open for your, and everyone else's abuse.

I appreciate the offer very much.  However, at the moment I am very busy
building the
next version of our software, which is written in C++.  I'm waiting on D to
become
mature enough so that I can pitch it to my boss as something we cannot live
without.
The first time I mentioned D to him as a useful technology, he asked if it
provided
reflection, a feature that would be very useful for our product.  I had to
admit that
it does not provide reflection yet.

It's a hard sale for a number of technical reasons.  I could elaborate, but
probably
nobody cares all that much.  Anyway, perhaps I will take you up on that
offer when I
have more time.

-Craig

Pragma wrote:
> Craig Black wrote:
>> Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.
> 
> Technically DDL does, although I'm hesitant to call it that since it doesn't have all the typical reflection bells-and-whistles yet.
> 
>>
>> If we have an instance of a class and we want to it's metadata via a reflection database, we should be able to query the reflection database using a hash table lookup on the name of the class.
>>
>> However, we could get the metadata more quickly if it was available via the TypeInfo class.  This could be made possible very easily if the TypeInfo class contained a reference to an IReflection interface.  This reference could be set by the reflection database at program startup.  This would provide a direct link to meta data for each class, and would make reflection a first class citizen in D.
> 
> The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably.  This is kind of contrary to D's goal as a systems programming language.  Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.
> 
> So I'd caution the addition of adding such a suite to D as something optional if at all possible.  Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in.
> 
> Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your .map file as a runtime dependency for the program.  While this works well, and is such a dead-simple way of addressing the problem, it's also easy to get out of sync IMO.
> 
> [PSA]
> 
> Craig, if you're at all interested in hammering out a reflection interface (for DDL?), the DDL Community wiki there is open for your, and everyone else's abuse.
> 
> http://www.dsource.org/projects/ddl/wiki/Community
> 
> I have some reflection concepts, and a few other ideas, outlined under the DevNotes section as well:
> 
> http://www.dsource.org/projects/ddl/wiki/DevNotes
> 
> Enjoy!
> 

Hum, it may be the case for runtime reflection, but as for compile-time reflection, it would have quite a lot of usefulness, and no performance(or other kind of) drawbacks of merely being available.
I'm not saying this about compile time reflection in the sense that it would serve as substitute to runtime reflection, it wouldn't. At least not in a complete, for-all-cases sense. I think both kinds have some uses that cannot be covered by the other kind.


-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D

On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:

> Craig Black wrote:
>> Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.

>> However, we could get the metadata more quickly if it was available via the TypeInfo class.  This could be made possible very easily if the TypeInfo class contained a reference to an IReflection interface.  This reference could be set by the reflection database at program startup.  This would provide a direct link to meta data for each class, and would make reflection a first class citizen in D.
> 
> The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably.  This is kind of contrary to D's goal as a systems programming language.  Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.

Different people undoubtably have contrasting ideas about what they'd
like to use reflection for.   In my mind, the most important application
of Java's reflection interface is the automation of object serialization.
Since I/O programming tends to be one of the most time consuming and
error-prone aspects of development, any sort of powerful automation
in this area is valuable.  I'm a newbie to D, but as far as I can tell
from the docs, the info to do something comparable isn't available in
the binary files or the compile time interface.

> So I'd caution the addition of adding such a suite to D as something optional if at all possible.  Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in.

It could be supported without bloat by making the info either something
which is either statically accessed or else dynamically loaded from
a supplemental database file.  Opening up useful interfaces to the
language parser and using them  in separate tools (i.e. like Open C++)
is another possible approach to compile time reflection.

Josh Stern wrote:
> On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:
> 
>> Craig Black wrote:
>>> Since DDL provides run-time reflection, and is finally ready to use, I thought it would be a good time to share my idea to improve performance when accessing metadata.
> 
>>> However, we could get the metadata more quickly if it was available via the TypeInfo class.  This could be made possible very easily if the TypeInfo class contained a reference to an IReflection interface.  This reference could be set by the reflection database at program startup.  This would provide a direct link to meta data for each class, and would make reflection a first class citizen in D.
>> The only reason why I'd consider full-on reflection for D a bad move is that it'll bloat program sizes considerably.  This is kind of contrary to D's goal as a systems programming language.  Taking into consideration the sheer size of your average .map file is a good indicator of just how bad things can get.
> 
> Different people undoubtably have contrasting ideas about what they'd
> like to use reflection for.   In my mind, the most important application
> of Java's reflection interface is the automation of object serialization.
> Since I/O programming tends to be one of the most time consuming and
> error-prone aspects of development, any sort of powerful automation
> in this area is valuable.  I'm a newbie to D, but as far as I can tell
> from the docs, the info to do something comparable isn't available in the binary files or the compile time interface.

I agree with you completely, Josh.  D really needs both a runtime and a compile-time reflection facility to suit all modes of use.  On the compile-time side, there have been some rather amazing efforts to use templates to provide traits and other info - but there's obviously some stuff missing from TypeInfo.

> 
>> So I'd caution the addition of adding such a suite to D as something optional if at all possible.  Even something minimalisitc would work - like the inclusion of just a hashtable of all the program's symbols (metadata as Craig put it above) that is *optionally* compiled in.
> 
> It could be supported without bloat by making the info either something
> which is either statically accessed or else dynamically loaded from
> a supplemental database file.  Opening up useful interfaces to the
> language parser and using them  in separate tools (i.e. like Open C++)
> is another possible approach to compile time reflection.

FWIW, I have the "dynamically loaded from a supplemental database file" part done.  DDL just uses the map file:

http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect

//reflect.d
module reflect;
import ddl.DefaultRegistry;
import mango.io.Stdout;

void test(){
    Stdout.put("Hello DDL World"c).put(CR);
}

void main(){
    auto registry = new DefaultRegistry();
    auto inSitu = registry.load("reflect.map");

    auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
    testFn();
}

It's not as pretty as Java's reflection suite, but IMO, it's only a few wrapper classes away from that level of utility.

-- 
- EricAnderton at yahoo

> 
> FWIW, I have the "dynamically loaded from a supplemental database file" part done.  DDL just uses the map file:
> 
> http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect
> 
> //reflect.d
> module reflect;
> import ddl.DefaultRegistry;
> import mango.io.Stdout;
> 
> void test(){
>     Stdout.put("Hello DDL World"c).put(CR);
> }
> 
> void main(){
>     auto registry = new DefaultRegistry();
>     auto inSitu = registry.load("reflect.map");
> 
>     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
>     testFn();
> }
> 
> It's not as pretty as Java's reflection suite, but IMO, it's only a few wrapper classes away from that level of utility.
> 

Isn't this dynamic binding rather than reflection? I think of reflection as the ability to find what the fields and methods are for an object. For example, with Java reflection has the following

Class c = someObject.getClass()
c.getFields();
c.getMethods();


Does DDL have the ability to lookup methods and fields? For example, how would you determine "reflect.test" without knowing it beforehand?

  Bradley

user@domain.invalid wrote:
>>
>> FWIW, I have the "dynamically loaded from a supplemental database file" part done.  DDL just uses the map file:
>>
>> http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect
>>
>> //reflect.d
>> module reflect;
>> import ddl.DefaultRegistry;
>> import mango.io.Stdout;
>>
>> void test(){
>>     Stdout.put("Hello DDL World"c).put(CR);
>> }
>>
>> void main(){
>>     auto registry = new DefaultRegistry();
>>     auto inSitu = registry.load("reflect.map");
>>
>>     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
>>     testFn();
>> }
>>
>> It's not as pretty as Java's reflection suite, but IMO, it's only a few wrapper classes away from that level of utility.
>>
> 
> Isn't this dynamic binding rather than reflection? I think of reflection as the ability to find what the fields and methods are for an object. For example, with Java reflection has the following
> 
> Class c = someObject.getClass()
> c.getFields();
> c.getMethods();

Ahh, I see what you mean.  The example I cited was really more along the lines of binding, but it does employ a crude form of reflection to get to the binding step (searches through metadata for a kind of symbol).

> 
> 
> Does DDL have the ability to lookup methods and fields? For example, how would you determine "reflect.test" without knowing it beforehand?

Yes and no.  It does have the ability to lookup methods and fields, but at present, telling the difference from what you do and don't want is still very much DIY:

auto inSitu = registry.load("reflect.map");
foreach(module; insitu.getModules){
	foreach(symbol; module.getSymbols){
            // each symbol.name is a mangled D symbol
            // that can be inspected for it's type and namespace
            // symbol.address provides where it's actually located
	    // (ddl.Demangle.d can handle some aspects of this)
        }
}

ExporSymbols in the above cover just about everything: ModuleInfo, Class Definitions, ctors, dtors, functions, fields, methods and even vtables, initalizers, asserts and the rest of the zoo of D symbols exposed in .obj files.

So you can look for all the metadata you need, without having to bind to a given symbol.  Granted, it still feels like grocery shopping out of the "scratched and dented" bin.  Having a proper facade over this is a much needed, next step.

-- 
- EricAnderton at yahoo