Walter and I discussed the idea below a long time (years) ago. Most likely it's also been discussed in this newsgroup a couple of times. Given the state of the compiler back then, back then it seemed like a super cool idea that's entirely realizable, it would just take time for the compiler to become as capable as needed. Nowadays we're in shape to tackle it.

Here "it" is.

Back when runtime reflection was being discussed, my response was "let's focus on compile-time reflection, and then we can do run-time reflection on demand as a library". Though this might sound sensible, I initially didn't have a design. Now here's what we can do.

Currently we have information about symbols as __traits(...) intrinsics wrapped in nice but scattered ways. Now that CTFE is good enough to manipulate structs and arrays thereof, we have the possibility to finally approach things in a nicely unified, structured way.

First, we need to prime std.reflection with a few abstractions that characterize entities in a D program.

class ModuleInfo {
@property:
    string name();
    ImportInfo[] imports();
    DataInfo[] data();
    FunctionInfo[] functions();
    ClassInfo[] classes();
    StructInfo[] structs(); // includes unions
    TemplateInfo[] templates();
    EnumInfo[] enums();
    bool hasStaticCtor(), hasStaticDtor(),
      hasSharedCtor(), hasSharedDtor();
}

Probably there are a few more pieces of data, but you get the idea. Then for each of the entities mentioned above we have a similar definition. For example:

enum Protection { isPublic, isPackage, isProtected, isPrivate }

class ClassInfo {
@property:
    string name();
    string baseName();
    string parentName(); // if applicable, null otherwise
    string[] interfaces();
    bool isShared();
    Protection protection();
    DataMemberInfo[] data();
    MethodInfo[] methods();
    Object defaultConstructor();
    ...
}

Then for an e.g. method declaration we'd have:

class MethodInfo {
@property:
    string name();
    bool isStatic(), isFinal(), isOverride();
    Protection protection();
    string[] parameterTypes();
    string[] parameterNames();
}

Some details may vary, e.g. some may be straight members instead of properties etc. (I used properties to allude to use of lazy gathering of information).

So so far we have a nice collection of structured data associated with the entities in a D program. Note how this structuring differs yet has similar power to the primitives in std.traits; std.traits offers unstructured bits of information on demand (e.g. ParameterTypeNames) etc. but the objects above group information together per entity declared. All of the above goes in std.reflection, of course.

===========

On to primitives that return such data.

Given that D can (since relatively recently) create and manipulate class objects during compilation too, it follows that the classes above can be accessed in two ways - through compile-time API and run-time API. When possible, the APIs may even use the same functions; some other times they will be necessarily different.

There are two possible approaches to discovering such information. One is by fetching the ModuleInfo for the whole module and navigating it. Another one is by using search primitives from strings.

So we should have e.g.

// inside std.reflection
ModuleInfo getModuleInfo(string moduleName);

so a CT call would go like:

// client code
static info = getModuleInfo("std.algorithm");

whereas a run-time call would be:

// client code
auto info = getModuleInfo("std.algorithm");

In the latter case, the module needs to save all needed information for ri, so it should plant this:

// inside std.algorithm
mixin(makeModuleInfoAvailableDynamically());

The mixin would generate all information needed and would store it for later dynamic use.

A search API would go like e.g.

ClassInfo getClassInfo(string className);

In this case the class name could be qualified with module information etc.

===========

With this design we unify compile-time and run-time type manipulation in simple ways, by defining structured information about declarations that can be queried during compilation or dynamically.

Please chime in with thoughts. Would someone want to pioneer this project?


Andrei

On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
> With this design we unify compile-time and run-time type manipulation in simple ways, by defining structured information about declarations that can be queried during compilation or dynamically.
> 
> Please chime in with thoughts.

Are you proposing that we replace std.traits or that std.reflection be built on top of it?

Having std.reflection will be awesome for a lot of stuff - particularly the more complicated stuff - and would be a great addition to Phobos, but it seems like overkill for a lot of basic template constraints, so I wouldn't want to see it replace std.traits.

- Jonathan M Davis

On 7/21/12 5:51 PM, Jonathan M Davis wrote:
> On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
>> With this design we unify compile-time and run-time type manipulation in
>> simple ways, by defining structured information about declarations that
>> can be queried during compilation or dynamically.
>>
>> Please chime in with thoughts.
>
> Are you proposing that we replace std.traits or that std.reflection be built on
> top of it?

On top. std.reflection would be a complete reflection solution, whereas std.traits gives casual type traits as needed.

> Having std.reflection will be awesome for a lot of stuff - particularly the more
> complicated stuff - and would be a great addition to Phobos, but it seems like
> overkill for a lot of basic template constraints, so I wouldn't want to see it
> replace std.traits.

My thoughts exactly.


Andrei

On Sat, 21 Jul 2012 14:51:30 -0700, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
>> With this design we unify compile-time and run-time type manipulation in
>> simple ways, by defining structured information about declarations that
>> can be queried during compilation or dynamically.
>>
>> Please chime in with thoughts.
>
> Are you proposing that we replace std.traits or that std.reflection be built on
> top of it?
>
> Having std.reflection will be awesome for a lot of stuff - particularly the more
> complicated stuff - and would be a great addition to Phobos, but it seems like
> overkill for a lot of basic template constraints, so I wouldn't want to see it
> replace std.traits.
>
> - Jonathan M Davis

I too would want it in a different module But my read of Andrei's proposal is that it would be.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/

On 2012-07-21 21:44:51 +0000, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> said:

> Please chime in with thoughts.

Unifying runtime and compile-time reflection is a good idea, in theory at least. At least it'd make compile-time reflection easier. But when I think about the use case, unification the way you're proposing doesn't seem that great.

If you need runtime reflection in one class, you probably need it on all the derived classes too. That's the real problem runtime reflection solves: it allows you to know something about the subclasses you might otherwise not know about (because your code only know the base class).

So to make runtime reflection not a hassle, the thing that generates the runtime reflection info should be inherited somehow.

The other thing is: you very rarely need runtime reflection of everything. Especially, why would you ever need runtime reflection for a struct, where all the reflected info is available at compile time? And why create info for all fields and methods in a class in a case where all you need to get at is a pair of serialize/unserialize functions?


-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

I agree with most things proposed, however I am not a fan of the idea of mixing in runtime reflection info. Many times, you want reflection info from a type that is not your own, and thus I believe reflection should be generated by specifying a type. More importantly, a method should exist for recursively generating reflection info.

Also, I'd like to see a hierarchal approach to reflection data. The main advantage to std.reflection would be being able to use it at run-time, at which point we can't simply rely on templates, and instead if we want to store something we must rely on a base type. I think the best approach would be a hierarchal system where all reflection data derives from MemberInfo. My ideal API would like something like: https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/

On 7/21/12 8:16 PM, Kapps wrote:
> I agree with most things proposed, however I am not a fan of the idea of
> mixing in runtime reflection info. Many times, you want reflection info
> from a type that is not your own, and thus I believe reflection should
> be generated by specifying a type. More importantly, a method should
> exist for recursively generating reflection info.

I confess I have trouble understanding each of the sentences above.

> Also, I'd like to see a hierarchal approach to reflection data. The main
> advantage to std.reflection would be being able to use it at run-time,
> at which point we can't simply rely on templates, and instead if we want
> to store something we must rely on a base type.

At no place in the proposed approach is the use of templates required or needed.

> I think the best
> approach would be a hierarchal system where all reflection data derives
> from MemberInfo. My ideal API would like something like:
> https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/

I cringed at

MemberType -> { Module, Type, Field, Method, Parameter }

I was hoping to get away from slapping tagging on types just for the sake of using inheritance.


Andrei

On 21-07-2012 23:44, Andrei Alexandrescu wrote:
> Walter and I discussed the idea below a long time (years) ago. Most
> likely it's also been discussed in this newsgroup a couple of times.
> Given the state of the compiler back then, back then it seemed like a
> super cool idea that's entirely realizable, it would just take time for
> the compiler to become as capable as needed. Nowadays we're in shape to
> tackle it.
>
> Here "it" is.
>
> Back when runtime reflection was being discussed, my response was "let's
> focus on compile-time reflection, and then we can do run-time reflection
> on demand as a library". Though this might sound sensible, I initially
> didn't have a design. Now here's what we can do.
>
> Currently we have information about symbols as __traits(...) intrinsics
> wrapped in nice but scattered ways. Now that CTFE is good enough to
> manipulate structs and arrays thereof, we have the possibility to
> finally approach things in a nicely unified, structured way.
>
> First, we need to prime std.reflection with a few abstractions that
> characterize entities in a D program.
>
> class ModuleInfo {
> @property:
>      string name();
>      ImportInfo[] imports();
>      DataInfo[] data();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      StructInfo[] structs(); // includes unions
>      TemplateInfo[] templates();
>      EnumInfo[] enums();
>      bool hasStaticCtor(), hasStaticDtor(),
>        hasSharedCtor(), hasSharedDtor();
> }
>
> Probably there are a few more pieces of data, but you get the idea. Then
> for each of the entities mentioned above we have a similar definition.
> For example:
>
> enum Protection { isPublic, isPackage, isProtected, isPrivate }
>
> class ClassInfo {
> @property:
>      string name();
>      string baseName();
>      string parentName(); // if applicable, null otherwise
>      string[] interfaces();
>      bool isShared();
>      Protection protection();
>      DataMemberInfo[] data();
>      MethodInfo[] methods();
>      Object defaultConstructor();
>      ...
> }
>
> Then for an e.g. method declaration we'd have:
>
> class MethodInfo {
> @property:
>      string name();
>      bool isStatic(), isFinal(), isOverride();
>      Protection protection();
>      string[] parameterTypes();
>      string[] parameterNames();
> }
>
> Some details may vary, e.g. some may be straight members instead of
> properties etc. (I used properties to allude to use of lazy gathering of
> information).
>
> So so far we have a nice collection of structured data associated with
> the entities in a D program. Note how this structuring differs yet has
> similar power to the primitives in std.traits; std.traits offers
> unstructured bits of information on demand (e.g. ParameterTypeNames)
> etc. but the objects above group information together per entity
> declared. All of the above goes in std.reflection, of course.
>
> ===========
>
> On to primitives that return such data.
>
> Given that D can (since relatively recently) create and manipulate class
> objects during compilation too, it follows that the classes above can be
> accessed in two ways - through compile-time API and run-time API. When
> possible, the APIs may even use the same functions; some other times
> they will be necessarily different.
>
> There are two possible approaches to discovering such information. One
> is by fetching the ModuleInfo for the whole module and navigating it.
> Another one is by using search primitives from strings.
>
> So we should have e.g.
>
> // inside std.reflection
> ModuleInfo getModuleInfo(string moduleName);
>
> so a CT call would go like:
>
> // client code
> static info = getModuleInfo("std.algorithm");
>
> whereas a run-time call would be:
>
> // client code
> auto info = getModuleInfo("std.algorithm");
>
> In the latter case, the module needs to save all needed information for
> ri, so it should plant this:
>
> // inside std.algorithm
> mixin(makeModuleInfoAvailableDynamically());
>
> The mixin would generate all information needed and would store it for
> later dynamic use.
>
> A search API would go like e.g.
>
> ClassInfo getClassInfo(string className);
>
> In this case the class name could be qualified with module information etc.
>
> ===========
>
> With this design we unify compile-time and run-time type manipulation in
> simple ways, by defining structured information about declarations that
> can be queried during compilation or dynamically.
>
> Please chime in with thoughts. Would someone want to pioneer this project?
>
>
> Andrei

I'm just curious about one thing: How do you plan to reify templates (they are Turing complete after all)?

-- 
Alex Rønne Petersen
alex@lycus.org
http://lycus.org

On 21/07/2012 23:44, Andrei Alexandrescu wrote:
> Walter and I discussed the idea below a long time (years) ago. Most
> likely it's also been discussed in this newsgroup a couple of times.
> Given the state of the compiler back then, back then it seemed like a
> super cool idea that's entirely realizable, it would just take time for
> the compiler to become as capable as needed. Nowadays we're in shape to
> tackle it.
>
> Here "it" is.
>
> Back when runtime reflection was being discussed, my response was "let's
> focus on compile-time reflection, and then we can do run-time reflection
> on demand as a library". Though this might sound sensible, I initially
> didn't have a design. Now here's what we can do.
>
> Currently we have information about symbols as __traits(...) intrinsics
> wrapped in nice but scattered ways. Now that CTFE is good enough to
> manipulate structs and arrays thereof, we have the possibility to
> finally approach things in a nicely unified, structured way.
>
> First, we need to prime std.reflection with a few abstractions that
> characterize entities in a D program.
>
> class ModuleInfo {
> @property:
> string name();
> ImportInfo[] imports();
> DataInfo[] data();
> FunctionInfo[] functions();
> ClassInfo[] classes();
> StructInfo[] structs(); // includes unions
> TemplateInfo[] templates();
> EnumInfo[] enums();
> bool hasStaticCtor(), hasStaticDtor(),
> hasSharedCtor(), hasSharedDtor();
> }
>
> Probably there are a few more pieces of data, but you get the idea. Then
> for each of the entities mentioned above we have a similar definition.
> For example:
>
> enum Protection { isPublic, isPackage, isProtected, isPrivate }
>
> class ClassInfo {
> @property:
> string name();
> string baseName();
> string parentName(); // if applicable, null otherwise
> string[] interfaces();
> bool isShared();
> Protection protection();
> DataMemberInfo[] data();
> MethodInfo[] methods();
> Object defaultConstructor();
> ...
> }
>
> Then for an e.g. method declaration we'd have:
>
> class MethodInfo {
> @property:
> string name();
> bool isStatic(), isFinal(), isOverride();
> Protection protection();
> string[] parameterTypes();
> string[] parameterNames();
> }
>
> Some details may vary, e.g. some may be straight members instead of
> properties etc. (I used properties to allude to use of lazy gathering of
> information).
>
> So so far we have a nice collection of structured data associated with
> the entities in a D program. Note how this structuring differs yet has
> similar power to the primitives in std.traits; std.traits offers
> unstructured bits of information on demand (e.g. ParameterTypeNames)
> etc. but the objects above group information together per entity
> declared. All of the above goes in std.reflection, of course.
>
> ===========
>
> On to primitives that return such data.
>
> Given that D can (since relatively recently) create and manipulate class
> objects during compilation too, it follows that the classes above can be
> accessed in two ways - through compile-time API and run-time API. When
> possible, the APIs may even use the same functions; some other times
> they will be necessarily different.
>
> There are two possible approaches to discovering such information. One
> is by fetching the ModuleInfo for the whole module and navigating it.
> Another one is by using search primitives from strings.
>
> So we should have e.g.
>
> // inside std.reflection
> ModuleInfo getModuleInfo(string moduleName);
>
> so a CT call would go like:
>
> // client code
> static info = getModuleInfo("std.algorithm");
>
> whereas a run-time call would be:
>
> // client code
> auto info = getModuleInfo("std.algorithm");
>
> In the latter case, the module needs to save all needed information for
> ri, so it should plant this:
>
> // inside std.algorithm
> mixin(makeModuleInfoAvailableDynamically());
>
> The mixin would generate all information needed and would store it for
> later dynamic use.
>
> A search API would go like e.g.
>
> ClassInfo getClassInfo(string className);
>
> In this case the class name could be qualified with module information etc.
>
> ===========
>
> With this design we unify compile-time and run-time type manipulation in
> simple ways, by defining structured information about declarations that
> can be queried during compilation or dynamically.
>
> Please chime in with thoughts. Would someone want to pioneer this project?
>
>
> Andrei

I don't understand. Are theses structures generated by the compiler or some kind of libraries at compile time on a per needed basis ?

On 7/21/12, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> class ModuleInfo {
> @property:
>      string name();
>      ImportInfo[] imports();
>      DataInfo[] data();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      StructInfo[] structs(); // includes unions
>      TemplateInfo[] templates();
>      EnumInfo[] enums();
>      bool hasStaticCtor(), hasStaticDtor(),
>        hasSharedCtor(), hasSharedDtor();
> }

Are class/struct/function/etc templates going to be stored in the templates field? Then you'd have to tag each template with a type, e.g. "class template" vs "function template" to be able to filter them out.

Otherwise classes/functions/etc could have an optional
"TemplateTypeInfo[] typeParams" field so you could filter out
templated from non-templated types by checking their typeParams field,
e.g.: auto tempClasses = filter!(a => !empty(a.typeParams)
)(modinfo.classes);

I use a similar structure to what you've defined for my code generator and it worked out nicely for me.