View mode: basic / threaded / horizontal-split · Log in · Help
July 21, 2012
Time for std.reflection
Walter and I discussed the idea below a long time (years) ago. Most 
likely it's also been discussed in this newsgroup a couple of times. 
Given the state of the compiler back then, back then it seemed like a 
super cool idea that's entirely realizable, it would just take time for 
the compiler to become as capable as needed. Nowadays we're in shape to 
tackle it.

Here "it" is.

Back when runtime reflection was being discussed, my response was "let's 
focus on compile-time reflection, and then we can do run-time reflection 
on demand as a library". Though this might sound sensible, I initially 
didn't have a design. Now here's what we can do.

Currently we have information about symbols as __traits(...) intrinsics 
wrapped in nice but scattered ways. Now that CTFE is good enough to 
manipulate structs and arrays thereof, we have the possibility to 
finally approach things in a nicely unified, structured way.

First, we need to prime std.reflection with a few abstractions that 
characterize entities in a D program.

class ModuleInfo {
@property:
    string name();
    ImportInfo[] imports();
    DataInfo[] data();
    FunctionInfo[] functions();
    ClassInfo[] classes();
    StructInfo[] structs(); // includes unions
    TemplateInfo[] templates();
    EnumInfo[] enums();
    bool hasStaticCtor(), hasStaticDtor(),
      hasSharedCtor(), hasSharedDtor();
}

Probably there are a few more pieces of data, but you get the idea. Then 
for each of the entities mentioned above we have a similar definition. 
For example:

enum Protection { isPublic, isPackage, isProtected, isPrivate }

class ClassInfo {
@property:
    string name();
    string baseName();
    string parentName(); // if applicable, null otherwise
    string[] interfaces();
    bool isShared();
    Protection protection();
    DataMemberInfo[] data();
    MethodInfo[] methods();
    Object defaultConstructor();
    ...
}

Then for an e.g. method declaration we'd have:

class MethodInfo {
@property:
    string name();
    bool isStatic(), isFinal(), isOverride();
    Protection protection();
    string[] parameterTypes();
    string[] parameterNames();
}

Some details may vary, e.g. some may be straight members instead of 
properties etc. (I used properties to allude to use of lazy gathering of 
information).

So so far we have a nice collection of structured data associated with 
the entities in a D program. Note how this structuring differs yet has 
similar power to the primitives in std.traits; std.traits offers 
unstructured bits of information on demand (e.g. ParameterTypeNames) 
etc. but the objects above group information together per entity 
declared. All of the above goes in std.reflection, of course.

===========

On to primitives that return such data.

Given that D can (since relatively recently) create and manipulate class 
objects during compilation too, it follows that the classes above can be 
accessed in two ways - through compile-time API and run-time API. When 
possible, the APIs may even use the same functions; some other times 
they will be necessarily different.

There are two possible approaches to discovering such information. One 
is by fetching the ModuleInfo for the whole module and navigating it. 
Another one is by using search primitives from strings.

So we should have e.g.

// inside std.reflection
ModuleInfo getModuleInfo(string moduleName);

so a CT call would go like:

// client code
static info = getModuleInfo("std.algorithm");

whereas a run-time call would be:

// client code
auto info = getModuleInfo("std.algorithm");

In the latter case, the module needs to save all needed information for 
ri, so it should plant this:

// inside std.algorithm
mixin(makeModuleInfoAvailableDynamically());

The mixin would generate all information needed and would store it for 
later dynamic use.

A search API would go like e.g.

ClassInfo getClassInfo(string className);

In this case the class name could be qualified with module information etc.

===========

With this design we unify compile-time and run-time type manipulation in 
simple ways, by defining structured information about declarations that 
can be queried during compilation or dynamically.

Please chime in with thoughts. Would someone want to pioneer this project?


Andrei
July 21, 2012
Re: Time for std.reflection
On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
> With this design we unify compile-time and run-time type manipulation in
> simple ways, by defining structured information about declarations that
> can be queried during compilation or dynamically.
> 
> Please chime in with thoughts.

Are you proposing that we replace std.traits or that std.reflection be built on 
top of it?

Having std.reflection will be awesome for a lot of stuff - particularly the more 
complicated stuff - and would be a great addition to Phobos, but it seems like 
overkill for a lot of basic template constraints, so I wouldn't want to see it 
replace std.traits.

- Jonathan M Davis
July 21, 2012
Re: Time for std.reflection
On 7/21/12 5:51 PM, Jonathan M Davis wrote:
> On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
>> With this design we unify compile-time and run-time type manipulation in
>> simple ways, by defining structured information about declarations that
>> can be queried during compilation or dynamically.
>>
>> Please chime in with thoughts.
>
> Are you proposing that we replace std.traits or that std.reflection be built on
> top of it?

On top. std.reflection would be a complete reflection solution, whereas 
std.traits gives casual type traits as needed.

> Having std.reflection will be awesome for a lot of stuff - particularly the more
> complicated stuff - and would be a great addition to Phobos, but it seems like
> overkill for a lot of basic template constraints, so I wouldn't want to see it
> replace std.traits.

My thoughts exactly.


Andrei
July 21, 2012
Re: Time for std.reflection
On Sat, 21 Jul 2012 14:51:30 -0700, Jonathan M Davis <jmdavisProg@gmx.com>  
wrote:

> On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:
>> With this design we unify compile-time and run-time type manipulation in
>> simple ways, by defining structured information about declarations that
>> can be queried during compilation or dynamically.
>>
>> Please chime in with thoughts.
>
> Are you proposing that we replace std.traits or that std.reflection be  
> built on
> top of it?
>
> Having std.reflection will be awesome for a lot of stuff - particularly  
> the more
> complicated stuff - and would be a great addition to Phobos, but it  
> seems like
> overkill for a lot of basic template constraints, so I wouldn't want to  
> see it
> replace std.traits.
>
> - Jonathan M Davis

I too would want it in a different module But my read of Andrei's proposal  
is that it would be.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
July 21, 2012
Re: Time for std.reflection
On 2012-07-21 21:44:51 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail@erdani.org> said:

> Please chime in with thoughts.

Unifying runtime and compile-time reflection is a good idea, in theory 
at least. At least it'd make compile-time reflection easier. But when I 
think about the use case, unification the way you're proposing doesn't 
seem that great.

If you need runtime reflection in one class, you probably need it on 
all the derived classes too. That's the real problem runtime reflection 
solves: it allows you to know something about the subclasses you might 
otherwise not know about (because your code only know the base class).

So to make runtime reflection not a hassle, the thing that generates 
the runtime reflection info should be inherited somehow.

The other thing is: you very rarely need runtime reflection of 
everything. Especially, why would you ever need runtime reflection for 
a struct, where all the reflected info is available at compile time? 
And why create info for all fields and methods in a class in a case 
where all you need to get at is a pair of serialize/unserialize 
functions?


-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/
July 22, 2012
Re: Time for std.reflection
I agree with most things proposed, however I am not a fan of the 
idea of mixing in runtime reflection info. Many times, you want 
reflection info from a type that is not your own, and thus I 
believe reflection should be generated by specifying a type. More 
importantly, a method should exist for recursively generating 
reflection info.

Also, I'd like to see a hierarchal approach to reflection data. 
The main advantage to std.reflection would be being able to use 
it at run-time, at which point we can't simply rely on templates, 
and instead if we want to store something we must rely on a base 
type. I think the best approach would be a hierarchal system 
where all reflection data derives from MemberInfo. My ideal API 
would like something like: 
https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/
July 22, 2012
Re: Time for std.reflection
On 7/21/12 8:16 PM, Kapps wrote:
> I agree with most things proposed, however I am not a fan of the idea of
> mixing in runtime reflection info. Many times, you want reflection info
> from a type that is not your own, and thus I believe reflection should
> be generated by specifying a type. More importantly, a method should
> exist for recursively generating reflection info.

I confess I have trouble understanding each of the sentences above.

> Also, I'd like to see a hierarchal approach to reflection data. The main
> advantage to std.reflection would be being able to use it at run-time,
> at which point we can't simply rely on templates, and instead if we want
> to store something we must rely on a base type.

At no place in the proposed approach is the use of templates required or 
needed.

> I think the best
> approach would be a hierarchal system where all reflection data derives
> from MemberInfo. My ideal API would like something like:
> https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/

I cringed at

MemberType -> { Module, Type, Field, Method, Parameter }

I was hoping to get away from slapping tagging on types just for the 
sake of using inheritance.


Andrei
July 22, 2012
Re: Time for std.reflection
On 21-07-2012 23:44, Andrei Alexandrescu wrote:
> Walter and I discussed the idea below a long time (years) ago. Most
> likely it's also been discussed in this newsgroup a couple of times.
> Given the state of the compiler back then, back then it seemed like a
> super cool idea that's entirely realizable, it would just take time for
> the compiler to become as capable as needed. Nowadays we're in shape to
> tackle it.
>
> Here "it" is.
>
> Back when runtime reflection was being discussed, my response was "let's
> focus on compile-time reflection, and then we can do run-time reflection
> on demand as a library". Though this might sound sensible, I initially
> didn't have a design. Now here's what we can do.
>
> Currently we have information about symbols as __traits(...) intrinsics
> wrapped in nice but scattered ways. Now that CTFE is good enough to
> manipulate structs and arrays thereof, we have the possibility to
> finally approach things in a nicely unified, structured way.
>
> First, we need to prime std.reflection with a few abstractions that
> characterize entities in a D program.
>
> class ModuleInfo {
> @property:
>      string name();
>      ImportInfo[] imports();
>      DataInfo[] data();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      StructInfo[] structs(); // includes unions
>      TemplateInfo[] templates();
>      EnumInfo[] enums();
>      bool hasStaticCtor(), hasStaticDtor(),
>        hasSharedCtor(), hasSharedDtor();
> }
>
> Probably there are a few more pieces of data, but you get the idea. Then
> for each of the entities mentioned above we have a similar definition.
> For example:
>
> enum Protection { isPublic, isPackage, isProtected, isPrivate }
>
> class ClassInfo {
> @property:
>      string name();
>      string baseName();
>      string parentName(); // if applicable, null otherwise
>      string[] interfaces();
>      bool isShared();
>      Protection protection();
>      DataMemberInfo[] data();
>      MethodInfo[] methods();
>      Object defaultConstructor();
>      ...
> }
>
> Then for an e.g. method declaration we'd have:
>
> class MethodInfo {
> @property:
>      string name();
>      bool isStatic(), isFinal(), isOverride();
>      Protection protection();
>      string[] parameterTypes();
>      string[] parameterNames();
> }
>
> Some details may vary, e.g. some may be straight members instead of
> properties etc. (I used properties to allude to use of lazy gathering of
> information).
>
> So so far we have a nice collection of structured data associated with
> the entities in a D program. Note how this structuring differs yet has
> similar power to the primitives in std.traits; std.traits offers
> unstructured bits of information on demand (e.g. ParameterTypeNames)
> etc. but the objects above group information together per entity
> declared. All of the above goes in std.reflection, of course.
>
> ===========
>
> On to primitives that return such data.
>
> Given that D can (since relatively recently) create and manipulate class
> objects during compilation too, it follows that the classes above can be
> accessed in two ways - through compile-time API and run-time API. When
> possible, the APIs may even use the same functions; some other times
> they will be necessarily different.
>
> There are two possible approaches to discovering such information. One
> is by fetching the ModuleInfo for the whole module and navigating it.
> Another one is by using search primitives from strings.
>
> So we should have e.g.
>
> // inside std.reflection
> ModuleInfo getModuleInfo(string moduleName);
>
> so a CT call would go like:
>
> // client code
> static info = getModuleInfo("std.algorithm");
>
> whereas a run-time call would be:
>
> // client code
> auto info = getModuleInfo("std.algorithm");
>
> In the latter case, the module needs to save all needed information for
> ri, so it should plant this:
>
> // inside std.algorithm
> mixin(makeModuleInfoAvailableDynamically());
>
> The mixin would generate all information needed and would store it for
> later dynamic use.
>
> A search API would go like e.g.
>
> ClassInfo getClassInfo(string className);
>
> In this case the class name could be qualified with module information etc.
>
> ===========
>
> With this design we unify compile-time and run-time type manipulation in
> simple ways, by defining structured information about declarations that
> can be queried during compilation or dynamically.
>
> Please chime in with thoughts. Would someone want to pioneer this project?
>
>
> Andrei

I'm just curious about one thing: How do you plan to reify templates 
(they are Turing complete after all)?

-- 
Alex Rønne Petersen
alex@lycus.org
http://lycus.org
July 22, 2012
Re: Time for std.reflection
On 21/07/2012 23:44, Andrei Alexandrescu wrote:
> Walter and I discussed the idea below a long time (years) ago. Most
> likely it's also been discussed in this newsgroup a couple of times.
> Given the state of the compiler back then, back then it seemed like a
> super cool idea that's entirely realizable, it would just take time for
> the compiler to become as capable as needed. Nowadays we're in shape to
> tackle it.
>
> Here "it" is.
>
> Back when runtime reflection was being discussed, my response was "let's
> focus on compile-time reflection, and then we can do run-time reflection
> on demand as a library". Though this might sound sensible, I initially
> didn't have a design. Now here's what we can do.
>
> Currently we have information about symbols as __traits(...) intrinsics
> wrapped in nice but scattered ways. Now that CTFE is good enough to
> manipulate structs and arrays thereof, we have the possibility to
> finally approach things in a nicely unified, structured way.
>
> First, we need to prime std.reflection with a few abstractions that
> characterize entities in a D program.
>
> class ModuleInfo {
> @property:
> string name();
> ImportInfo[] imports();
> DataInfo[] data();
> FunctionInfo[] functions();
> ClassInfo[] classes();
> StructInfo[] structs(); // includes unions
> TemplateInfo[] templates();
> EnumInfo[] enums();
> bool hasStaticCtor(), hasStaticDtor(),
> hasSharedCtor(), hasSharedDtor();
> }
>
> Probably there are a few more pieces of data, but you get the idea. Then
> for each of the entities mentioned above we have a similar definition.
> For example:
>
> enum Protection { isPublic, isPackage, isProtected, isPrivate }
>
> class ClassInfo {
> @property:
> string name();
> string baseName();
> string parentName(); // if applicable, null otherwise
> string[] interfaces();
> bool isShared();
> Protection protection();
> DataMemberInfo[] data();
> MethodInfo[] methods();
> Object defaultConstructor();
> ...
> }
>
> Then for an e.g. method declaration we'd have:
>
> class MethodInfo {
> @property:
> string name();
> bool isStatic(), isFinal(), isOverride();
> Protection protection();
> string[] parameterTypes();
> string[] parameterNames();
> }
>
> Some details may vary, e.g. some may be straight members instead of
> properties etc. (I used properties to allude to use of lazy gathering of
> information).
>
> So so far we have a nice collection of structured data associated with
> the entities in a D program. Note how this structuring differs yet has
> similar power to the primitives in std.traits; std.traits offers
> unstructured bits of information on demand (e.g. ParameterTypeNames)
> etc. but the objects above group information together per entity
> declared. All of the above goes in std.reflection, of course.
>
> ===========
>
> On to primitives that return such data.
>
> Given that D can (since relatively recently) create and manipulate class
> objects during compilation too, it follows that the classes above can be
> accessed in two ways - through compile-time API and run-time API. When
> possible, the APIs may even use the same functions; some other times
> they will be necessarily different.
>
> There are two possible approaches to discovering such information. One
> is by fetching the ModuleInfo for the whole module and navigating it.
> Another one is by using search primitives from strings.
>
> So we should have e.g.
>
> // inside std.reflection
> ModuleInfo getModuleInfo(string moduleName);
>
> so a CT call would go like:
>
> // client code
> static info = getModuleInfo("std.algorithm");
>
> whereas a run-time call would be:
>
> // client code
> auto info = getModuleInfo("std.algorithm");
>
> In the latter case, the module needs to save all needed information for
> ri, so it should plant this:
>
> // inside std.algorithm
> mixin(makeModuleInfoAvailableDynamically());
>
> The mixin would generate all information needed and would store it for
> later dynamic use.
>
> A search API would go like e.g.
>
> ClassInfo getClassInfo(string className);
>
> In this case the class name could be qualified with module information etc.
>
> ===========
>
> With this design we unify compile-time and run-time type manipulation in
> simple ways, by defining structured information about declarations that
> can be queried during compilation or dynamically.
>
> Please chime in with thoughts. Would someone want to pioneer this project?
>
>
> Andrei

I don't understand. Are theses structures generated by the compiler or 
some kind of libraries at compile time on a per needed basis ?
July 22, 2012
Re: Time for std.reflection
On 7/21/12, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> class ModuleInfo {
> @property:
>      string name();
>      ImportInfo[] imports();
>      DataInfo[] data();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      StructInfo[] structs(); // includes unions
>      TemplateInfo[] templates();
>      EnumInfo[] enums();
>      bool hasStaticCtor(), hasStaticDtor(),
>        hasSharedCtor(), hasSharedDtor();
> }

Are class/struct/function/etc templates going to be stored in the
templates field? Then you'd have to tag each template with a type,
e.g. "class template" vs "function template" to be able to filter them
out.

Otherwise classes/functions/etc could have an optional
"TemplateTypeInfo[] typeParams" field so you could filter out
templated from non-templated types by checking their typeParams field,
e.g.: auto tempClasses = filter!(a => !empty(a.typeParams)
)(modinfo.classes);

I use a similar structure to what you've defined for my code generator
and it worked out nicely for me.
« First   ‹ Prev
1 2 3 4 5
Top | Discussion index | About this forum | D home