July 22, 2012
On Sunday, 22 July 2012 at 00:21:31 UTC, Andrei Alexandrescu wrote:
> On 7/21/12 8:16 PM, Kapps wrote:
>> I agree with most things proposed, however I am not a fan of the idea of
>> mixing in runtime reflection info. Many times, you want reflection info
>> from a type that is not your own, and thus I believe reflection should
>> be generated by specifying a type. More importantly, a method should
>> exist for recursively generating reflection info.
>
> I confess I have trouble understanding each of the sentences above.

What I meant was that when something requires reflection, it generally requires that everything it contains has reflection information as well. If using a mixin directly in the module itself, when a module does not include this mixin it will never get reflection information. This may or may not be desired. However it completely prevents many common uses of reflection, including serialization. So long as there is any dependency on any type or method that does not have reflection info, the entire operation would fail. This means that people will either start adding reflection to modules that do not need it in fear that someone may try to use a member of their module, or too sparsely thus making reflection all but unuseable for many common situations.

>
>> Also, I'd like to see a hierarchal approach to reflection data. The main
>> advantage to std.reflection would be being able to use it at run-time,
>> at which point we can't simply rely on templates, and instead if we want
>> to store something we must rely on a base type.
>
> At no place in the proposed approach is the use of templates required or needed.

What I mean by this is that people rely on the ability to use templates as a replacement to inheritance or common interfaces. For example, ranges rely on methods existing rather than using an interface for a range. With reflection, this isn't ideal. For example, if you wanted to get all the fields and properties in a module, including nested ones, you'd currently have to get all fields, in a module, add those, then get all types in a module, then all types in that type, then get all fields from those, etc. With a hierarchal approach where MemberInfo has a Children property, you'd simply get all children that are a FieldInfo or PropertyInfo, and recurse for all returned values that have children.

>
>> I think the best
>> approach would be a hierarchal system where all reflection data derives
>> from MemberInfo. My ideal API would like something like:
>> https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/
>
> I cringed at
>
> MemberType -> { Module, Type, Field, Method, Parameter }
>
> I was hoping to get away from slapping tagging on types just for the sake of using inheritance.
>

Admittedly, the MemberType wouldn't be necessary and I don't see as having a huge benefit. In almost all cases you know what you want to operate on, inheritance simply makes the API cleaner and gives access to common methods more easily.


July 22, 2012
On Sunday, July 22, 2012 05:19:39 Kapps wrote:
> What I meant was that when something requires reflection, it generally requires that everything it contains has reflection information as well. If using a mixin directly in the module itself, when a module does not include this mixin it will never get reflection information. This may or may not be desired. However it completely prevents many common uses of reflection, including serialization. So long as there is any dependency on any type or method that does not have reflection info, the entire operation would fail. This means that people will either start adding reflection to modules that do not need it in fear that someone may try to use a member of their module, or too sparsely thus making reflection all but unuseable for many common situations.

Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the cost of having it. That means that however it's done is going to require that every type that has it be marked in one way or another to enable that functionality. That's part of the cost of being a systems language. Whatever solution we come up with must take that into account.

- Jonathan M Davis
July 22, 2012
On Sunday, 22 July 2012 at 03:47:16 UTC, Jonathan M Davis wrote:
> Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the
> cost of having it. That means that however it's done is going to require that
> every type that has it be marked in one way or another to enable that
> functionality. That's part of the cost of being a systems language. Whatever
> solution we come up with must take that into account.
>
> - Jonathan M Davis

But the whole point is that not every type has it; only those that are registered. It just happens that you can register types besides yourself, which is an absolute necessity for many common purposes of reflection. For serialization for example, the serializer / deserializer should create reflection information for all types contained in a class.
July 22, 2012
On 7/21/12 8:28 PM, Alex Rønne Petersen wrote:
> I'm just curious about one thing: How do you plan to reify templates
> (they are Turing complete after all)?

I don't know.

Andrei

P.S. Please don't overquote.
July 22, 2012
On 7/21/12 8:41 PM, deadalnix wrote:
> I don't understand. Are theses structures generated by the compiler or
> some kind of libraries at compile time on a per needed basis ?

These are all library structures. Internally they use std.traits, which in turns uses compiler magic with __traits.

Andrei

P.S. What this thing wit quoting a long message to make a 1-line point? Is that a thing?
July 22, 2012
On 7/21/12 8:46 PM, Andrej Mitrovic wrote:
> On 7/21/12, Andrei Alexandrescu<SeeWebsiteForEmail@erdani.org>  wrote:
>> class ModuleInfo {
>> @property:
>>       string name();
>>       ImportInfo[] imports();
>>       DataInfo[] data();
>>       FunctionInfo[] functions();
>>       ClassInfo[] classes();
>>       StructInfo[] structs(); // includes unions
>>       TemplateInfo[] templates();
>>       EnumInfo[] enums();
>>       bool hasStaticCtor(), hasStaticDtor(),
>>         hasSharedCtor(), hasSharedDtor();
>> }
>
> Are class/struct/function/etc templates going to be stored in the
> templates field? Then you'd have to tag each template with a type,
> e.g. "class template" vs "function template" to be able to filter them
> out.

Perhaps we could accommodate parameterized types together with non-parameterized types by having e.g. additional properties that are null for non-parameterized types.

> Otherwise classes/functions/etc could have an optional
> "TemplateTypeInfo[] typeParams" field so you could filter out
> templated from non-templated types by checking their typeParams field,
> e.g.: auto tempClasses = filter!(a =>  !empty(a.typeParams)
> )(modinfo.classes);
>
> I use a similar structure to what you've defined for my code generator
> and it worked out nicely for me.

That sounds great!

Andrei
July 22, 2012
On 22/07/2012 06:48, Andrei Alexandrescu wrote:
> On 7/21/12 8:41 PM, deadalnix wrote:
>> I don't understand. Are theses structures generated by the compiler or
>> some kind of libraries at compile time on a per needed basis ?
>
> These are all library structures. Internally they use std.traits, which
> in turns uses compiler magic with __traits.
>
> Andrei
>

Sounds good, but what about derived classes ?

> P.S. What this thing wit quoting a long message to make a 1-line point?
> Is that a thing?

It is what happen when you editor collapse the text :D
July 22, 2012
On 22/07/2012 05:46, Jonathan M Davis wrote:
> Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the
> cost of having it. That means that however it's done is going to require that
> every type that has it be marked in one way or another to enable that
> functionality.

I don't think both are that related here.

Yes, we are a system language, so this have to be opt-in. But no, this doesn't means that every type have to be marked to be reflected.

I'd expect from std.reflection that it is able to reflect recursively from the marked starting point.
July 22, 2012
On Sat, Jul 21, 2012 at 11:44 PM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

Nice!

> class ModuleInfo {
> @property:
>     string name();
>     ImportInfo[] imports();
>     DataInfo[] data();
>     FunctionInfo[] functions();
>     ClassInfo[] classes();
>     StructInfo[] structs(); // includes unions
>     TemplateInfo[] templates();
>     EnumInfo[] enums();
>     bool hasStaticCtor(), hasStaticDtor(),
>       hasSharedCtor(), hasSharedDtor();
> }

A few questions:

1) About templates, would class/struct/function templates be in there? As in:

class Foo(T,U) { ... }
=>
class TemplateInfo {
@property:
    string name();
    FunctionInfo[] functions();
    ClassInfo[] classes();
    TemplateInfo[] templates(); // Yes, templates in templates are
possible, and so on, recursively
...
}

2) Why classes, as opposed to structs? Would inheritance/hierarchies
play a role there? Or reference semantics?
Note that between structs, classes, templates and modules there is a
lot of redundancy. A possibility could be to have an AggregateInfo
base class.

3) For a class, a (to my eyes) standard request would be to get the
parents chain, up to Object. Could that be added in ClassInfo, or as
an external free function? (edit: scratch that, you're talking about
lazy gathering below).

4) How would that allows queries like "Here is class C, give me all its available subclasses."? Hmm, wait, I get it: extract classes from the module, and recursively from imported modules. From these classes, extract the parent classes and so on, until the search ranged over the whole inheritance tree. I guess inheritance info could be standard enough for std.reflection to provide such a search.

5) The compiler can emit JSON output giving a partial view of the same information. As a long-term goal, I suggest these infos should be compatible somehow. The JSON output should be enriched, and we should ascertain that using std.json to read this kind of automatically-generated information should give std.reflection infos back.

6) Is really all the necessary info available through std.traits and __traits? Imported modules and their subtilities (renamed functions, etc) seem out of reach, no?

7) I know your examples are not complete, but don't forget aliases and symbols, and module-level values. Since these can be of any type, I'm not sure how they are managed in your scheme. I mean, you cannot have IntInfo[], DoubleInfo[], ...

8) (more a remark than a question) To me, it's another example of a
tree-like structure. I humbly suggest we get a std.tree somewhere, to
manipulate generic n-ary trees: mapping on trees, searching
(extracting elements or whole subtrees), folding trees, etc. That is,
a module that deals with trees/graphs not so much as containers, but
as a way to represent the relationships between elements. I'm not sure
I'm so clear here, but ideally std.json, std.xml, std.reflection and
probably a few others should use it as a low-level basis.
Btw, I have tree / trees algo (including depth-first / breadth-first
ranges) / graph / graph algo (strongly connected components, search in
graphs, ...) on github, but it's code that predate std.container and
is in limbo, pending std.allocator.


Philippe
July 22, 2012
On 7/22/12 9:39 AM, Philippe Sigaud wrote:
> 1) About templates, would class/struct/function templates be in there?
> As in:
>
> class Foo(T,U) { ... }
> =>
> class TemplateInfo {
> @property:
>      string name();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      TemplateInfo[] templates(); // Yes, templates in templates are
> possible, and so on, recursively
> ....
> }

Yah, ideally all entities definable in a D module should be available via reflection. But I focused on things that e.g. the user of a dynamically-loaded library would be interested in: functions, classes.

std.reflection could become the lynchpin for dynamic library use; once the library is loaded (with dlopen or such), the client needs to call getModuleInfo() (a C function that can be found with dlsym()) and then get access to pointers to functions necessary for doing all other work. (Note that my examples don't yet include pointers to executable code yet.)

> 2) Why classes, as opposed to structs? Would inheritance/hierarchies
> play a role there? Or reference semantics?
> Note that between structs, classes, templates and modules there is a
> lot of redundancy. A possibility could be to have an AggregateInfo
> base class.

Initially I used struct, but then I figured reference semantics are more natural for storing cross-entity information, as you indeed took advantage of in your TemplateInfo.

> 3) For a class, a (to my eyes) standard request would be to get the
> parents chain, up to Object. Could that be added in ClassInfo, or as
> an external free function? (edit: scratch that, you're talking about
> lazy gathering below).

I used strings because I assumed it's easy to just query the ClassInfo given a string. But yes, we could use references to ClassInfo and InterfaceInfo etc. in a transitive manner.

> 4) How would that allows queries like "Here is class C, give me all
> its available subclasses."? Hmm, wait, I get it: extract classes from
> the module, and recursively from imported modules. From these classes,
> extract the parent classes and so on, until the search ranged over the
> whole inheritance tree. I guess inheritance info could be standard
> enough for std.reflection to provide such a search.

Something like that. Note that such a query is not particularly OO-ish, because getting a class' cone (totality of subclasses) works against the modularity that inheritance is meant for. I don't think we should make getting class cones particularly easy.

> 5) The compiler can emit JSON output giving a partial view of the same
> information. As a long-term goal, I suggest these infos should be
> compatible somehow. The JSON output should be enriched, and we should
> ascertain that using std.json to read this kind of
> automatically-generated information should give std.reflection infos
> back.

Yes, that's a great connection that Walter and I discussed a bit.

> 6) Is really all the necessary info available through std.traits and
> __traits? Imported modules and their subtilities (renamed functions,
> etc) seem out of reach, no?

We'll need indeed to enhance __traits with what's needed. Much of the point of std.reflection is to determine exactly what's there and what's needed. And that starts with the data structures design (the algorithmic aspects are minor).

> 7) I know your examples are not complete, but don't forget aliases and
> symbols, and module-level values. Since these can be of any type, I'm
> not sure how they are managed in your scheme. I mean, you cannot have
> IntInfo[], DoubleInfo[], ...

I sort of eschewed part of that by using strings for types.

> 8) (more a remark than a question) To me, it's another example of a
> tree-like structure. I humbly suggest we get a std.tree somewhere, to
> manipulate generic n-ary trees: mapping on trees, searching
> (extracting elements or whole subtrees), folding trees, etc. That is,
> a module that deals with trees/graphs not so much as containers, but
> as a way to represent the relationships between elements. I'm not sure
> I'm so clear here, but ideally std.json, std.xml, std.reflection and
> probably a few others should use it as a low-level basis.
> Btw, I have tree / trees algo (including depth-first / breadth-first
> ranges) / graph / graph algo (strongly connected components, search in
> graphs, ...) on github, but it's code that predate std.container and
> is in limbo, pending std.allocator.

Well you're the resident crazy-stuff-during-compilation guy. Did you try your trees during compilation?


Andrei