View mode: basic / threaded / horizontal-split · Log in · Help
July 22, 2012
Re: Time for std.reflection
On Sunday, 22 July 2012 at 00:21:31 UTC, Andrei Alexandrescu 
wrote:
> On 7/21/12 8:16 PM, Kapps wrote:
>> I agree with most things proposed, however I am not a fan of 
>> the idea of
>> mixing in runtime reflection info. Many times, you want 
>> reflection info
>> from a type that is not your own, and thus I believe 
>> reflection should
>> be generated by specifying a type. More importantly, a method 
>> should
>> exist for recursively generating reflection info.
>
> I confess I have trouble understanding each of the sentences 
> above.

What I meant was that when something requires reflection, it 
generally requires that everything it contains has reflection 
information as well. If using a mixin directly in the module 
itself, when a module does not include this mixin it will never 
get reflection information. This may or may not be desired. 
However it completely prevents many common uses of reflection, 
including serialization. So long as there is any dependency on 
any type or method that does not have reflection info, the entire 
operation would fail. This means that people will either start 
adding reflection to modules that do not need it in fear that 
someone may try to use a member of their module, or too sparsely 
thus making reflection all but unuseable for many common 
situations.

>
>> Also, I'd like to see a hierarchal approach to reflection 
>> data. The main
>> advantage to std.reflection would be being able to use it at 
>> run-time,
>> at which point we can't simply rely on templates, and instead 
>> if we want
>> to store something we must rely on a base type.
>
> At no place in the proposed approach is the use of templates 
> required or needed.

What I mean by this is that people rely on the ability to use 
templates as a replacement to inheritance or common interfaces. 
For example, ranges rely on methods existing rather than using an 
interface for a range. With reflection, this isn't ideal. For 
example, if you wanted to get all the fields and properties in a 
module, including nested ones, you'd currently have to get all 
fields, in a module, add those, then get all types in a module, 
then all types in that type, then get all fields from those, etc. 
With a hierarchal approach where MemberInfo has a Children 
property, you'd simply get all children that are a FieldInfo or 
PropertyInfo, and recurse for all returned values that have 
children.

>
>> I think the best
>> approach would be a hierarchal system where all reflection 
>> data derives
>> from MemberInfo. My ideal API would like something like:
>> https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/
>
> I cringed at
>
> MemberType -> { Module, Type, Field, Method, Parameter }
>
> I was hoping to get away from slapping tagging on types just 
> for the sake of using inheritance.
>

Admittedly, the MemberType wouldn't be necessary and I don't see 
as having a huge benefit. In almost all cases you know what you 
want to operate on, inheritance simply makes the API cleaner and 
gives access to common methods more easily.
July 22, 2012
Re: Time for std.reflection
On Sunday, July 22, 2012 05:19:39 Kapps wrote:
> What I meant was that when something requires reflection, it
> generally requires that everything it contains has reflection
> information as well. If using a mixin directly in the module
> itself, when a module does not include this mixin it will never
> get reflection information. This may or may not be desired.
> However it completely prevents many common uses of reflection,
> including serialization. So long as there is any dependency on
> any type or method that does not have reflection info, the entire
> operation would fail. This means that people will either start
> adding reflection to modules that do not need it in fear that
> someone may try to use a member of their module, or too sparsely
> thus making reflection all but unuseable for many common
> situations.

Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the 
cost of having it. That means that however it's done is going to require that 
every type that has it be marked in one way or another to enable that 
functionality. That's part of the cost of being a systems language. Whatever 
solution we come up with must take that into account.

- Jonathan M Davis
July 22, 2012
Re: Time for std.reflection
On Sunday, 22 July 2012 at 03:47:16 UTC, Jonathan M Davis wrote:
> Runtime reflection _must_ be opt-in. We do _not_ want to make 
> all types pay the
> cost of having it. That means that however it's done is going 
> to require that
> every type that has it be marked in one way or another to 
> enable that
> functionality. That's part of the cost of being a systems 
> language. Whatever
> solution we come up with must take that into account.
>
> - Jonathan M Davis

But the whole point is that not every type has it; only those 
that are registered. It just happens that you can register types 
besides yourself, which is an absolute necessity for many common 
purposes of reflection. For serialization for example, the 
serializer / deserializer should create reflection information 
for all types contained in a class.
July 22, 2012
Re: Time for std.reflection
On 7/21/12 8:28 PM, Alex Rønne Petersen wrote:
> I'm just curious about one thing: How do you plan to reify templates
> (they are Turing complete after all)?

I don't know.

Andrei

P.S. Please don't overquote.
July 22, 2012
Re: Time for std.reflection
On 7/21/12 8:41 PM, deadalnix wrote:
> I don't understand. Are theses structures generated by the compiler or
> some kind of libraries at compile time on a per needed basis ?

These are all library structures. Internally they use std.traits, which 
in turns uses compiler magic with __traits.

Andrei

P.S. What this thing wit quoting a long message to make a 1-line point? 
Is that a thing?
July 22, 2012
Re: Time for std.reflection
On 7/21/12 8:46 PM, Andrej Mitrovic wrote:
> On 7/21/12, Andrei Alexandrescu<SeeWebsiteForEmail@erdani.org>  wrote:
>> class ModuleInfo {
>> @property:
>>       string name();
>>       ImportInfo[] imports();
>>       DataInfo[] data();
>>       FunctionInfo[] functions();
>>       ClassInfo[] classes();
>>       StructInfo[] structs(); // includes unions
>>       TemplateInfo[] templates();
>>       EnumInfo[] enums();
>>       bool hasStaticCtor(), hasStaticDtor(),
>>         hasSharedCtor(), hasSharedDtor();
>> }
>
> Are class/struct/function/etc templates going to be stored in the
> templates field? Then you'd have to tag each template with a type,
> e.g. "class template" vs "function template" to be able to filter them
> out.

Perhaps we could accommodate parameterized types together with 
non-parameterized types by having e.g. additional properties that are 
null for non-parameterized types.

> Otherwise classes/functions/etc could have an optional
> "TemplateTypeInfo[] typeParams" field so you could filter out
> templated from non-templated types by checking their typeParams field,
> e.g.: auto tempClasses = filter!(a =>  !empty(a.typeParams)
> )(modinfo.classes);
>
> I use a similar structure to what you've defined for my code generator
> and it worked out nicely for me.

That sounds great!

Andrei
July 22, 2012
Re: Time for std.reflection
On 22/07/2012 06:48, Andrei Alexandrescu wrote:
> On 7/21/12 8:41 PM, deadalnix wrote:
>> I don't understand. Are theses structures generated by the compiler or
>> some kind of libraries at compile time on a per needed basis ?
>
> These are all library structures. Internally they use std.traits, which
> in turns uses compiler magic with __traits.
>
> Andrei
>

Sounds good, but what about derived classes ?

> P.S. What this thing wit quoting a long message to make a 1-line point?
> Is that a thing?

It is what happen when you editor collapse the text :D
July 22, 2012
Re: Time for std.reflection
On 22/07/2012 05:46, Jonathan M Davis wrote:
> Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the
> cost of having it. That means that however it's done is going to require that
> every type that has it be marked in one way or another to enable that
> functionality.

I don't think both are that related here.

Yes, we are a system language, so this have to be opt-in. But no, this 
doesn't means that every type have to be marked to be reflected.

I'd expect from std.reflection that it is able to reflect recursively 
from the marked starting point.
July 22, 2012
Re: Time for std.reflection
On Sat, Jul 21, 2012 at 11:44 PM, Andrei Alexandrescu
<SeeWebsiteForEmail@erdani.org> wrote:

Nice!

> class ModuleInfo {
> @property:
>     string name();
>     ImportInfo[] imports();
>     DataInfo[] data();
>     FunctionInfo[] functions();
>     ClassInfo[] classes();
>     StructInfo[] structs(); // includes unions
>     TemplateInfo[] templates();
>     EnumInfo[] enums();
>     bool hasStaticCtor(), hasStaticDtor(),
>       hasSharedCtor(), hasSharedDtor();
> }

A few questions:

1) About templates, would class/struct/function templates be in there?
As in:

class Foo(T,U) { ... }
=>
class TemplateInfo {
@property:
   string name();
   FunctionInfo[] functions();
   ClassInfo[] classes();
   TemplateInfo[] templates(); // Yes, templates in templates are
possible, and so on, recursively
...
}

2) Why classes, as opposed to structs? Would inheritance/hierarchies
play a role there? Or reference semantics?
Note that between structs, classes, templates and modules there is a
lot of redundancy. A possibility could be to have an AggregateInfo
base class.

3) For a class, a (to my eyes) standard request would be to get the
parents chain, up to Object. Could that be added in ClassInfo, or as
an external free function? (edit: scratch that, you're talking about
lazy gathering below).

4) How would that allows queries like "Here is class C, give me all
its available subclasses."? Hmm, wait, I get it: extract classes from
the module, and recursively from imported modules. From these classes,
extract the parent classes and so on, until the search ranged over the
whole inheritance tree. I guess inheritance info could be standard
enough for std.reflection to provide such a search.

5) The compiler can emit JSON output giving a partial view of the same
information. As a long-term goal, I suggest these infos should be
compatible somehow. The JSON output should be enriched, and we should
ascertain that using std.json to read this kind of
automatically-generated information should give std.reflection infos
back.

6) Is really all the necessary info available through std.traits and
__traits? Imported modules and their subtilities (renamed functions,
etc) seem out of reach, no?

7) I know your examples are not complete, but don't forget aliases and
symbols, and module-level values. Since these can be of any type, I'm
not sure how they are managed in your scheme. I mean, you cannot have
IntInfo[], DoubleInfo[], ...

8) (more a remark than a question) To me, it's another example of a
tree-like structure. I humbly suggest we get a std.tree somewhere, to
manipulate generic n-ary trees: mapping on trees, searching
(extracting elements or whole subtrees), folding trees, etc. That is,
a module that deals with trees/graphs not so much as containers, but
as a way to represent the relationships between elements. I'm not sure
I'm so clear here, but ideally std.json, std.xml, std.reflection and
probably a few others should use it as a low-level basis.
Btw, I have tree / trees algo (including depth-first / breadth-first
ranges) / graph / graph algo (strongly connected components, search in
graphs, ...) on github, but it's code that predate std.container and
is in limbo, pending std.allocator.


Philippe
July 22, 2012
Re: Time for std.reflection
On 7/22/12 9:39 AM, Philippe Sigaud wrote:
> 1) About templates, would class/struct/function templates be in there?
> As in:
>
> class Foo(T,U) { ... }
> =>
> class TemplateInfo {
> @property:
>      string name();
>      FunctionInfo[] functions();
>      ClassInfo[] classes();
>      TemplateInfo[] templates(); // Yes, templates in templates are
> possible, and so on, recursively
> ....
> }

Yah, ideally all entities definable in a D module should be available 
via reflection. But I focused on things that e.g. the user of a 
dynamically-loaded library would be interested in: functions, classes.

std.reflection could become the lynchpin for dynamic library use; once 
the library is loaded (with dlopen or such), the client needs to call 
getModuleInfo() (a C function that can be found with dlsym()) and then 
get access to pointers to functions necessary for doing all other work. 
(Note that my examples don't yet include pointers to executable code yet.)

> 2) Why classes, as opposed to structs? Would inheritance/hierarchies
> play a role there? Or reference semantics?
> Note that between structs, classes, templates and modules there is a
> lot of redundancy. A possibility could be to have an AggregateInfo
> base class.

Initially I used struct, but then I figured reference semantics are more 
natural for storing cross-entity information, as you indeed took 
advantage of in your TemplateInfo.

> 3) For a class, a (to my eyes) standard request would be to get the
> parents chain, up to Object. Could that be added in ClassInfo, or as
> an external free function? (edit: scratch that, you're talking about
> lazy gathering below).

I used strings because I assumed it's easy to just query the ClassInfo 
given a string. But yes, we could use references to ClassInfo and 
InterfaceInfo etc. in a transitive manner.

> 4) How would that allows queries like "Here is class C, give me all
> its available subclasses."? Hmm, wait, I get it: extract classes from
> the module, and recursively from imported modules. From these classes,
> extract the parent classes and so on, until the search ranged over the
> whole inheritance tree. I guess inheritance info could be standard
> enough for std.reflection to provide such a search.

Something like that. Note that such a query is not particularly OO-ish, 
because getting a class' cone (totality of subclasses) works against the 
modularity that inheritance is meant for. I don't think we should make 
getting class cones particularly easy.

> 5) The compiler can emit JSON output giving a partial view of the same
> information. As a long-term goal, I suggest these infos should be
> compatible somehow. The JSON output should be enriched, and we should
> ascertain that using std.json to read this kind of
> automatically-generated information should give std.reflection infos
> back.

Yes, that's a great connection that Walter and I discussed a bit.

> 6) Is really all the necessary info available through std.traits and
> __traits? Imported modules and their subtilities (renamed functions,
> etc) seem out of reach, no?

We'll need indeed to enhance __traits with what's needed. Much of the 
point of std.reflection is to determine exactly what's there and what's 
needed. And that starts with the data structures design (the algorithmic 
aspects are minor).

> 7) I know your examples are not complete, but don't forget aliases and
> symbols, and module-level values. Since these can be of any type, I'm
> not sure how they are managed in your scheme. I mean, you cannot have
> IntInfo[], DoubleInfo[], ...

I sort of eschewed part of that by using strings for types.

> 8) (more a remark than a question) To me, it's another example of a
> tree-like structure. I humbly suggest we get a std.tree somewhere, to
> manipulate generic n-ary trees: mapping on trees, searching
> (extracting elements or whole subtrees), folding trees, etc. That is,
> a module that deals with trees/graphs not so much as containers, but
> as a way to represent the relationships between elements. I'm not sure
> I'm so clear here, but ideally std.json, std.xml, std.reflection and
> probably a few others should use it as a low-level basis.
> Btw, I have tree / trees algo (including depth-first / breadth-first
> ranges) / graph / graph algo (strongly connected components, search in
> graphs, ...) on github, but it's code that predate std.container and
> is in limbo, pending std.allocator.

Well you're the resident crazy-stuff-during-compilation guy. Did you try 
your trees during compilation?


Andrei
1 2 3 4 5
Top | Discussion index | About this forum | D home