On Saturday, 21 July 2012 at 21:44:52 UTC, Andrei Alexandrescu wrote:

>
> Please chime in with thoughts. Would someone want to pioneer this project?
>

There are some questions.

1. static info = getModuleInfo("std.algorithm");

The language does not allow you to use CTFE parameter values as arguments to __traits/templates. Therefore, to be able to build meta-objects at compile-time, you would have to:

static info = getModuleInfo!"std.algorithm";

2. Then, what is the preferred way of referencing compiler objects - with strings or directly:

import std.algorithm;
static info = getModuleInfo!(std.algorithm);
?

I think using strings while we have direct access to those objects is not an incredibly good idea. Those strings will be mixed-in by getXxxInfo anyway:

ModuleInfo getModuleInfo(string name)()
{
    ...
    mixin("import " ~ name ~ ";");
    mixin("alias __traits(getMembers, " ~ name ~ ") members;");
    foreach (m; members)
    ...
}

3. auto info = getModuleInfo("std.algorithm");

There is no way to mutate global data structures at compile-time, therefore you would have to build, at run time, a data structure aggregating all meta-objects coming from various modules.

That could be achieved with static constructors:

module std.reflection;

// application-wide module info registry
private ModuleInfo[string] moduleInfos;

// run-time module info getter
string getModuleInfo(string s)
{
    return moduleInfos[s];
}

string makeModuleInfoAvailableDynamically()
{
    return q{
        shared static this() { shared static mi = getModuleInfo!(__traits(thisModule));
            moduleInfos[mi.name] = mi; }
    };
}

Problematic because mixing-in makeModuleInfoAvailableDynamically would mean that circular imports are no longer allowed for that module. Remember the whining babies complaining about this very use case a while ago?

What about changing the language so that static constructors marked with @system are exempted from circular dependency checking?

3. If you are against inheritance, why classes and not structs?

4. How the whole thing is intended to interact with dynamic link libraries?

On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu wrote:
> Yah, ideally all entities definable in a D module should be available via reflection. But I focused on things that e.g. the user of a dynamically-loaded library would be interested in: functions, classes.

Plugins are usually done by providing factory methods that create needed objects, the methods are usually obtained with dlsym.

On 7/22/12, Philippe Sigaud <philippe.sigaud@gmail.com> wrote:
> 2) Why classes, as opposed to structs?

I think either way you'd need reference semantics. For example maybe you're doing code-generation at compile-time but you need to rename a class name in a typeinfo returned by std.reflection before doing any processign. With reference semantics you only have to change one class and all other types which refer to such a class will have access to the new name.

You could use structs as well, and actually I use structs in my codegenerator (with a similar layout to what Andrei posted). Each struct (e.g. Class/Function) stores Symbols, which are structs with an ID and a Type. I can look up each symbol in a SymTable which actually holds the structures with data. So a Symbol is like a fake pointer, and the SymTable would be the memory. I originally planned to use classes but some serialization frameworks didn't work with those so I settled using structs and a bit of template mixin magic instead.

All of this std.reflection talk is quite exciting actually. If you had AST information about your entire D library you could do some really cool things. You could make a better documentation generator than ddoc, or export the AST into a file for a code-completion plugin, or create a wrapper C library which enables other languages to use your D library.

"Kagamin" <spam@here.lot> writes:

> On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu wrote:
>> Yah, ideally all entities definable in a D module should be available via reflection. But I focused on things that e.g. the user of a dynamically-loaded library would be interested in: functions, classes.
>
> Plugins are usually done by providing factory methods that create needed objects, the methods are usually obtained with dlsym.

That's what you'd do in a language that doesn't have something like D's Object.factory(). In D's case, however, you'd have the "factory method" be the ctor.

That's all hypothetical, however, since there's no D ABI for shared objects yet...

-- 
The volume of a pizza of thickness a and radius z can be described by the following formula:

pi zz a

On Sun, Jul 22, 2012 at 4:28 PM, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> std.reflection could become the lynchpin for dynamic library use; once the
> library is loaded (with dlopen or such), the client needs to call
> getModuleInfo() (a C function that can be found with dlsym()) and then get
> access to pointers to functions necessary for doing all other work. (Note
> that my examples don't yet include pointers to executable code yet.)

I wouldn't know. I have no experience with dlsym().

>> 2) Why classes, as opposed to structs? Would inheritance/hierarchies
>> play a role there? Or reference semantics?
>> Note that between structs, classes, templates and modules there is a
>> lot of redundancy. A possibility could be to have an AggregateInfo
>> base class.
>
>
> Initially I used struct, but then I figured reference semantics are more natural for storing cross-entity information, as you indeed took advantage of in your TemplateInfo.

I realized a few minutes after posting I answered my own question: because this is a self-referencing structure (a tree, a graph), which are easier to code with classes than structs.


>> 4) How would that allows queries like "Here is class C, give me all its available subclasses."? Hmm, wait, I get it: extract classes from the module, and recursively from imported modules. From these classes, extract the parent classes and so on, until the search ranged over the whole inheritance tree. I guess inheritance info could be standard enough for std.reflection to provide such a search.
>
>
> Something like that. Note that such a query is not particularly OO-ish, because getting a class' cone (totality of subclasses) works against the modularity that inheritance is meant for. I don't think we should make getting class cones particularly easy.

Right. Since people here asked this question, I thought that was a common request in OO. I'm more a structs and mixins guy, myself.


>> 5) The compiler can emit JSON output giving a partial view of the same information. As a long-term goal, I suggest these infos should be compatible somehow. The JSON output should be enriched, and we should ascertain that using std.json to read this kind of automatically-generated information should give std.reflection infos back.
>
>
> Yes, that's a great connection that Walter and I discussed a bit.

Good to know.


>> 6) Is really all the necessary info available through std.traits and __traits? Imported modules and their subtilities (renamed functions, etc) seem out of reach, no?
>
>
> We'll need indeed to enhance __traits with what's needed. Much of the point of std.reflection is to determine exactly what's there and what's needed. And that starts with the data structures design (the algorithmic aspects are minor).

+1 for enhancing __traits locally.

- having __traits(allMembers, xxx) work on simple module names, and
not only on qualified package.module names
- having a way to get imports and a way to know whether they are
static / renaming import


As for the data structures, other people's designs allured to in this thread seem similar to your proposal.

I have two other questions:

About functions: should they be subject to reflection also, or not? They have no fields, inner functions are totally interned, etc. All a user need is the 'interface', right? (name, and everything that's in the type: return type, parameters, purity, etc)

About imports, what about inner imports, now that they are authorized in almost any scope? My gut feeling right now is that a user does not care if class C internally import std.algorithm in one of its methods, but I could be wrong.


>> 7) I know your examples are not complete, but don't forget aliases and symbols, and module-level values. Since these can be of any type, I'm not sure how they are managed in your scheme. I mean, you cannot have IntInfo[], DoubleInfo[], ...
>
>
> I sort of eschewed part of that by using strings for types.

I see. They can be managed like fields in an aggregate (struct / classes), as there are many similarities between D modules and classes / structs.

class ModuleInfo {
@property:
...
   FieldInfo[] data; // also used in StructInfo and ClassInfo
}

class FieldInfo {
@property:
    string name();
    bool isStatic();
    Protection protection();
    string type();
}

As long as this info is available at CT, FieldInfo.type can be mixed-in and used in code.

what I'm not sure I get in your design is why some informations are encoded in their own structure, like Protection above (the code is copy-pasted from yours, I'd guess Protection is an enumeration), and then some others are encoded as strings (types). Is that because the values Protection can take are known in advance (and finite)?

I wondered whether a design like this could be interesting?:

abstract class FieldInfo {}

class Field(T) : FieldInfo {
@property:
    string name();
    bool isStatic();
    Protection protection();
    alias T Type;
}

But that doesn't cut it, AFAICT: different fields can be stored in a FieldInfo[] array, but the type information is not easier to get, anyway. So forget it.


This kind of manipulation is why I got interested in fully polymorphic trees (tuples of tuples...), able to store any value, while keeping the type information visible. The drastic consequence is to have a tree type depend on its entire content. But I'm coming from Static Typing Land here, whereas this introspection stuff is more dynamic.

Anyway, back to gobal values: aliases should be there also. A simple AliasInfo class?


> Well you're the resident crazy-stuff-during-compilation guy.

Ah! I wish.

I had this wonderful idea of having code be parsed at CT, semantically analyzed, transformed into some machine code at CT and... , oh wait.


> Did you try your trees during compilation?

Just did. They fail :) Either segmentation fault (core dumped) or an
error telling me class literals  cannot be returned from CTFE.
Hmm, this is old code. I'll have a look since in other projects, I can
obtain trees at CT.


Philippe

On 2012-07-21 23:44, Andrei Alexandrescu wrote:
> Walter and I discussed the idea below a long time (years) ago. Most
> likely it's also been discussed in this newsgroup a couple of times.
> Given the state of the compiler back then, back then it seemed like a
> super cool idea that's entirely realizable, it would just take time for
> the compiler to become as capable as needed. Nowadays we're in shape to
> tackle it.
>
> Here "it" is.
>
> Back when runtime reflection was being discussed, my response was "let's
> focus on compile-time reflection, and then we can do run-time reflection
> on demand as a library". Though this might sound sensible, I initially
> didn't have a design. Now here's what we can do.

I've been waiting for this :)

-- 
/Jacob Carlborg

On 2012-07-22 02:16, Kapps wrote:
> I agree with most things proposed, however I am not a fan of the idea of
> mixing in runtime reflection info. Many times, you want reflection info
> from a type that is not your own, and thus I believe reflection should
> be generated by specifying a type. More importantly, a method should
> exist for recursively generating reflection info.

I agree.

-- 
/Jacob Carlborg

On Sun, Jul 22, 2012 at 5:10 PM, Max Samukha <maxsamukha@gmail.com> wrote:

> The language does not allow you to use CTFE parameter values as arguments to __traits/templates. Therefore, to be able to build meta-objects at compile-time, you would have to:
>
> static info = getModuleInfo!"std.algorithm";

Maybe I don't get your comment, but AFAICT, the language does allow you to use CTFE parameters values as arguments to templates:

template Twice(double d)
{
    enum Twice = d * 2;
}

double foo(double d)
{
    return d+1.0;
}

void main()
{
    enum t = Twice!(foo(1.0));
    pragma(msg, t);
}

On 2012-07-22 14:04, deadalnix wrote:

> I'd expect from std.reflection that it is able to reflect recursively
> from the marked starting point.

I really hope so.

-- 
/Jacob Carlborg

On 2012-07-22 06:48, Andrei Alexandrescu wrote:

> P.S. What this thing wit quoting a long message to make a 1-line point?
> Is that a thing?

It's the new hip thing :)

-- 
/Jacob Carlborg