The subject of compile-time reflection has been an important one to me. I have been musing on it since about the time I started writing Pyd. Here is the current state of my thoughts on the matter.

----
Functions
----

When talking about functions, a given symbol may refer to multiple functions:

void foo() {}
void foo(int i) {}
void foo(int i, int j, int k=20) {}

The first thing a compile-time reflection mechanism needs is a way to, given a symbol, derive a tuple of the signatures of the function overloads. There is no immediately obvious syntax for this.

The is() expression has so far been the catch-all location for many of D's reflection capabilities. However, is() operates on types, not arbitrary symbols.

A property is more promising. Re-using the .tupleof property is one idea:

foo.tupleof => Tuple!(void function(), void function(int), void function(int, int, int))

However, I am not sure how plausible it is to have a property on a symbol like this. Another alternative is to have some keyword act as a function (as typeof and typeid do, for instance). I propose adding "tupleof" as an actual keyword:

tupleof(foo) => Tuple!(void function(), void function(int), void function(int, int, int))

I will be using this syntax throughout the rest of this post. For the sake of consistency, tupleof(Foo) should do what Foo.tupleof does now.

To umabiguously refer to a specific overload of a function, two pieces of information are required: The function's symbol, and the signature of the overload. When doing compile-time reflection, one is typically working with one specific overload at a time. While a function pointer does refer to one specific overload, it is important to note that function pointers are not compile-time entities! Therefore, the following idiom is common:

template UseFunction(alias func, func_t) {}

That is, any given template that does something with a function requires both the function's symbol and the signature of the particular overload to operate on to be useful.

It should be clear, then, that automatically deriving the overloads of a given function is very important. Another piece of information that is useful is whether a given function has default arguments, and how many. The tupleof() syntax can be re-used for this:

tupleof(foo, void function(int, int, int)) => Tuple!(void function(int, int))

Here, we pass tupleof() the symbol of a function, and the signature of a particular overload of that function. The result is a tuple of the various signatures it is valid to call the overload with, ignoring the /actual/ signature of the function. The most useful piece of information here is the /number/ of elements in the tuple, which will be equal to the number of default arguments supported by the overload.

One might be tempted to place these additional function signatures in the original tuple derived by tupleof(foo). However, this is not desirable. Consider: We can say any of the following:

void function() fn1 = &foo;
void function(int) fn2 = &foo;
void function(int, int, int) fn3 = &foo;

But we /cannot/ say this:

void function(int, int) fn4 = &foo; // ERROR!

A given function-symbol therefore has two sets of function signatures associated with it: The actual signatures of the functions, and the additional signatures it may be called with due to default arguments. These two sets are not equal in status, and should not be treated as such.

----
Member functions
----

Here is where things get really complicated.

class A {
    void bar() {}
    void bar(int i) {}
    void bar(int i, int j, int k=20) {}

    void baz(real r) {}

    static void foobar() {}
    final void foobaz() {}
}

class B : A {
    void foo() {}
    override void baz(real r) {}
}

D does not really have pointers to member functions. It is possible to fake them with some delegate trickery. In particular, there is no way to directly call an alias of a member function. This is important, as I will get to later.

The first mechanism needed is a way to get all of the member functions of a class. I suggest the addition of a .methodsof class property, which will derive a tuple of aliases of the class's member functions.

A.methodsof => Tuple!(A.bar, A.baz, A.foobar, A.foobaz)
B.methodsof => Tuple!(A.bar, A.foobar, A.foobaz, B.foo, B.baz)

The order of the members in this tuple is not important. Inherited member functions are included, as well. Note that these are tuples of symbol aliases! Since these are function symbols, all of the mechanisms suggested earlier for regular function symbols should still work!

tupleof(A.bar) => Tuple!(void function(), void function(int), void function(int, int, int))

And so forth.

There are three kinds of member functions: virtual, static, and final. The next important mechanism that is needed is a way to distinguish these from each other. An important rule of function overloading works in our favor, here: A given function symbol can only refer to functions which are all virtual, all static, or all final. Therefore, this should be considered a property of the symbol, as opposed to one of the function itself.

The actual syntax for this mechanism needs to be determined. D has 'static' and 'final' keywords, but no 'virtual' keyword. Additionally, the 'static' keyword has been overloaded with many meanings, and I hesitate suggesting we add another. Nonetheless, I do.

static(A.bar == static) == false
static(A.bar == final) == false
static(A.bar == virtual) == true

The syntax is derived from that of the is() expression. The grammar would look something like this:

StaticExpression:
	static ( Symbol == SymbolSpecialization )

SymbolSpecialization:
	static
	final
	virtual

Here, 'virtual' is a context-sensitive keyword, not unlike the 'exit' in 'scope(exit)'. If the Symbol is not a member function, it is an error.

A hole presents itself in this scheme. We can get all of the function symbols of a class's member functions. From these, we can get the signatures of their overloads. From /these/, can get get pointers to the member functions, do some delegate trickery, and actually call them. This is all well and good.

But there is a problem when a method has default arguments. As explained earlier, we can't do this:

// Error! None of the overloads match!
void function(int, int) member_func = &A.bar;

Even though we can say:

A a = new A;
a.bar(1, 2);

The simplest solution is to introduce some way to call an alias of a method directly. There are a few options. My favorite is to take a cue from Python, and allow the following:

alias A.bar fn;
A a = new A;
fn(a, 1, 2);

That is, allow the user to explicitly call the method with the instance as the first parameter. This should be allowed generally, as in:

A.bar(a);
A.baz(a, 5.5);

Given these mechanisms, combined with the existing mechanisms to derive the return type and parameter type tuple from a function type, D's compile-time reflection capabilities would be vastly more powerful.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org

Kirk McDonald wrote:
<snip>
> The first thing a compile-time reflection mechanism needs is a way to, given a symbol, derive a tuple of the signatures of the function overloads. There is no immediately obvious syntax for this.

Overloads are entirely separate functions. If you go through ClassInfo.vtbl, that gives you each overload separately. And the only thing overloads have in common are their names. So is this actually important?

The annoyance is when you have overloads, how do you refer to the correct one?

---
void foo(int i) {}
void foo(int i, char[] str) {}
auto f1 = &foo; // works, but which does it give?
void function(int, char[]) f2 = &foo; // does the right thing
---

That works fine, as long as one person wrote it all specifically for the cases it's used in. What if you're sending a function reference to some other piece of code that can take any function?

---
void foo(int i) {}
void foo(int i, char[] str) {}
some_other_lib.use_some_function(&foo);
---

Well, you can do the obvious workaround:

---
void function(int) foo_to_send = &foo;
some_other_lib.use_some_function(&foo_to_send);
---

It just bothers me that the only way to specify which reference is by assignment to a function reference with a specific signature. I'd prefer a syntax more like:
auto f1 = &foo(int, char[]);

Unambiguous, since references don't have an opCall, and shorter. But the present workaround is merely annoying, and only mildly at that.

<snip>
> foo.tupleof => Tuple!(void function(), void function(int), void function(int, int, int))

Okay, sounds simple enough, but why do you need each overload? They're separate functions; they don't have anything in common, strictly speaking, except the name.

> It should be clear, then, that automatically deriving the overloads of a given function is very important. Another piece of information that is useful is whether a given function has default arguments, and how many. The tupleof() syntax can be re-used for this:
> 
> tupleof(foo, void function(int, int, int)) => Tuple!(void function(int, int))

This is interesting. Currently, there's no way to get which arguments are omissible.

...
> D does not really have pointers to member functions. It is possible to fake them with some delegate trickery. In particular, there is no way to directly call an alias of a member function. This is important, as I will get to later.

The reason for this is obvious; the compiler rewrites the functions as you describe below. I haven't looked, but I'd guess a delegate is something like this:
struct delegate (T, R, U...) {
   void* func;
   T obj;
   R opCall(U u) { return *func(obj, u); }
}

...
> A.methodsof => Tuple!(A.bar, A.baz, A.foobar, A.foobaz)
> B.methodsof => Tuple!(A.bar, A.foobar, A.foobaz, B.foo, B.baz)

You can already get this through A.classinfo.vtbl. That contains all functions that are valid for the class. You can create a template that will determine which functions are inherited, though you cannot say which override functions from the base class with any certainty.

> static(A.bar == static) == false
> static(A.bar == final) == false
> static(A.bar == virtual) == true

Currently, there's no way I know of to get this information. The is(typeof()) system works with all functions, static or not, whether you use them from a type or from an instance.

Hack? Do everything from instance variables. Dunno what to do about final, though.

Checking about final stuff matters if you want your program to behave differently toward final methods, but nothing in ClassInfo prevents you from replacing a final method.

Christopher Wright wrote:
> Kirk McDonald wrote:
> <snip>
> 
>> The first thing a compile-time reflection mechanism needs is a way to, given a symbol, derive a tuple of the signatures of the function overloads. There is no immediately obvious syntax for this.
> 
> 
> Overloads are entirely separate functions. If you go through ClassInfo.vtbl, that gives you each overload separately. And the only thing overloads have in common are their names. So is this actually important?
> 

I am interested in compile-time reflection. The vtable doesn't exist at compile-time. The only way to refer to a function at compile-time is through a combination of the symbol name and the signature.

Anything involving function pointers is necessarily a runtime operation.

[snip]
> <snip>
> 
>> foo.tupleof => Tuple!(void function(), void function(int), void function(int, int, int))
> 
> 
> Okay, sounds simple enough, but why do you need each overload? They're separate functions; they don't have anything in common, strictly speaking, except the name.
> 

Same reason: It is a strictly compile-time operation.

>> It should be clear, then, that automatically deriving the overloads of a given function is very important. Another piece of information that is useful is whether a given function has default arguments, and how many. The tupleof() syntax can be re-used for this:
>>
>> tupleof(foo, void function(int, int, int)) => Tuple!(void function(int, int))
> 
> 
> This is interesting. Currently, there's no way to get which arguments are omissible.
> 

Not entirely true; see std.bind.minNumArgs (which is a variadic version, by h3r3tic, of a non-variadic template I wrote before we had tuples). I am simply proposing making this a language feature, rather than the hack it currently is.

> ...
> 
>> D does not really have pointers to member functions. It is possible to fake them with some delegate trickery. In particular, there is no way to directly call an alias of a member function. This is important, as I will get to later.
> 
> 
> The reason for this is obvious; the compiler rewrites the functions as you describe below. I haven't looked, but I'd guess a delegate is something like this:
> struct delegate (T, R, U...) {
>    void* func;
>    T obj;
>    R opCall(U u) { return *func(obj, u); }
> }
> 
> ...
> 

The ABI is different; the instance is passed in a register, or somesuch.

>> A.methodsof => Tuple!(A.bar, A.baz, A.foobar, A.foobaz)
>> B.methodsof => Tuple!(A.bar, A.foobar, A.foobaz, B.foo, B.baz)
> 
> 
> You can already get this through A.classinfo.vtbl. That contains all functions that are valid for the class. You can create a template that will determine which functions are inherited, though you cannot say which override functions from the base class with any certainty.
> 

Again, the vtable is not compile-time information.

>> static(A.bar == static) == false
>> static(A.bar == final) == false
>> static(A.bar == virtual) == true
> 
> 
> Currently, there's no way I know of to get this information. The is(typeof()) system works with all functions, static or not, whether you use them from a type or from an instance.
> 
> Hack? Do everything from instance variables. Dunno what to do about final, though.
> 
> Checking about final stuff matters if you want your program to behave differently toward final methods, but nothing in ClassInfo prevents you from replacing a final method.

My interest is in improving Pyd. Given a class, I want to generate a Python type wrapping the entirety of that class, as well as a D subclass of the type, providing virtual dispatching to the Python type. This means I effectively have to know /everything/ about the class and its methods, at compile-time.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org

Reply to Kirk,

> void foo() {}
> void foo(int i) {}
> void foo(int i, int j, int k=20) {}
> The first thing a compile-time reflection mechanism needs is a way to,
> given a symbol, derive a tuple of the signatures of the function
> overloads. 
[...]
> 
> foo.tupleof => Tuple!(void function(), void function(int), void
> function(int, int, int))
> 

Why have this give a type tuple? I'd have it give alias to the overloads them selves.

BCS wrote:
> Reply to Kirk,
> 
>> void foo() {}
>> void foo(int i) {}
>> void foo(int i, int j, int k=20) {}
>> The first thing a compile-time reflection mechanism needs is a way to,
>> given a symbol, derive a tuple of the signatures of the function
>> overloads. 
> 
> [...]
> 
>>
>> foo.tupleof => Tuple!(void function(), void function(int), void
>> function(int, int, int))
>>
> 
> Why have this give a type tuple? I'd have it give alias to the overloads them selves.
> 
> 

Aliases operate on symbols. All of those functions have the same symbol. An alias to a specific overload is nonsensical.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org

Kirk McDonald wrote:
<snip>

It would be great to have the functionality you propose in D.

> 
> There are three kinds of member functions: virtual, static, and final. The next important mechanism that is needed is a way to distinguish these from each other. An important rule of function overloading works in our favor, here: A given function symbol can only refer to functions which are all virtual, all static, or all final. Therefore, this should be considered a property of the symbol, as opposed to one of the function itself.
> 
> The actual syntax for this mechanism needs to be determined. D has 'static' and 'final' keywords, but no 'virtual' keyword. Additionally, the 'static' keyword has been overloaded with many meanings, and I hesitate suggesting we add another. Nonetheless, I do.
> 
> static(A.bar == static) == false
> static(A.bar == final) == false
> static(A.bar == virtual) == true

This looks very confusing to me, however, if you'd replace the first static with 'is' it makes sense:

is(A.bar == static) == false
is(A.bar == final) == false
is(A.bar == virtual) == true

Is there a problem with this, grammar-wise?

Btw, if introducing 'virtual' is undesirable, it can be left out since it can be inferred from being final nor static, and for this some library side syntactic sugar can be made.

Lutger wrote:
> Kirk McDonald wrote:
> <snip>
> 
> It would be great to have the functionality you propose in D.
> 
>>
>> There are three kinds of member functions: virtual, static, and final. The next important mechanism that is needed is a way to distinguish these from each other. An important rule of function overloading works in our favor, here: A given function symbol can only refer to functions which are all virtual, all static, or all final. Therefore, this should be considered a property of the symbol, as opposed to one of the function itself.
>>
>> The actual syntax for this mechanism needs to be determined. D has 'static' and 'final' keywords, but no 'virtual' keyword. Additionally, the 'static' keyword has been overloaded with many meanings, and I hesitate suggesting we add another. Nonetheless, I do.
>>
>> static(A.bar == static) == false
>> static(A.bar == final) == false
>> static(A.bar == virtual) == true
> 
> 
> This looks very confusing to me, however, if you'd replace the first static with 'is' it makes sense:
> 
> is(A.bar == static) == false
> is(A.bar == final) == false
> is(A.bar == virtual) == true
> 
> Is there a problem with this, grammar-wise?
> 
> Btw, if introducing 'virtual' is undesirable, it can be left out since it can be inferred from being final nor static, and for this some library side syntactic sugar can be made.
> 

Grammar wise, using 'virtual' would cause some problems. (What if you have a type called 'virtual' and want to compare some other type to it?) There is another, more serious, problem: is() is defined such that if the first thing passed to it isn't a type, it returns false. (That is, one of its purposes is to test if something is a valid type.) I think that making an exception to this rule for 'static' and 'final' could potentially be confusing. Therefore, some other keyword is required. I picked 'static' more or less arbitrarily. It also allows the use of 'virtual' directly.

-- 
Kirk McDonald
http://kirkmcdonald.blogspot.com
Pyd: Connecting D and Python
http://pyd.dsource.org

Kirk McDonald wrote:
...
>> This looks very confusing to me, however, if you'd replace the first static with 'is' it makes sense:
>>
>> is(A.bar == static) == false
>> is(A.bar == final) == false
>> is(A.bar == virtual) == true
>>
>> Is there a problem with this, grammar-wise?
>>
>> Btw, if introducing 'virtual' is undesirable, it can be left out since it can be inferred from being final nor static, and for this some library side syntactic sugar can be made.
>>
> 
> Grammar wise, using 'virtual' would cause some problems. (What if you have a type called 'virtual' and want to compare some other type to it?) There is another, more serious, problem: is() is defined such that if the first thing passed to it isn't a type, it returns false. (That is, one of its purposes is to test if something is a valid type.) I think that making an exception to this rule for 'static' and 'final' could potentially be confusing. Therefore, some other keyword is required. I picked 'static' more or less arbitrarily. It also allows the use of 'virtual' directly.
> 

I see. Well, it's not that important, the big thing is if the functionality will be included. Still I would argue that it is preferable to use 'is', because it's both clearer and more consistent.

The problem with is() returning false when the symbol tested isn't a type occurs now too, I don't understand how this is different from current cases with static / final / virtual?

If we need to hose virtual, one would have to write:
!(is(A.bar == static) || is(A.bar == final))

This is less than ideal, but I can live with that for the benefit it gives. In practice I would wrap it anyway in some traits module.

Time for a response without reading the entire message 8-D

http://www.dsource.org/projects/tango.tools/browser/trunk/tools/rodin

(FYI: Rodin will work with either Tango or Phobos)

 - Gregor Richards

Reply to Kirk,

> BCS wrote:
> 
>> Reply to Kirk,
>> 
>>> void foo() {}
>>> void foo(int i) {}
>>> void foo(int i, int j, int k=20) {}
>>> The first thing a compile-time reflection mechanism needs is a way
>>> to,
>>> given a symbol, derive a tuple of the signatures of the function
>>> overloads.
>> [...]
>> 
>>> foo.tupleof => Tuple!(void function(), void function(int), void
>>> function(int, int, int))
>>> 
>> Why have this give a type tuple? I'd have it give alias to the
>> overloads them selves.
>> 
> Aliases operate on symbols. All of those functions have the same
> symbol. An alias to a specific overload is nonsensical.
> 

I would clam that an alias to a specific overload being nonsensical should in it's self be nonsensical. Why shouldn't I be able to get an alias to a specific function even if it is overloaded?

I would propose that an unqualified function name (or an alias of it) would be the set of overloads (or alias to them) and a qualified name would be a specific function.

I would be interested in way this would be a bad idea.

void foo(int i){}
void foo(Object i){}

alias foo bar; // bar == {foo(int) and foo(Object)}

int i;
bar(i); // works
bar(null); // works

// proposed syntax for overload resolution
alias foo(Object) baz;

baz(i); // fails int i can't be converted to Object
baz(null); // works