> Thank you, I'm glad you agree.

Of course. Reflection should _never_ expose private stuff.

> Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a dynamic-object for certain fields/functions.

Hmmm..... Yeap. Exactly. Anything else i would say would be a derivative of it. I can't do it currently. Also, as discussed before, i should never see private stuff this way (any any other way).

> Would dynamic overload be a subset of prototype-style objects?

Huh? Dynamic overloading is basically multi-methods. It chooses the
implementation based on run-time type of the parameters.
This could be further improved by allowing overloading, based on
arbitrary predicates (much like template constraints).

> Would dynamic modules simply container for dynamic functions/objects?

Basically, yes. Dynamic module would contain typemically typed variables, functions, and types.

>  Something more?

Cant thing of anything else ATM. Will tell you if i do.

On Tue, Oct 25, 2011 at 8:29 PM, Robert Jacques <sandford@jhu.edu> wrote:
> On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
>>
>> I didn't mean reflection for violating access specifications. In that case, of course, it's error-prone.
>
> Thank you, I'm glad you agree.
>
>> What i meant was a set of dynamic counterparts of static features of D:
>> * Dynamic interfaces
>> * Dynamic overloading
>> * Dynamic modules
>> * ...
>> That's what i had in mind.
>
> Care to explain what would each of those should look like?
>
> Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a dynamic-object for certain fields/functions.
>
> Would dynamic overload be a subset of prototype-style objects?
>
> Would dynamic modules simply container for dynamic functions/objects? Something more?
>

On 10/25/11 1:05 PM, Robert Jacques wrote:
> On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan
> <gor.f.gyolchanyan@gmail.com> wrote:
>> As i said before, enabling reflection manually is an act of enabling a
>> certain range of stuff you can do with the code. Forgetting to enable
>> reflection can lead to serious problems, when the reflection is
>> required.
>
> And when, precisely, is reflection _required_? Granted, in other
> languages it is the only tool for the job. But we have a very big
> toolkit in D, and so far, I have yet to see any example (including
> Rails) which _requires_ the holistic runtime reflection proposed in
> these threads.

This is completely true. Something like:

class Person < ActiveRecord::Base
  belongs_to :company
end

could be done at compile-time. Same as retrieving the schema from mysql: could be done at compile-time, like Nemerle does (or so I remember someone said that).

Me too, believes that greater compile-time capabilities will make almost all (but never all) runtime reflection unnecessary.

Am 25.10.11 16:41, schrieb Robert Jacques:
> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>> Hi,
>>
>> I would love to see runtime reflection be available in D. I am
>> currently reading Walter Bright's "The D
>> Programming Language" and with every page I read I became more and
>> more fascinated and enthusiastic
>> about D. But then I started to wonder why I haven't read about runtime
>> reflection yet, so I did some Internet
>> research and was a bit disappointed this feature isn't available. D is
>> really cool, it's not only C done right,
>> but also Java and C#. Actually, to me, it's one of the best
>> programming language available yet. It's really a
>> pitty RTTI is not available. So I hope your approach will somehow
>> influence D's future :)
>>
>> Keep going,
>> Jonny
>
> Hi Jonny,
> Well, some RTTI is available. How much RTTI is needed, really depends on
> what you're trying to do. Therefore whether D has or hasn't got RTTI
> depends heavily on what any particular person means by RTTI. Do you have
> some use cases in mind? Maybe prototype objects, duck-typing/casting or
> serialization? Or perhaps something you wrote?
>
> Sorry for all the questions, but I feel as if we don't have a resident
> reflection expert in the dialog, or even the opinions of
> basic/experienced users. We all know that 'RTTI' is good, but we're a
> little vague on the how and why. (Well, beyond the use cases above)

Hi Robert,

Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
"
[...]
- Discover and modify source code constructions (such as code blocks, classes, methods, protocols, etc.) as a first-class object at runtime.
- Convert a string matching the symbolic name of a class or function into a reference to or invocation of that class or function.
[...]
"

Here is what I would dream of for arbitrary objects/classes (not necessarily known at compile-time):
- Query an object for its list of methods together with their signatures. Select a method, bind some values to its arguments, call it, and retrieve the return type (if any).
- Query an object for its public fields (at least), and provide a way to get/set their values.
- Query an object's class for all implemented interfaces and its base class.
- Query a module for all type definitions and provide a way to introspect these types in more detail. For instance, it would be really cool if I could find a class with name "Car" in module "cars", get a list of all defined constructors, select one, bind values to the constructor's parameters, and create a corresponding object.

Now you might ask where is the use case. I think, a very important one, besides creating GUI designers, is to be able to implement a Dependency Injection container [2,3,4] like, for example, the one provided by the Spring Framework for Java or .NET. In short, they allow you define how object trees should be build up solely by specifying this tree in an XML file. Within the XML file you can create new instances of classes of arbitrary types. You can specify which constructors to use for instantiation, and what values the constructor's arguments (if any) should have. You can even bind such an argument to an object previously created by the XML definition. You can call methods on existing objects, e.g. in order to call setter to further initialize an object with values you also define in that XML file. That XML file can be loaded when your program starts an make the DI container component instantiate all needed objects for you. There is no need for recompilation if the XML file is changed. Just restart your application.

Implementing such a DI container heavily depends on reflection, because the DI container component doesn't know anything about the objects to be created during runtime.

Qt also extends C++ with a reflection mechanism through the help of its meta object compiler (moc). It analyses the C++ source code, generates meta class definitions [6,7] and weaves them into your Qt class. Hence, in Qt, you can query an object for fields, methods, interfaces, etc. and you can call methods with arbitrary parameters, or you can instantiate a class using an arbitrary constructor. Consequently, somone implemented a DI container for C++ which is based on Qt and works more or less the same way the Spring DI container does. You can build up object trees simply by specifying such trees in an XML file.

I don't go into why dependency injection is a very powerful feature. This is Martin Fowler's [3] job ;) But when I program with C++ I miss such a flexible dependency injection mechanism a lot. And I hope this will eventually be available for D.

Cheers,
Jonny

[1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
[2] http://en.wikipedia.org/wiki/Dependency_injection
[3] http://martinfowler.com/articles/injection.html
[4] http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
[5] http://qtioccontainer.sourceforge.net/
[6] http://doc.qt.nokia.com/stable/qmetaobject.html
[7] http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
> Am 25.10.11 16:41, schrieb Robert Jacques:
>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>>> Hi,
>>>
>>> I would love to see runtime reflection be available in D. I am
>>> currently reading Walter Bright's "The D
>>> Programming Language" and with every page I read I became more and
>>> more fascinated and enthusiastic
>>> about D. But then I started to wonder why I haven't read about runtime
>>> reflection yet, so I did some Internet
>>> research and was a bit disappointed this feature isn't available. D is
>>> really cool, it's not only C done right,
>>> but also Java and C#. Actually, to me, it's one of the best
>>> programming language available yet. It's really a
>>> pitty RTTI is not available. So I hope your approach will somehow
>>> influence D's future :)
>>>
>>> Keep going,
>>> Jonny
>>
>> Hi Jonny,
>> Well, some RTTI is available. How much RTTI is needed, really depends on
>> what you're trying to do. Therefore whether D has or hasn't got RTTI
>> depends heavily on what any particular person means by RTTI. Do you have
>> some use cases in mind? Maybe prototype objects, duck-typing/casting or
>> serialization? Or perhaps something you wrote?
>>
>> Sorry for all the questions, but I feel as if we don't have a resident
>> reflection expert in the dialog, or even the opinions of
>> basic/experienced users. We all know that 'RTTI' is good, but we're a
>> little vague on the how and why. (Well, beyond the use cases above)
>
> Hi Robert,
>
> Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
> "
> [...]
> - Discover and modify source code constructions (such as code blocks,
> classes, methods, protocols, etc.) as a first-class object at runtime.
> - Convert a string matching the symbolic name of a class or function
> into a reference to or invocation of that class or function.
> [...]
> "
>
> Here is what I would dream of for arbitrary objects/classes (not
> necessarily known at compile-time):
> - Query an object for its list of methods together with their
> signatures. Select a method, bind some values to its arguments, call it,
> and retrieve the return type (if any).
> - Query an object for its public fields (at least), and provide a way to
> get/set their values.
> - Query an object's class for all implemented interfaces and its base class.
> - Query a module for all type definitions and provide a way to
> introspect these types in more detail. For instance, it would be really
> cool if I could find a class with name "Car" in module "cars", get a
> list of all defined constructors, select one, bind values to the
> constructor's parameters, and create a corresponding object.
>
> Now you might ask where is the use case. I think, a very important one,
> besides creating GUI designers, is to be able to implement a Dependency
> Injection container [2,3,4] like, for example, the one provided by the
> Spring Framework for Java or .NET. In short, they allow you define how
> object trees should be build up solely by specifying this tree in an XML
> file. Within the XML file you can create new instances of classes of
> arbitrary types. You can specify which constructors to use for
> instantiation, and what values the constructor's arguments (if any)
> should have. You can even bind such an argument to an object previously
> created by the XML definition. You can call methods on existing objects,
> e.g. in order to call setter to further initialize an object with values
> you also define in that XML file. That XML file can be loaded when your
> program starts an make the DI container component instantiate all needed
> objects for you. There is no need for recompilation if the XML file is
> changed. Just restart your application.
>
> Implementing such a DI container heavily depends on reflection, because
> the DI container component doesn't know anything about the objects to be
> created during runtime.
>
> Qt also extends C++ with a reflection mechanism through the help of its
> meta object compiler (moc). It analyses the C++ source code, generates
> meta class definitions [6,7] and weaves them into your Qt class. Hence,
> in Qt, you can query an object for fields, methods, interfaces, etc. and
> you can call methods with arbitrary parameters, or you can instantiate a
> class using an arbitrary constructor. Consequently, somone implemented a
> DI container for C++ which is based on Qt and works more or less the
> same way the Spring DI container does. You can build up object trees
> simply by specifying such trees in an XML file.
>
> I don't go into why dependency injection is a very powerful feature.
> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
> such a flexible dependency injection mechanism a lot. And I hope this
> will eventually be available for D.
>
> Cheers,
> Jonny
>
> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
> [2] http://en.wikipedia.org/wiki/Dependency_injection
> [3] http://martinfowler.com/articles/injection.html
> [4]
> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
> [5] http://qtioccontainer.sourceforge.net/
> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
> [7]
> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx

Hi Jonny,
Thank you for your informative (and well cited) post. It has provided me with a new take on an old design pattern and some enjoyable reading. In return, let me outline my opinion of reflection in D today, and tomorrow, as it pertains to your wish list.

Reflection in D today is very different from the host of VM languages that have popularized the concept. Being a compiled systems language, actual runtime self-modification is too virus like to become at a language level feature. However, given the compilation speed of D, people have made proof of concept libraries that essentially wrapped the compiler and dynamically loaded the result. As LDC uses LLVM, which has a jit backend, I'd expect to see something get into and D 'eval' library into etc eventually. (phobos uses the BOOST license, which isn't compatible with LLVM).

Compile-time reflection and generation of code, on the other hand, is something D does in spades. It fulfills your dream list, although I think module level reflection might only be available in the github version. The API design is still in flux and we are actively iterating / improving it as find new uses cases and bugs. The current plan is to migrate all the traits functions over to a special 'meta' namespace (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid libraries for each of the concepts I listed, (prototype objects, duck-typing/casting or serialization), have been written using the compile-time meta-programming features in D. So that's the good.

On the other hand, D's runtime capabilities are limited to object.factory, the under implemented RTTI and library solutions which manually expose information gathered by D's compile-time mechanisms. And so far, these tools have been more than enough, from a functionality point of view. Most of our desire for better runtime reflection stems from a desire for efficiency, composition, cleanliness of user syntax and simplification of library code. These are all important issues for the widespread use of reflection based libraries, but they're not 'I can't implement X' issues.

As for the future, I believe that the division in D between compile-time and run-time reflection warrants a serious look at the design of the run-time half of the system. To that end, I believe that implementing reflection in a library would be the best way to experiment and iterate an API. To that end, I have a proposal in the review queue to improve std.variant which contains dynamic dispatch (i.e. the ability to get/set public fields and call methods), duck-typeing/casting and prototype-style objects. Notably, it's missing introspection capabilities as thats what I'm most unsure about API wise, and simplest to add. Designing reflection inside a library keeps Walter & Co's bandwidth free for other bugs/features and provides a very good stress test of D's meta-programming capabilities. (Both of which I think are good things)
(I am soliciting feedback, if you care to take a look: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

And there are many reasons we might want to experiment with D's runtime reflection API instead of just copying someone. For example, take Dependency Injection. Using Java-style reflection, DI suffers from the Reflection injection and Unsafe Reflection security vulnerabilities. To do DI safely, you have to validate your inputs and D's compile-time reflection provides a perfect way to implement validated DI. Every time I hear about some form of injection, be it SQL or JSON or Reflection, hit the news, makes me think that 5-10 years from now well look back on the use of unvalidated dynamic code constructs the same way we do about null terminated arrays today.

On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob@me.com> wrote:
> On 2011-10-25 16:21, Robert Jacques wrote:
[snip]
>> Both are protection attributes. The problem of taking the delegate of a
>> mutable member on a const object, is the exact same problem as taking
>> the delegate of a private member.
>
> I don't agree.

[snip]

>> Bypassing the protection type system and delegate escape are very
>> different things. The escape of a delegate is fully controlled by the
>> coder of the class/module; with bypass mechanisms, the library writer
>> has no control.
>
> So how should this be fixed? Make it a compile error to create a
> delegate from a private method?

I think we are taking about different things. Today in D, due to a bug, for example

foo.bar()

won't compile because it's private/not-const/etc. But

(&foo.bar)()

will compile. That 'feature' is what I thought you were referring to. If the writer of a class wants to let a delegate of a private member function escape, I don't have a fundamental problem with it. But I do have a problem with someone who isn't the writer of the class doing so.

[snip]

>> Still, __traits is still the full fledged compile-time reflection
>> feature and doesn't bypass protection attributes.
>
> Yeah, but for a serialization library I want to be able to serialize
> private fields. This is known issue that this breaks encapsulation.

I understand that at times when serialization has to break encapsulation. My opinion though, is that classes should have to opt in, before the serializer goes ahead and breaks their carefully designed encapsulation. i.e. someone should have to code review the class to make sure everything is kosher. For code you are developing, this is as simple and straightforward as adding a single mixin statement to the class.

I really like Gor's idea of @noreflect for this. You have a network class you don't want as easily reversible, or a security class, or private fields such as password. You just mark the module, type, methods, or fields, as @noreflect. It could even disable compile-time reflection if desired.

It would be important, however, to be able to pass a flag to the compiler that marks everything as @noreflect. The idea of opt-in reflection is just silly. Standard example is a serialization library. Oh hey, you wanna serialize a field of type Vector3? Sorry, the person forgot to mark it as @reflected. Oh, you wanna serialize these properties of some class? Sorry, not reflected. Opt-out is definitely the way to go, as it gives the best of all worlds (and removes all overhead if you just pass in something like --version=noreflect to dmd).

On 24/10/2011 10:11 PM, Robert Jacques wrote:
> On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson
> <metalcaedes@gmail.com> wrote:
>> Am 24.10.2011 17:37, schrieb Robert Jacques:
>>> That's a very heavy price to pay, just
>>> from a program maintenance perspective. And if you consider someone
>>> writing medical or financial software, the privacy concerns of exposing
>>> private variable to all become very real.
>>>
>>
>> I don't think marking a method as private is a good way to hide critical
>> information. If you can actually call methods (like in real calls, not
>> some kind of webservice), you can as well use pointers to read the
>> memory directly.
>> Or he could run it in a debugger. Or (with root privileges) just look at
>> the whole memory of the process. Or ...
>>
>> Cheers,
>> - Daniel
>
> By your argument, because can ever be truly secure there's no point to
> trying to secure it. I don't buy that. There is a world of difference
> between obj.reflect("SSN"), and trying to figure out 1) what objects
> hold SSN numbers 2) what's its field offset is and 3) what its memory
> representations is. (because remember, you don't have the source code)
> And that's assuming you can compile D code against the library and the
> library writer has directly exposed the critical object. More likely,
> you'll have to repeat steps 1-3 multiple times to find the data you're
> after. And runtime reflection means access will get exposed across
> language barriers, e.g. in your application's macro language.
>
> Furthermore, security focused languages seem to be centering around the
> concept of capabilities, of which private, package, protected, public
> and extern are the most rudimentary form of.
>
> Getting back to your original point, I wasn't suggesting that private is
> a panacea for security concerns, but it is one of the more powerful
> tools in our tool box.

The point of runtime reflection is not your own code, that's the thing.
By having opt-in reflection, you are at the mercy that every single component, every piece of code, every library, that you did not write, the author decided to enable reflection. It is an unreasonable assumption.


On 25/10/2011 11:44 PM, Robert Jacques wrote:
> I understand that at times when serialization has to break
> encapsulation. My opinion though, is that classes should have to opt in,
> before the serializer goes ahead and breaks their carefully designed
> encapsulation. i.e. someone should have to code review the class to make
> sure everything is kosher. For code you are developing, this is as
> simple and straightforward as adding a single mixin statement to the class.

It is the responsibility of the one using your code to determine whether they want take the risk and access private members.
Sometimes, it's needed. I have used private methods in production code when programming in .NET, just because the developers of it did not realize that something that's private would be useful to some people. It saved many, many, hours of pain and effort of rolling your own solution. If I upgrade the app to a newer version of .NET and the code breaks, that's the risk that was taken. Presumably the person accessing these things the author did not intend has taken steps to ensure it doesn't break if things change (or doesn't intend to link to a newer version of the library).

On 2011-10-26 07:44, Robert Jacques wrote:
> On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob@me.com> wrote:
>> On 2011-10-25 16:21, Robert Jacques wrote:
> [snip]
>>> Both are protection attributes. The problem of taking the delegate of a
>>> mutable member on a const object, is the exact same problem as taking
>>> the delegate of a private member.
>>
>> I don't agree.
>
> [snip]
>
>>> Bypassing the protection type system and delegate escape are very
>>> different things. The escape of a delegate is fully controlled by the
>>> coder of the class/module; with bypass mechanisms, the library writer
>>> has no control.
>>
>> So how should this be fixed? Make it a compile error to create a
>> delegate from a private method?
>
> I think we are taking about different things. Today in D, due to a bug,
> for example
>
> foo.bar()
>
> won't compile because it's private/not-const/etc. But
>
> (&foo.bar)()
>
> will compile. That 'feature' is what I thought you were referring to. If
> the writer of a class wants to let a delegate of a private member
> function escape, I don't have a fundamental problem with it. But I do
> have a problem with someone who isn't the writer of the class doing so.

I'm referring to the above but, as I've already written and you haven't replied to is, I don't understand how it can be prevented.

> [snip]
>
>>> Still, __traits is still the full fledged compile-time reflection
>>> feature and doesn't bypass protection attributes.
>>
>> Yeah, but for a serialization library I want to be able to serialize
>> private fields. This is known issue that this breaks encapsulation.
>
> I understand that at times when serialization has to break
> encapsulation. My opinion though, is that classes should have to opt in,
> before the serializer goes ahead and breaks their carefully designed
> encapsulation. i.e. someone should have to code review the class to make
> sure everything is kosher. For code you are developing, this is as
> simple and straightforward as adding a single mixin statement to the class.

It's not straightforward when serializing third party types. One of my goals when I started writing my serialization library was to, somewhere down the road, create a window/interface builder for DWT. The builder would (de)serialize the widgets available in DWT and save it as an XML file. Since I'm the current maintainer of DWT I could go and modify DWT to fit my needs but I don't think that's the right approach and will make merging future versions of SWT more difficult.

-- 
/Jacob Carlborg

On 2011-10-26 07:59, Kapps wrote:
> I really like Gor's idea of @noreflect for this. You have a network
> class you don't want as easily reversible, or a security class, or
> private fields such as password. You just mark the module, type,
> methods, or fields, as @noreflect. It could even disable compile-time
> reflection if desired.
>
> It would be important, however, to be able to pass a flag to the
> compiler that marks everything as @noreflect. The idea of opt-in
> reflection is just silly. Standard example is a serialization library.
> Oh hey, you wanna serialize a field of type Vector3? Sorry, the person
> forgot to mark it as @reflected. Oh, you wanna serialize these
> properties of some class? Sorry, not reflected. Opt-out is definitely
> the way to go, as it gives the best of all worlds (and removes all
> overhead if you just pass in something like --version=noreflect to dmd).

I agree, than can be SO annoying just because someone forgot to think about reflection/serialization.

-- 
/Jacob Carlborg

Hello Robert,

Am 26.10.11 07:16, schrieb Robert Jacques:
> On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>> Am 25.10.11 16:41, schrieb Robert Jacques:
>>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>>>> [...]
>> Hi Robert,
>>
>> Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
>> "
>> [...]
>> - Discover and modify source code constructions (such as code blocks,
>> classes, methods, protocols, etc.) as a first-class object at runtime.
>> - Convert a string matching the symbolic name of a class or function
>> into a reference to or invocation of that class or function.
>> [...]
>> "
>>
>> Here is what I would dream of for arbitrary objects/classes (not
>> necessarily known at compile-time):
>> - Query an object for its list of methods together with their
>> signatures. Select a method, bind some values to its arguments, call it,
>> and retrieve the return type (if any).
>> - Query an object for its public fields (at least), and provide a way to
>> get/set their values.
>> - Query an object's class for all implemented interfaces and its base
>> class.
>> - Query a module for all type definitions and provide a way to
>> introspect these types in more detail. For instance, it would be really
>> cool if I could find a class with name "Car" in module "cars", get a
>> list of all defined constructors, select one, bind values to the
>> constructor's parameters, and create a corresponding object.
>>
>> [...]
>>
>> Implementing such a DI container heavily depends on reflection, because
>> the DI container component doesn't know anything about the objects to be
>> created during runtime.
>>
>> Qt also extends C++ with a reflection mechanism through the help of its
>> meta object compiler (moc). It analyses the C++ source code, generates
>> meta class definitions [6,7] and weaves them into your Qt class. Hence,
>> in Qt, you can query an object for fields, methods, interfaces, etc. and
>> you can call methods with arbitrary parameters, or you can instantiate a
>> class using an arbitrary constructor. Consequently, somone implemented a
>> DI container for C++ which is based on Qt and works more or less the
>> same way the Spring DI container does. You can build up object trees
>> simply by specifying such trees in an XML file.
>>
>> I don't go into why dependency injection is a very powerful feature.
>> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
>> such a flexible dependency injection mechanism a lot. And I hope this
>> will eventually be available for D.
>>
>> Cheers,
>> Jonny
>>
>> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
>> [2] http://en.wikipedia.org/wiki/Dependency_injection
>> [3] http://martinfowler.com/articles/injection.html
>> [4]
>> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
>>
>> [5] http://qtioccontainer.sourceforge.net/
>> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
>> [7]
>> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
>>
>
> Hi Jonny,
> Thank you for your informative (and well cited) post. It has provided me
> with a new take on an old design pattern and some enjoyable reading. In
> return, let me outline my opinion of reflection in D today, and
> tomorrow, as it pertains to your wish list.

Many thanks to you, too, for your very elaborate answer :)

> Reflection in D today is very different from the host of VM languages
> that have popularized the concept. Being a compiled systems language,
> actual runtime self-modification is too virus like to become at a
> language level feature. However, given the compilation speed of D,
> people have made proof of concept libraries that essentially wrapped the
> compiler and dynamically loaded the result. As LDC uses LLVM, which has
> a jit backend, I'd expect to see something get into and D 'eval' library
> into etc eventually. (phobos uses the BOOST license, which isn't
> compatible with LLVM).

I know, that "runtime self-modification" and runtime code generation is a "dangerous" feature. And there really are rare cases where using such an approach might justify the risc in using it. Although this feature is not on my wish list, it might be good for generating dynamic proxies to arbitrary object instances like they are used by some ORMs. See Hibernate/NHibernate, for example [1,2]. Another example is aspect-oriented programming. But while I can't see the exacty reason for it, such a feature might indeed be a feature which is more appropriate for VM languages.

> Compile-time reflection and generation of code, on the other hand, is
> something D does in spades. It fulfills your dream list, although I
> think module level reflection might only be available in the github
> version. The API design is still in flux and we are actively iterating /
> improving it as find new uses cases and bugs. The current plan is to
> migrate all the traits functions over to a special 'meta' namespace
> (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
> libraries for each of the concepts I listed, (prototype objects,
> duck-typing/casting or serialization), have been written using the
> compile-time meta-programming features in D. So that's the good.

A absolutely agree! D's compile-time reflection is very good. And it's hard to top this.

> On the other hand, D's runtime capabilities are limited to
> object.factory, the under implemented RTTI and library solutions which
> manually expose information gathered by D's compile-time mechanisms. And
> so far, these tools have been more than enough, from a functionality
> point of view. Most of our desire for better runtime reflection stems
> from a desire for efficiency, composition, cleanliness of user syntax
> and simplification of library code. These are all important issues for
> the widespread use of reflection based libraries, but they're not 'I
> can't implement X' issues.

You are right, there IS certainly always some way to 'implement X'. But, as you know, there is always a consideration of the effort you need to implement X.

> As for the future, I believe that the division in D between compile-time
> and run-time reflection warrants a serious look at the design of the
> run-time half of the system. To that end, I believe that implementing
> reflection in a library would be the best way to experiment and iterate
> an API. To that end, I have a proposal in the review queue to improve
> std.variant which contains dynamic dispatch (i.e. the ability to get/set
> public fields and call methods), duck-typeing/casting and
> prototype-style objects. Notably, it's missing introspection
> capabilities as thats what I'm most unsure about API wise, and simplest
> to add. Designing reflection inside a library keeps Walter & Co's
> bandwidth free for other bugs/features and provides a very good stress
> test of D's meta-programming capabilities. (Both of which I think are
> good things)

I've got no problem with an approach which puts runtime reflection capabiities into a separate library. No matter were you look at, Java, C#, or Qt, all have a library for reflection purpose. I do not see, however, how this might be done without compiler support. As already mentioned, Qt has its own moc compiler, which parses your C++ source code and generates the necessary infra structure. I'm still a beginner with respect to D, so I don't know what is really already possible, and what not. But my current feeling is that a similar approach would also be needed for D. If the D compiler itself, or another post-compiler component should generate the meta information for runtime reflection is another question. I could live with both, although I'd prefer the former built-in one.

> (I am soliciting feedback, if you care to take a look:
> https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

Thanks for this link, I'll certainly have a look at it.

> And there are many reasons we might want to experiment with D's runtime
> reflection API instead of just copying someone. For example, take
> Dependency Injection. Using Java-style reflection, DI suffers from the
> Reflection injection and Unsafe Reflection security vulnerabilities. To
> do DI safely, you have to validate your inputs and D's compile-time
> reflection provides a perfect way to implement validated DI. Every time
> I hear about some form of injection, be it SQL or JSON or Reflection,
> hit the news, makes me think that 5-10 years from now well look back on
> the use of unvalidated dynamic code constructs the same way we do about
> null terminated arrays today.

Considering security vulnerabilities is of course a very important matter. However, I think security must be assured a by software's design. Enforcing it solely by programming language constructs will not work for all cases. Particularly not, if this programming language allows direct access to a computer's memory, like C, C++, and D does. There is no sandbox out-of-the-box, where the compiled program runs in. So if you have a private field in a class that carries a password, one has to make sure it's only in memory as long as it is required. And if it is not required anymore one should clear it out with zeros, for example. So making a field private is by no means a secure solution. A hacker will not give up just because the official API declares a field to be private.

I consider 'unsafe reflection', as you call it, as a tool. Let's compare it to a knife, for instance. It is really a useful tool, I think nobody doubts, but at the same time it can be very dangerous. But inspite of this fact, everybody has not only one at home, I guess. Pointers are a useful tool, too. But they are also dangerous and still available in D, which is a good thing, because you can use this tool where needed. And actually, I don't think runtime reflection must be unsafe. The reflection mechanism provides type information for an object's properties, functions, arguments, etc. So validated DI is even possible with runtime reflection. Let's consider the the XML configuration of object trees once again. If you want to store a string to an int-field, for instance, then the DI container can refuse to do this, because it has access to all information required to enforce correct value types.

One more use case for reflection is data binding with GUI components. This approach is heavily used in Windows Presentation Foundation library [3,4]. GUI components can update an object's properties by using reflection. You don't need to register listeners for this purpose anymore.

That said, I don't think dynamic code constructs will be old-fashioned in 5-10 years, because you don't have to go an "unvalidated dynamic" way with runtime reflection.

BTW, as this thread also discusses an opt-in or opt-out implementation for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I guess the binaries are meant) is not as bad as it might sound with today's memory sizes. And if one wants to avoid it for optimization purpose, one can do it. The use of D as a systems programming language for embedded systems is, as I've read somewhere, not a first citizen anyway, because you'll get a lot of code for the garbage collection mechanism.

Cheers,
Jonny

[1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
[2] http://en.wikipedia.org/wiki/Nhibernate
[3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
[4] http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx