October 25, 2011
On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob@me.com> wrote:
> On 2011-10-25 07:24, Robert Jacques wrote:
[snip]

>>> On
>>> the other hand it feels like I'm stretching D's metaprogramming
>>> capabilities in my serialization library. That's just because D doesn't
>>> have proper reflection.
>>
>> I've written three serialization libraries and two versions of a runtime
>> reflection library; the first one was prior to the string bugs in CTFE
>> being fixed and ran DMD out of memory. I found serialization to be
>> simple and straight forward. And I found compiler bugs to be a greater
>> issue with reflection than any limits in D's capabilities. Indeed, given
>> I wrote the same library using two completely different mechanisms,
>> gives me great hope for D. So, I don't agree with your assessment, but I
>> do understand where you're coming from. A lot of D's metaprogramming
>> capabilities are undocumented, and knowing the right tool for the job is
>> essential to getting nice, clean code.
>
> For example, getting the name of a instance variable:
>
> T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];
>
> I wouldn't consider that simple and straight forward. This works for
> both D1 and D2, don't know if D2 has a better way.

Not really, for the advanced user. I do find using "some string".length to be cleaner in these cases where you have to slice .stringof. i.e.

    static Value from(T)(T value) {
        //...
        static if(is(T == struct) || is(T == class)) {
            Value[string] result;
            foreach(i,v;value.tupleof) {
                result[value.tupleof[i].stringof["value.".length..$]] = from(v);
            }
            return Value( result );
        }
        //..
    }

That said, I think anyone can read the above and understand what's happening. And even writing that code would be straightforward is .stringof and .tupleof were better documented.

Also, the first serialization library I wrote used __traits instead. It was more awkward, mainly because __traits doesn't an 'allFields' member, just an 'allMembers'. If it did then the above becomes:

foreach(name;__traits(allFields,value) {
    result[name] = from(__traits(getMember, value, name));
}

which really is straightforward. Now I originally used 'allMembers' and then filtered out everything I didn't want, which was fairly straightforward but verbose.

> Your serialization library only works on public fields? Doesn't sound
> very useful.

In the interest of full disclosure, the library I wrote was quick and dirty and does not handle ploy-morphism, cycle detection, aliasing, etc (mainly because JSON doesn't); I used traits for my first one (so protection was observed) but .tupleof for my second and third and was lazy about it. It's one line of code to add protection to tuple of, but I've never gotten around to it. I figured my code would be replace by orange or std.serialization or something.

That said, I have never serialized a private variable and have never wanted too. The serialized objects are, for the most part, part of the external API to my program. I have to do validation checks on the inputs and changing anything takes a week to get the other teams to propagate the changes on their side. There are also a couple of classes which I serialize in a binary format as a poor mans database, but again, there's no need for anything in it to be private.

[snip]

>> Actually, you're not supposed to be able to; this is bug 1983.
>> http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
>> https://github.com/D-Programming-Language/dmd/pull/71
>
> I don't see what that has to do with private, it's about const, as far
> as I can see.

Both are protection attributes. The problem of taking the delegate of a mutable member on a const object, is the exact same problem as taking the delegate of a private member.

In short, you can't 'bypass private using a delegate.' That you can today, is simple a bug in the compiler.

> I don't see how calling a private method through a
> delegate can be easily solved. I see three solutions:
>
> 1. Make it a compile time error to create a delegate from a private
> method. This would not allow to call that method via a delegate in the
> same module, which is possible if you call the method directly.
>
> 2. Implement some, most likely, very advanced data flow analysis in the
> compiler to detect when a delegate pointing to a private method escapes
> the module. Results in a compile time error. Will most likely not happen.
>
> 3. Make it a runtime error when the delegate is actually called. This
> would probably need some pieces of 2 as well. I don't think the
> community will be happy about adding runtime checks like these.
>
> Oh, BTW the same needs to be done for function pointers as well since
> you can build a delegate out of a function pointer and an object.

Bypassing the protection type system and delegate escape are very different things. The escape of a delegate is fully controlled by the coder of the class/module; with bypass mechanisms, the library writer has no control.

>>> In Ruby an instance variable is always private. I don't think you can
>>> declare a public instance variable, one uses setters/getters for that.
>>> In D you can declare both private and public instance variables.
>>>
>>> In Ruby you can get/set a value of an instance variable using
>>> reflection, instance_variable_set/get.
>>> In D you can get/set a value of a private instance variable using
>>> compile time reflection, i.e. .tupleof.
>>
>> Yes, but only for source code you control.
>
> You don't need control of the source, you only need the static type. Of
> course an opaque struct won't work.

Sorry, that's what I meant by source you control. i.e. it is available to you to read in some form. Runtime reflection would allow you access to knowledge of opaque types.

>> Actually, baring bugs in DMD, .tupleof is the only way to access private
>> fields and there is _no way_ to access private members; And .tupleof
>> only works if you have access to the source code. Furthermore, .tupleof,
>> god-send that it is, isn't part of the D spec; unlike .stringof, it
>> didn't make it into TDPL nor into the D documentation. D's official
>> method of compile-time reflection is __traits. And __traits doesn't
>> bypass protection attributes. So precedence would be against
>> runtime-reflection having access to private members.
>
> The compiler has .tupleof and it's mentioned in the spec at
> d-programming-language.org: http://d-programming-language.org/struct.html

I stand corrected. Sorry, I just looked at TDPL's index, the page on Properties and googled it. Now that I look for it, it's also on the Class page.

Still, __traits is still the full fledged compile-time reflection feature and doesn't bypass protection attributes.
October 25, 2011
On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
> Hi,
>
> I would love to see runtime reflection be available in D. I am currently reading Walter Bright's "The D
> Programming Language" and with every page I read I became more and more fascinated and enthusiastic
> about D. But then I started to wonder why I haven't read about runtime reflection yet, so I did some Internet
> research and was a bit disappointed this feature isn't available. D is really cool, it's not only C done right,
> but also Java and C#. Actually, to me, it's one of the best programming language available yet. It's really a
> pitty RTTI is not available. So I hope your approach will somehow influence D's future :)
>
> Keep going,
> Jonny

Hi Jonny,
Well, some RTTI is available. How much RTTI is needed, really depends on what you're trying to do. Therefore whether D has or hasn't got RTTI depends heavily on what any particular person means by RTTI. Do you have some use cases in mind? Maybe prototype objects, duck-typing/casting or serialization? Or perhaps something you wrote?

Sorry for all the questions, but I feel as if we don't have a resident reflection expert in the dialog, or even the opinions of basic/experienced users. We all know that 'RTTI' is good, but we're a little vague on the how and why. (Well, beyond the use cases above)
October 25, 2011
On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> You can always write @noreflect: at the start of your module and it
> will work as expected. what's the problem with that?
> Also, compiler would be able to automatically add that attribute to
> any symbol, which is private and does not escape the module.
> Again, not including reflection is purely _optimization_ issue and
> should not be interfered with it's functionality.
> Reflection is something you rarely have in mind when designing, for
> example, classes.
> Requiring explicit request for reflection defeats the purpose.
>
> You could just as well say "writing public on every member is a
> clutter". It is, but you don't have to do it to achieve the same
> effect.

You can always write @reflect: at the start of your module and will work as expected. what's the problem with that?

Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things.

Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.
October 25, 2011
As i said before, enabling reflection manually is an act of enabling a
certain range of stuff you can do with the code. Forgetting to enable
reflection can lead to serious problems, when the reflection is
required.
disabling reflection manually is an act of optimization purely.
Forgetting to disable the reflection can only lead to unnecessary data
being stored.

The second scenario is much less error-prone.
yes, it _may_ lead to code bloat if you forget to disable it where it
is _absolutely positively unneeded_, but forgetting to enable it will
just make your code useless in many situations.

On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford@jhu.edu> wrote:
> On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
>>
>> You can always write @noreflect: at the start of your module and it
>> will work as expected. what's the problem with that?
>> Also, compiler would be able to automatically add that attribute to
>> any symbol, which is private and does not escape the module.
>> Again, not including reflection is purely _optimization_ issue and
>> should not be interfered with it's functionality.
>> Reflection is something you rarely have in mind when designing, for
>> example, classes.
>> Requiring explicit request for reflection defeats the purpose.
>>
>> You could just as well say "writing public on every member is a clutter". It is, but you don't have to do it to achieve the same effect.
>
> You can always write @reflect: at the start of your module and will work as expected. what's the problem with that?
>
> Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things.
>
> Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.
>
October 25, 2011
Requiring explicit request for reflection contradicts D's "safety
first, performance second" policy.
lack of reflection is a safety issue too, because you won't be able to
do some checks.

On Tue, Oct 25, 2011 at 7:31 PM, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> As i said before, enabling reflection manually is an act of enabling a
> certain range of stuff you can do with the code. Forgetting to enable
> reflection can lead to serious problems, when the reflection is
> required.
> disabling reflection manually is an act of optimization purely.
> Forgetting to disable the reflection can only lead to unnecessary data
> being stored.
>
> The second scenario is much less error-prone.
> yes, it _may_ lead to code bloat if you forget to disable it where it
> is _absolutely positively unneeded_, but forgetting to enable it will
> just make your code useless in many situations.
>
> On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford@jhu.edu> wrote:
>> On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
>>>
>>> You can always write @noreflect: at the start of your module and it
>>> will work as expected. what's the problem with that?
>>> Also, compiler would be able to automatically add that attribute to
>>> any symbol, which is private and does not escape the module.
>>> Again, not including reflection is purely _optimization_ issue and
>>> should not be interfered with it's functionality.
>>> Reflection is something you rarely have in mind when designing, for
>>> example, classes.
>>> Requiring explicit request for reflection defeats the purpose.
>>>
>>> You could just as well say "writing public on every member is a clutter". It is, but you don't have to do it to achieve the same effect.
>>
>> You can always write @reflect: at the start of your module and will work as expected. what's the problem with that?
>>
>> Rhetoric aside, Timon wasn't referring to code clutter on behalf of the programmer. He was referring to code bloat in the compiled program. The two are very different things.
>>
>> Also, just because reflection is something you rarely have in mind when designing, doesn't mean it shouldn't be part of the design API. Reflection is somewhat equivalent to the extern protection modifiers and so should warren careful consideration. Furthermore, bugs with reflection, say due to a change in the API, don't break during compilation; they break at runtime. And practically, there's a decent change your customer, not your test suite, is going to find it.
>>
>
October 25, 2011
On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> As i said before, enabling reflection manually is an act of enabling a
> certain range of stuff you can do with the code. Forgetting to enable
> reflection can lead to serious problems, when the reflection is
> required.

And when, precisely, is reflection _required_? Granted, in other languages it is the only tool for the job. But we have a very big toolkit in D, and so far, I have yet to see any example (including Rails) which _requires_ the holistic runtime reflection proposed in these threads.

> disabling reflection manually is an act of optimization purely.

No, it isn't. Go read the rest of these threads for counter-arguments. Or the Java Best Practices guide. Or search the Open Web Application Security Project for reflection vulnerabilities.

> Forgetting to disable the reflection can only lead to unnecessary data
> being stored.

Alternatively, sensitive information being exposed.

> The second scenario is much less error-prone.

Incorrect, the second scenario is a security hole waiting to be exploited.

> yes, it _may_ lead to code bloat if you forget to disable it where it
> is _absolutely positively unneeded_,

Having written a (limited) runtime reflection library for D, yes it causes bloat even if you only enable it where _absolutely positively needed_.

> but forgetting to enable it will
> just make your code useless in many situations.

Correction: it will make code designed to _require_ runtime-reflection incompatible with code not designed to support reflection. Given the percentage of code that requires reflection (very small) and the percentage of code which wouldn't be compatible or trivially made to be compatible, (very small) I'm fine with that.
October 25, 2011
On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> Requiring explicit request for reflection contradicts D's "safety
> first, performance second" policy.

Umm, it's "safe defaults, performance possible".

> lack of reflection is a safety issue too, because you won't be able to
> do some checks.

What would you use reflection to check?
October 25, 2011
I didn't mean reflection for violating access specifications. In that
case, of course, it's error-prone.
What i meant was a set of dynamic counterparts of static features of D:
* Dynamic interfaces
* Dynamic overloading
* Dynamic modules
* ...
That's what i had in mind.

On Tue, Oct 25, 2011 at 8:08 PM, Robert Jacques <sandford@jhu.edu> wrote:
> On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
>>
>> Requiring explicit request for reflection contradicts D's "safety first, performance second" policy.
>
> Umm, it's "safe defaults, performance possible".
>
>> lack of reflection is a safety issue too, because you won't be able to do some checks.
>
> What would you use reflection to check?
>
October 25, 2011
On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> I didn't mean reflection for violating access specifications. In that
> case, of course, it's error-prone.

Thank you, I'm glad you agree.

> What i meant was a set of dynamic counterparts of static features of D:
> * Dynamic interfaces
> * Dynamic overloading
> * Dynamic modules
> * ...
> That's what i had in mind.

Care to explain what would each of those should look like?

Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a dynamic-object for certain fields/functions.

Would dynamic overload be a subset of prototype-style objects?

Would dynamic modules simply container for dynamic functions/objects? Something more?
October 25, 2011
On 2011-10-25 16:21, Robert Jacques wrote:
> On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob@me.com> wrote:
>> For example, getting the name of a instance variable:
>>
>> T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];
>>
>> I wouldn't consider that simple and straight forward. This works for
>> both D1 and D2, don't know if D2 has a better way.
>
> Not really, for the advanced user. I do find using "some string".length
> to be cleaner in these cases where you have to slice .stringof. i.e.
>
> static Value from(T)(T value) {
> //...
> static if(is(T == struct) || is(T == class)) {
> Value[string] result;
> foreach(i,v;value.tupleof) {
> result[value.tupleof[i].stringof["value.".length..$]] = from(v);
> }
> return Value( result );
> }
> //..
> }
>
> That said, I think anyone can read the above and understand what's
> happening. And even writing that code would be straightforward is
> .stringof and .tupleof were better documented.

Yes, I found the snippet I use to get the name of a field by trail and error.

> Also, the first serialization library I wrote used __traits instead. It
> was more awkward, mainly because __traits doesn't an 'allFields' member,
> just an 'allMembers'. If it did then the above becomes:
>
> foreach(name;__traits(allFields,value) {
> result[name] = from(__traits(getMember, value, name));
> }
>
> which really is straightforward. Now I originally used 'allMembers' and
> then filtered out everything I didn't want, which was fairly
> straightforward but verbose.

__traits(getMember) is more clean when you the name of all the fields you want.

>> Your serialization library only works on public fields? Doesn't sound
>> very useful.
>
> In the interest of full disclosure, the library I wrote was quick and
> dirty and does not handle ploy-morphism, cycle detection, aliasing, etc
> (mainly because JSON doesn't); I used traits for my first one (so
> protection was observed) but .tupleof for my second and third and was
> lazy about it. It's one line of code to add protection to tuple of, but
> I've never gotten around to it. I figured my code would be replace by
> orange or std.serialization or something.

Well, things get a bit more complicated when the library needs to support ploy-morphism, cycle detection, aliasing and so on.

> That said, I have never serialized a private variable and have never
> wanted too. The serialized objects are, for the most part, part of the
> external API to my program. I have to do validation checks on the inputs
> and changing anything takes a week to get the other teams to propagate
> the changes on their side. There are also a couple of classes which I
> serialize in a binary format as a poor mans database, but again, there's
> no need for anything in it to be private.

Ok, I see.

>
> [snip]
>
>>> Actually, you're not supposed to be able to; this is bug 1983.
>>> http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
>>> https://github.com/D-Programming-Language/dmd/pull/71
>>
>> I don't see what that has to do with private, it's about const, as far
>> as I can see.
>
> Both are protection attributes. The problem of taking the delegate of a
> mutable member on a const object, is the exact same problem as taking
> the delegate of a private member.

I don't agree.

> In short, you can't 'bypass private using a delegate.' That you can
> today, is simple a bug in the compiler.
>
>> I don't see how calling a private method through a
>> delegate can be easily solved. I see three solutions:
>>
>> 1. Make it a compile time error to create a delegate from a private
>> method. This would not allow to call that method via a delegate in the
>> same module, which is possible if you call the method directly.
>>
>> 2. Implement some, most likely, very advanced data flow analysis in the
>> compiler to detect when a delegate pointing to a private method escapes
>> the module. Results in a compile time error. Will most likely not happen.
>>
>> 3. Make it a runtime error when the delegate is actually called. This
>> would probably need some pieces of 2 as well. I don't think the
>> community will be happy about adding runtime checks like these.
>>
>> Oh, BTW the same needs to be done for function pointers as well since
>> you can build a delegate out of a function pointer and an object.
>
> Bypassing the protection type system and delegate escape are very
> different things. The escape of a delegate is fully controlled by the
> coder of the class/module; with bypass mechanisms, the library writer
> has no control.

So how should this be fixed? Make it a compile error to create a delegate from a private method?

>>>> In Ruby an instance variable is always private. I don't think you can
>>>> declare a public instance variable, one uses setters/getters for that.
>>>> In D you can declare both private and public instance variables.
>>>>
>>>> In Ruby you can get/set a value of an instance variable using
>>>> reflection, instance_variable_set/get.
>>>> In D you can get/set a value of a private instance variable using
>>>> compile time reflection, i.e. .tupleof.
>>>
>>> Yes, but only for source code you control.
>>
>> You don't need control of the source, you only need the static type. Of
>> course an opaque struct won't work.
>
> Sorry, that's what I meant by source you control. i.e. it is available
> to you to read in some form. Runtime reflection would allow you access
> to knowledge of opaque types.

Ok, actually I haven't thought of how runtime reflection would be have with an opaque type.

>>> Actually, baring bugs in DMD, .tupleof is the only way to access private
>>> fields and there is _no way_ to access private members; And .tupleof
>>> only works if you have access to the source code. Furthermore, .tupleof,
>>> god-send that it is, isn't part of the D spec; unlike .stringof, it
>>> didn't make it into TDPL nor into the D documentation. D's official
>>> method of compile-time reflection is __traits. And __traits doesn't
>>> bypass protection attributes. So precedence would be against
>>> runtime-reflection having access to private members.
>>
>> The compiler has .tupleof and it's mentioned in the spec at
>> d-programming-language.org: http://d-programming-language.org/struct.html
>
> I stand corrected. Sorry, I just looked at TDPL's index, the page on
> Properties and googled it. Now that I look for it, it's also on the
> Class page.

No problem.

> Still, __traits is still the full fledged compile-time reflection
> feature and doesn't bypass protection attributes.

Yeah, but for a serialization library I want to be able to serialize private fields. This is known issue that this breaks encapsulation.

-- 
/Jacob Carlborg