On 2011-10-24 17:37, Robert Jacques wrote:
> I'm sorry if I've come across that way. I'm well aware with the reasons
> for wanting access to private methods/fields via reflection and have
> mentioned them in previous posts. What I've tried to point out is very
> eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
> to bypass protection guarantees essentially makes all declarations
> public in nature, if not extern. That's a very heavy price to pay, just
> from a program maintenance perspective. And if you consider someone
> writing medical or financial software, the privacy concerns of exposing
> private variable to all become very real.

If someone is using a library and starts to mess with calling private methods they shouldn't they'll most likely mess things up for them self.

> Back to my actual argument, what I was pointing out was that the
> toolboxes of various languages differ heavily. Just because Ruby has a
> very cool library which uses runtime reflection, one shouldn't simply
> state 'D must have runtime reflection.' One needs to first decompose
> what the library is trying to achieve from the language mechanisms used
> to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
> does D's solution compare to Ruby's?

Of course. I'm just showing that another language/library uses it and therefore there could be a chance that should be doable in D as well.

> My impression of what ActiveRecord, (which my be wrong), is that its
> using runtime-reflection to do various meta-programming activities and
> that under-the-hood, it stretching Ruby to the breaking point in order
> to do so. D, on the other hand, has a ton of meta-programming specific
> features, and doesn't need to rely on reflection to do so. This is
> partly to do with Ruby supporting 'eval' and thus making no distinction
> between compile-time and run-time behavior; that's correct for Ruby, but
> D is a different kettle of fish.

I doubt that ActiveRecord is stretching Ruby to the breaking point. On the other hand it feels like I'm stretching D's metaprogramming capabilities in my serialization library. That's just because D doesn't have proper reflection.

In Ruby there is very seldom reason to use "eval". That's because of the excellent support for runtime reflection. There's no clumsy syntax, no hacks, no need to instantiate reflectors and similar. You just call a method like you would call any other method:

class Foo
    def bar
    end
end

foo = Foo.new
foo.bar()
foo.send("bar")

In the above code, "bar" is called first using regular method call syntax and then using reflection.

>>> I mean, considering that Ruby is a dynamic language, would bar even be
>>> considered private from inside the super-class's ctor? Is Ruby's private
>>> even comparable to D's private? Given that Ruby, by design, requires
>>> fields to be private and for all external accesses to happen via
>>> methods, should we consider a way to violate that contract an example of
>>> mis-feature?
>>
>> There is no constructor involved here. "before_save" is class method
>> called when the class "Foo" is loaded.
>
> So it's the static ctor, then.

I guess you could call it that.

>> You can do basically whatever you want with the code in Ruby. You can
>> replace an existing method/class in the standard library with your own.
>> You can easily add methods to existing classes. It's the programmer that
>> is responsible to not mess up things, just as in D where you can cast
>> away immutable, shared and do unsafe things with pointers.
>
> Well one one hand, none of those things are allowed in SafeD (And you
> know, that's not a resounding argument for reflection of private
> members). On the other, you didn't answered my other two questions. In
> fact, the impression I'm getting is that Ruby's private isn't comparable
> to D's private.

If we talk about methods it's easiest. If you call a private method in Ruby using the regular method call syntax you get an error. If you do the same in D you get an error as well. The only difference is in Ruby it will be a runtime error and in D it will be a compile time error.

In Ruby you can bypass private using reflection with the "send" method. In D you can bypass private using a delegate.

In Ruby an instance variable is always private. I don't think you can declare a public instance variable, one uses setters/getters for that.
In D you can declare both private and public instance variables.

In Ruby you can get/set a value of an instance variable using reflection, instance_variable_set/get.
In D you can get/set a value of a private instance variable using compile time reflection, i.e. .tupleof.

Now, since we can call private methods and get/set private instance variables using the features we have now, why shouldn't we be able to do the same with runtime reflection?

BTW, I don't think it's a mis-feature.

-- 
/Jacob Carlborg

Am 24.10.2011 17:37, schrieb Robert Jacques:
> That's a very heavy price to pay, just
> from a program maintenance perspective. And if you consider someone
> writing medical or financial software, the privacy concerns of exposing
> private variable to all become very real.
> 

I don't think marking a method as private is a good way to hide critical
information. If you can actually call methods (like in real calls, not
some kind of webservice), you can as well use pointers to read the
memory directly.
Or he could run it in a debugger. Or (with root privileges) just look at
the whole memory of the process. Or ...

Cheers,
- Daniel

On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson <metalcaedes@gmail.com> wrote:
> Am 24.10.2011 17:37, schrieb Robert Jacques:
>> That's a very heavy price to pay, just
>> from a program maintenance perspective. And if you consider someone
>> writing medical or financial software, the privacy concerns of exposing
>> private variable to all become very real.
>>
>
> I don't think marking a method as private is a good way to hide critical
> information. If you can actually call methods (like in real calls, not
> some kind of webservice), you can as well use pointers to read the
> memory directly.
> Or he could run it in a debugger. Or (with root privileges) just look at
> the whole memory of the process. Or ...
>
> Cheers,
> - Daniel

By your argument, because can ever be truly secure there's no point to trying to secure it. I don't buy that. There is a world of difference between obj.reflect("SSN"), and trying to figure out 1) what objects hold SSN numbers 2) what's its field offset is and 3) what its memory representations is. (because remember, you don't have the source code) And that's assuming you can compile D code against the library and the library writer has directly exposed the critical object. More likely, you'll have to repeat steps 1-3 multiple times to find the data you're after. And runtime reflection means access will get exposed across language barriers, e.g. in your application's macro language.

Furthermore, security focused languages seem to be centering around the concept of capabilities, of which private, package, protected, public and extern are the most rudimentary form of.

Getting back to your original point, I wasn't suggesting that private is a panacea for security concerns, but it is one of the more powerful tools in our tool box.

On Mon, 24 Oct 2011 12:38:12 -0400, Jacob Carlborg <doob@me.com> wrote:

> On 2011-10-24 17:37, Robert Jacques wrote:
>> I'm sorry if I've come across that way. I'm well aware with the reasons
>> for wanting access to private methods/fields via reflection and have
>> mentioned them in previous posts. What I've tried to point out is very
>> eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
>> to bypass protection guarantees essentially makes all declarations
>> public in nature, if not extern. That's a very heavy price to pay, just
>> from a program maintenance perspective. And if you consider someone
>> writing medical or financial software, the privacy concerns of exposing
>> private variable to all become very real.
>
> If someone is using a library and starts to mess with calling private
> methods they shouldn't they'll most likely mess things up for them self.

Aren't private methods by definition something people shouldn't be using?

>> Back to my actual argument, what I was pointing out was that the
>> toolboxes of various languages differ heavily. Just because Ruby has a
>> very cool library which uses runtime reflection, one shouldn't simply
>> state 'D must have runtime reflection.' One needs to first decompose
>> what the library is trying to achieve from the language mechanisms used
>> to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
>> does D's solution compare to Ruby's?
>
> Of course. I'm just showing that another language/library uses it and
> therefore there could be a chance that should be doable in D as well.

I understand and appreciate your intent. However, do so should be accompanied by a critical dissection of the language/library, both to get those unfamiliar up to speed and to frame the conversation in terms of how best can we improve D.

>> My impression of what ActiveRecord, (which my be wrong), is that its
>> using runtime-reflection to do various meta-programming activities and
>> that under-the-hood, it stretching Ruby to the breaking point in order
>> to do so. D, on the other hand, has a ton of meta-programming specific
>> features, and doesn't need to rely on reflection to do so. This is
>> partly to do with Ruby supporting 'eval' and thus making no distinction
>> between compile-time and run-time behavior; that's correct for Ruby, but
>> D is a different kettle of fish.
>
> I doubt that ActiveRecord is stretching Ruby to the breaking point.

:) I don't know myself. I've just heard that the internals of Rails are deep, dark and full of non-idiomatic Rudy code.

> On
> the other hand it feels like I'm stretching D's metaprogramming
> capabilities in my serialization library. That's just because D doesn't
> have proper reflection.

I've written three serialization libraries and two versions of a runtime reflection library; the first one was prior to the string bugs in CTFE being fixed and ran DMD out of memory. I found serialization to be simple and straight forward. And I found compiler bugs to be a greater issue with reflection than any limits in D's capabilities. Indeed, given I wrote the same library using two completely different mechanisms, gives me great hope for D. So, I don't agree with your assessment, but I do understand where you're coming from. A lot of D's metaprogramming capabilities are undocumented, and knowing the right tool for the job is essential to getting nice, clean code.



> In Ruby there is very seldom reason to use "eval". That's because of the
> excellent support for runtime reflection. There's no clumsy syntax, no
> hacks, no need to instantiate reflectors and similar. You just call a
> method like you would call any other method:
>
> class Foo
>      def bar
>      end
> end
>
> foo = Foo.new
> foo.bar()
> foo.send("bar")
>
> In the above code, "bar" is called first using regular method call
> syntax and then using reflection.
>
>>>> I mean, considering that Ruby is a dynamic language, would bar even be
>>>> considered private from inside the super-class's ctor? Is Ruby's private
>>>> even comparable to D's private? Given that Ruby, by design, requires
>>>> fields to be private and for all external accesses to happen via
>>>> methods, should we consider a way to violate that contract an example of
>>>> mis-feature?
>>>
>>> There is no constructor involved here. "before_save" is class method
>>> called when the class "Foo" is loaded.
>>
>> So it's the static ctor, then.
>
> I guess you could call it that.

(Just checking, but you do know D has static ctors.)

>>> You can do basically whatever you want with the code in Ruby. You can
>>> replace an existing method/class in the standard library with your own.
>>> You can easily add methods to existing classes. It's the programmer that
>>> is responsible to not mess up things, just as in D where you can cast
>>> away immutable, shared and do unsafe things with pointers.
>>
>> Well one one hand, none of those things are allowed in SafeD (And you
>> know, that's not a resounding argument for reflection of private
>> members). On the other, you didn't answered my other two questions. In
>> fact, the impression I'm getting is that Ruby's private isn't comparable
>> to D's private.
>
> If we talk about methods it's easiest. If you call a private method in
> Ruby using the regular method call syntax you get an error. If you do
> the same in D you get an error as well. The only difference is in Ruby
> it will be a runtime error and in D it will be a compile time error.
>
> In Ruby you can bypass private using reflection with the "send" method.
> In D you can bypass private using a delegate.

Actually, you're not supposed to be able to; this is bug 1983. http://d.puremagic.com/issues/show_bug.cgi?id=1983  and  https://github.com/D-Programming-Language/dmd/pull/71

> In Ruby an instance variable is always private. I don't think you can
> declare a public instance variable, one uses setters/getters for that.
> In D you can declare both private and public instance variables.
>
> In Ruby you can get/set a value of an instance variable using
> reflection, instance_variable_set/get.
> In D you can get/set a value of a private instance variable using
> compile time reflection, i.e. .tupleof.

Yes, but only for source code you control.

> Now, since we can call private methods and get/set private instance
> variables using the features we have now, why shouldn't we be able to do
> the same with runtime reflection?
>
> BTW, I don't think it's a mis-feature.

Actually, baring bugs in DMD, .tupleof is the only way to access private fields and there is _no way_ to access private members; And .tupleof only works if you have access to the source code. Furthermore, .tupleof, god-send that it is, isn't part of the D spec; unlike .stringof, it didn't make it into TDPL nor into the D documentation. D's official method of compile-time reflection is __traits. And __traits doesn't bypass protection attributes. So precedence would be against runtime-reflection having access to private members.

On 2011-10-25 07:24, Robert Jacques wrote:
> Aren't private methods by definition something people shouldn't be using?

Yes, that's what I mean. But sometimes it CAN be useful.

> :) I don't know myself. I've just heard that the internals of Rails are
> deep, dark and full of non-idiomatic Rudy code.

The parts that I've looked at from Rails are not that special.

>> On
>> the other hand it feels like I'm stretching D's metaprogramming
>> capabilities in my serialization library. That's just because D doesn't
>> have proper reflection.
>
> I've written three serialization libraries and two versions of a runtime
> reflection library; the first one was prior to the string bugs in CTFE
> being fixed and ran DMD out of memory. I found serialization to be
> simple and straight forward. And I found compiler bugs to be a greater
> issue with reflection than any limits in D's capabilities. Indeed, given
> I wrote the same library using two completely different mechanisms,
> gives me great hope for D. So, I don't agree with your assessment, but I
> do understand where you're coming from. A lot of D's metaprogramming
> capabilities are undocumented, and knowing the right tool for the job is
> essential to getting nice, clean code.

For example, getting the name of a instance variable:

T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];

I wouldn't consider that simple and straight forward. This works for both D1 and D2, don't know if D2 has a better way.

Your serialization library only works on public fields? Doesn't sound very useful.

> (Just checking, but you do know D has static ctors.)

Yes.

> Actually, you're not supposed to be able to; this is bug 1983.
> http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
> https://github.com/D-Programming-Language/dmd/pull/71

I don't see what that has to do with private, it's about const, as far as I can see. I don't see how calling a private method through a delegate can be easily solved. I see three solutions:

1. Make it a compile time error to create a delegate from a private method. This would not allow to call that method via a delegate in the same module, which is possible if you call the method directly.

2. Implement some, most likely, very advanced data flow analysis in the compiler to detect when a delegate pointing to a private method escapes the module. Results in a compile time error. Will most likely not happen.

3. Make it a runtime error when the delegate is actually called. This would probably need some pieces of 2 as well. I don't think the community will be happy about adding runtime checks like these.

Oh, BTW the same needs to be done for function pointers as well since you can build a delegate out of a function pointer and an object.

>> In Ruby an instance variable is always private. I don't think you can
>> declare a public instance variable, one uses setters/getters for that.
>> In D you can declare both private and public instance variables.
>>
>> In Ruby you can get/set a value of an instance variable using
>> reflection, instance_variable_set/get.
>> In D you can get/set a value of a private instance variable using
>> compile time reflection, i.e. .tupleof.
>
> Yes, but only for source code you control.

You don't need control of the source, you only need the static type. Of course an opaque struct won't work.

> Actually, baring bugs in DMD, .tupleof is the only way to access private
> fields and there is _no way_ to access private members; And .tupleof
> only works if you have access to the source code. Furthermore, .tupleof,
> god-send that it is, isn't part of the D spec; unlike .stringof, it
> didn't make it into TDPL nor into the D documentation. D's official
> method of compile-time reflection is __traits. And __traits doesn't
> bypass protection attributes. So precedence would be against
> runtime-reflection having access to private members.

The compiler has .tupleof and it's mentioned in the spec at d-programming-language.org: http://d-programming-language.org/struct.html

-- 
/Jacob Carlborg

I said it once, but nobody responded.

Why not generate full reflection for everything by default and add an attribute line @noreflect as an _optimization_ tool in cases, where this thing is definitely not gonna be used in reflection?

On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:
> I said it once, but nobody responded.
>
> Why not generate full reflection for everything by default and add an
> attribute line @noreflect as an _optimization_ tool in cases, where
> this thing is definitely not gonna be used in reflection?

As I see it, because you almost never need 'full reflection for everything', and idiomatic D code is not supposed to contain clutter like that.

You can always write @noreflect: at the start of your module and it
will work as expected. what's the problem with that?
Also, compiler would be able to automatically add that attribute to
any symbol, which is private and does not escape the module.
Again, not including reflection is purely _optimization_ issue and
should not be interfered with it's functionality.
Reflection is something you rarely have in mind when designing, for
example, classes.
Requiring explicit request for reflection defeats the purpose.

You could just as well say "writing public on every member is a clutter". It is, but you don't have to do it to achieve the same effect.

On Tue, Oct 25, 2011 at 2:58 PM, Timon Gehr <timon.gehr@gmx.ch> wrote:
> On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:
>>
>> I said it once, but nobody responded.
>>
>> Why not generate full reflection for everything by default and add an attribute line @noreflect as an _optimization_ tool in cases, where this thing is definitely not gonna be used in reflection?
>
> As I see it, because you almost never need 'full reflection for everything', and idiomatic D code is not supposed to contain clutter like that.
>

Hi,

I would love to see runtime reflection be available in D. I am currently reading Walter Bright's "The D
Programming Language" and with every page I read I became more and more fascinated and enthusiastic
about D. But then I started to wonder why I haven't read about runtime reflection yet, so I did some Internet
research and was a bit disappointed this feature isn't available. D is really cool, it's not only C done right,
but also Java and C#. Actually, to me, it's one of the best programming language available yet. It's really a
pitty RTTI is not available. So I hope your approach will somehow influence D's future :)

Keep going,
Jonny

Hear, hear!

On Tue, Oct 25, 2011 at 5:40 PM, Jonny Dee <jonnyd@gmx.net> wrote:
> Hi,
>
> I would love to see runtime reflection be available in D. I am currently reading Walter Bright's "The D
> Programming Language" and with every page I read I became more and more fascinated and enthusiastic
> about D. But then I started to wonder why I haven't read about runtime reflection yet, so I did some Internet
> research and was a bit disappointed this feature isn't available. D is really cool, it's not only C done right,
> but also Java and C#. Actually, to me, it's one of the best programming language available yet. It's really a
> pitty RTTI is not available. So I hope your approach will somehow influence D's future :)
>
> Keep going,
> Jonny
>