On 02/14/2012 01:28 AM, Jonathan M Davis wrote:
> On Tuesday, February 14, 2012 00:36:31 Timon Gehr wrote:
>> On 02/14/2012 12:26 AM, Jonathan M Davis wrote:
>>> On Tuesday, February 14, 2012 00:15:40 Martin Nowak wrote:
>>>> Can you elaborate on what issues you see with NVI. After all it's only
>>>> the final public
>>>> method that needs to call the virtual private methods.
>>>
>>> As I explain in that bug report, NVI can be acheived with protected
>>> rather than private. So, we can do NVI now with private being
>>> non-virtual. However, if we make private virtual, then the compiler can
>>> no longer inline private functions in classes unless they're marked as
>>> final. That's a performance hit to pretty much every class ever
>>> written.
>>>
>>> Yes, the conscientious programmer will mark their classes' private
>>> functions final unless they intend to use them for NVI, but the
>>> _default_ is then inefficient for little to no gain. All you gain is
>>> the ability to do NVI with private instead of protected, and that is
>>> not even vaguely worth the performance hit to almost every D class ever
>>> IMHO.
>>>
>>> - Jonathan M Davis
>>
>> Virtual private does not allow for cross-module NVI, so what would be
>> the point anyway?
>
> Why wouldn't it? The whole point of NVI is that you override it but don't call
> it - the base class does that. So, the derived class could override the
> private function but couldn't call it. It _is_ a bit weird though, all the
> same.
>

It would break information hiding. A private method declared in a different module is not supposed to be visible (hopefully will get fixed), let alone overrideable.

> Regardless, I'm against making private virtual because of the needless
> performance hit.
>

There are no performance implications because the compiler has the whole module ready to analyse. Therefore it can de-virtualise any calls to private members that are never overridden. A trivial addition to the class layout could furthermore allow the compiler to remove the vtable entry for such a member function entirely (essentially, virtual private members would get negative vtable offsets). I actually think that should be implemented in order to get a potential TDPL error off the table.

On Tuesday, February 14, 2012 01:55:51 Timon Gehr wrote:
> On 02/14/2012 01:28 AM, Jonathan M Davis wrote:
> It would break information hiding. A private method declared in a
> different module is not supposed to be visible (hopefully will get
> fixed), let alone overrideable.

You misunderstand then. Private symbols are most definitely visible. Access modifiers do _not_ affect visibility, only accessibility. Private symbols are visible _by design_. That's how it works in C++, and D inherited that. The whole point of this thread is to discuss whether access modifiers should affect visibility, but right now they _do_, and it's _on purpose_ that they do. It's not a bug.

As it stands, if I declare a private function which conflicts with a public function, the compiler will complain. For example, If I were to add a private abs function to std.file which took an int, then any module which imported both std.math and std.file would have to use std.math.abs instead of just abs, because there would be a conflict. The fact that std.file's abs was private wouldn't mean anything to overload resolution.

In fact, just recently, a private abs function was added to core.time for what it needed, and it caused problems. It had to be changed to _abs to not screw up std.math.abs.

The whole point of this thread is to discuss whether access modifiers should be changed to affect visibility rather than access.

> > Regardless, I'm against making private virtual because of the needless performance hit.
> 
> There are no performance implications because the compiler has the whole module ready to analyse. Therefore it can de-virtualise any calls to private members that are never overridden. A trivial addition to the class layout could furthermore allow the compiler to remove the vtable entry for such a member function entirely (essentially, virtual private members would get negative vtable offsets). I actually think that should be implemented in order to get a potential TDPL error off the table.

No. It can't. Because a derived class in an entirely different module could override the private function. private does _not_ affect visibility. The whole idea of overriding private for NVI is to allow derived classes to override the private function's implementation but not call it. If private becomes virtual, then the compiler loses the ability to inline non-final private functions unless it can guarantee that it's dealing with that exact type, which it usually can't do in the type's own member functions.

- Jonathan M Davis

On 02/14/2012 01:55 AM, Timon Gehr wrote:
>
> There are no performance implications because the compiler has the whole
> module ready to analyse. Therefore it can de-virtualise any calls to
> private members that are never overridden. A trivial addition to the
> class layout could furthermore allow the compiler to remove the vtable
> entry for such a member function entirely (essentially, virtual private
> members would get negative vtable offsets). I actually think that should
> be implemented in order to get a potential TDPL error off the table.
>

Nevermind, separate compilation could get tricky.

On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
> On Tuesday, February 14, 2012 01:55:51 Timon Gehr wrote:
>> On 02/14/2012 01:28 AM, Jonathan M Davis wrote:
>> It would break information hiding. A private method declared in a
>> different module is not supposed to be visible (hopefully will get
>> fixed), let alone overrideable.
>
> You misunderstand then. Private symbols are most definitely visible. Access
> modifiers do _not_ affect visibility, only accessibility. Private symbols are
> visible _by design_. That's how it works in C++, and D inherited that. The
> whole point of this thread is to discuss whether access modifiers should affect
> visibility, but right now they _do_, and it's _on purpose_ that they do. It's
> not a bug.
>

It is a bug on purpose.


> As it stands, if I declare a private function which conflicts with a public
> function, the compiler will complain. For example, If I were to add a private
> abs function to std.file which took an int, then any module which imported both
> std.math and std.file would have to use std.math.abs instead of just abs,
> because there would be a conflict. The fact that std.file's abs was private
> wouldn't mean anything to overload resolution.
>

Borken.

> In fact, just recently, a private abs function was added to core.time for what
> it needed, and it caused problems. It had to be changed to _abs to not screw
> up std.math.abs.
>

Exactly what I am saying. This is laughable (!) and should vanish from Ds otherwise very sane approach at modularity.

> The whole point of this thread is to discuss whether access modifiers should be
> changed to affect visibility rather than access.
>

Most definitely yes. The only hindrance are implementation issues.

>>> Regardless, I'm against making private virtual because of the needless
>>> performance hit.
>>
>> There are no performance implications because the compiler has the whole
>> module ready to analyse. Therefore it can de-virtualise any calls to
>> private members that are never overridden. A trivial addition to the
>> class layout could furthermore allow the compiler to remove the vtable
>> entry for such a member function entirely (essentially, virtual private
>> members would get negative vtable offsets). I actually think that should
>> be implemented in order to get a potential TDPL error off the table.
>
> No. It can't. Because a derived class in an entirely different module could
> override the private function. private does _not_ affect visibility.

Well, not being able override what cannot be accesses is a quite basic requirement of security. Private members cannot be overriden in a different module.

> The whole idea of overriding private for NVI is to allow derived classes to override the
> private function's implementation but not call it.

This is confusing method implementation with overriding. I just read the relevant sections of TDPL:

"D fully supports NVI by providing _specific guarantees_ when *interfaces* use access specifiers."

It is an explicit special case because private wouldn't have much of a meaning for interfaces otherwise. This means private interface members would not be hidden, they are special _already_.


> If private becomes virtual, then the compiler loses the ability to inline non-final private functions
> unlessit can guarantee that it's dealing with that exact type, which it 
usually can't do in the type's own member functions.
>

The optimisation works for all private class members. The issue is the virtual table layout for separate compilation with .di files, therefore it is not a good idea to make private methods virtual. But that is not required to satisfy TDPL anyway:

As far as I can see TDPL only requires private *interface* members to be implementable. That is not a problem at all, interface vtbls use thunking anyway. Therefore the private members could just be plugged there and be called virtually through base interface calls and non-virtually otherwise.

On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:
> On 02/14/2012 02:16 AM, Jonathan M Davis wrote:

> Well, not being able override what cannot be accesses is a quite basic requirement of security. Private members cannot be overriden in a different module.

Have you ever read up on NVI? That's how it's supposed to work. The whole idea is to have derived classes override functions without being able to use them. It makes it so that the public function is non-virtual and contains whatever stuff you want to guarantee is called before or after the private function, which is then virtual.

http://www.gotw.ca/publications/mill18.htm

> > The whole idea of overriding private for NVI is to allow derived classes to override the private function's implementation but not call it.
> 
> This is confusing method implementation with overriding. I just read the relevant sections of TDPL:
> 
> "D fully supports NVI by providing _specific guarantees_ when *interfaces* use access specifiers."
> 
> It is an explicit special case because private wouldn't have much of a meaning for interfaces otherwise. This means private interface members would not be hidden, they are special _already_.

The fact that TDPL talks about interfaces in specific could be a good argument for _not_ making private virtual for classes. But regardless, as it stands, _nothing_ which is private is hidden. So, there's nothing special going on with interfaces and private, except that you can't override them to actually do NVI, so they're kind of pointless.

> > If private becomes virtual, then the compiler loses the ability to inline non-final private functions unlessit can guarantee that it's dealing with that exact type, which it
> usually can't do in the type's own member functions.
> 
> The optimisation works for all private class members. The issue is the virtual table layout for separate compilation with .di files, therefore it is not a good idea to make private methods virtual. But that is not required to satisfy TDPL anyway:
> 
> As far as I can see TDPL only requires private *interface* members to be implementable. That is not a problem at all, interface vtbls use thunking anyway. Therefore the private members could just be plugged there and be called virtually through base interface calls and non-virtually otherwise.

The optimization only works as long as the compiler can guarantee that no derived classes override the private functions, since it doesn't necessarily know about all of the derived classes. If D allowed NVI with classes the way that C++ does by making private virtual, then derived classes would _have_ to be able to override private functions in base classes which are in separate modules; they might even be in separate libraries. And then the compiler could not guarantee that no derived classes overrode the private functions, and the optimization could not be made. Hence why I'm arguing against making private functions virtual.

Personally, I'd very much like to see access level affect visibility, but Walter must be convinced.

- Jonathan M Davis

On 02/14/2012 03:05 AM, Jonathan M Davis wrote:
> On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:
>> On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
>
>> Well, not being able override what cannot be accesses is a quite basic
>> requirement of security. Private members cannot be overriden in a
>> different module.
>
> Have you ever read up on NVI? That's how it's supposed to work. The whole idea
> is to have derived classes override functions without being able to use them.
> It makes it so that the public function is non-virtual and contains whatever
> stuff you want to guarantee is called before or after the private function,
> which is then virtual.
>
> http://www.gotw.ca/publications/mill18.htm
>

That article uses C++. It is not really a fair comparison, because in D there is no way to explicitly declare a method "private virtual", and D's private is different from C++'s private, and C++ has a different sub-typing and inheritance model. I think there are different design trade-offs at play.

But I can make better sense of your point of view now: When I was talking about virtual I meant just a function that has an entry in a vtbl and is invoked virtually, while you probably had C++'s semantics in mind.


>>> The whole idea of overriding private for NVI is to allow derived classes
>>> to override the private function's implementation but not call it.
>>
>> This is confusing method implementation with overriding. I just read the
>> relevant sections of TDPL:
>>
>> "D fully supports NVI by providing _specific guarantees_ when
>> *interfaces* use access specifiers."
>>
>> It is an explicit special case because private wouldn't have much of a
>> meaning for interfaces otherwise. This means private interface members
>> would not be hidden, they are special _already_.
>
> The fact that TDPL talks about interfaces in specific could be a good argument
> for _not_ making private virtual for classes. But regardless, as it stands,
> _nothing_ which is private is hidden. So, there's nothing special going on
> with interfaces and private, except that you can't override them to actually
> do NVI, so they're kind of pointless.
>

The fact that they cannot be implemented in a class that implements the interface seems to be a [TDPL] bug.


>>> If private becomes virtual, then the compiler loses the ability to
>>> inline non-final private functions unlessit can guarantee that it's
>>> dealing with that exact type, which it
>> usually can't do in the type's own member functions.
>>
>> The optimisation works for all private class members. The issue is the
>> virtual table layout for separate compilation with .di files, therefore
>> it is not a good idea to make private methods virtual. But that is not
>> required to satisfy TDPL anyway:
>>
>> As far as I can see TDPL only requires private *interface* members to be
>> implementable. That is not a problem at all, interface vtbls use
>> thunking anyway. Therefore the private members could just be plugged
>> there and be called virtually through base interface calls and
>> non-virtually otherwise.
>
> The optimization only works as long as the compiler can guarantee that no
> derived classes override the private functions, since it doesn't necessarily
> know about all of the derived classes.

The optimization works under the precondition that derived classes from a different module cannot override private member functions.

> If D allowed NVI with classes the way that C++ does by making private virtual, then derived classes would _have_ to
> be able to override private functions in base classes which are in separate
> modules; they might even be in separate libraries. And then the compiler could
> not guarantee that no derived classes overrode the private functions, and the
> optimization could not be made. Hence why I'm arguing against making private
> functions virtual.
>

Here you probably mean virtual as in C++. I agree.

> Personally, I'd very much like to see access level affect visibility, but
> Walter must be convinced.
>

I think there were some issues with overload sets, but I think overloads with differing accessibility should just be disallowed.

On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:
> On 02/14/2012 03:05 AM, Jonathan M Davis wrote:
> > On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:
> >> On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
> >> 
> >> Well, not being able override what cannot be accesses is a quite basic requirement of security. Private members cannot be overriden in a different module.
> > 
> > Have you ever read up on NVI? That's how it's supposed to work. The whole idea is to have derived classes override functions without being able to use them. It makes it so that the public function is non-virtual and contains whatever stuff you want to guarantee is called before or after the private function, which is then virtual.
> > 
> > http://www.gotw.ca/publications/mill18.htm
> 
> That article uses C++. It is not really a fair comparison, because in D there is no way to explicitly declare a method "private virtual", and D's private is different from C++'s private, and C++ has a different sub-typing and inheritance model. I think there are different design trade-offs at play.
> 
> But I can make better sense of your point of view now: When I was talking about virtual I meant just a function that has an entry in a vtbl and is invoked virtually, while you probably had C++'s semantics in mind.

I'm not aware of _any_ difference between virtual functions in C++ and D except for the fact that all public and protected member functions are always virtual in D (unless the compiler is able to determine that it can make them non- virtual), and currently private and protected functions are never virtual. What differences are you referring to?

- Jonathan M Davis

On 02/14/2012 03:54 AM, Jonathan M Davis wrote:
> On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:
>> On 02/14/2012 03:05 AM, Jonathan M Davis wrote:
>>> On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:
>>>> On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
>>>>
>>>> Well, not being able override what cannot be accesses is a quite basic
>>>> requirement of security. Private members cannot be overriden in a
>>>> different module.
>>>
>>> Have you ever read up on NVI? That's how it's supposed to work. The whole
>>> idea is to have derived classes override functions without being able to
>>> use them. It makes it so that the public function is non-virtual and
>>> contains whatever stuff you want to guarantee is called before or after
>>> the private function, which is then virtual.
>>>
>>> http://www.gotw.ca/publications/mill18.htm
>>
>> That article uses C++. It is not really a fair comparison, because in D
>> there is no way to explicitly declare a method "private virtual", and
>> D's private is different from C++'s private, and C++ has a different
>> sub-typing and inheritance model. I think there are different design
>> trade-offs at play.
>>
>> But I can make better sense of your point of view now: When I was
>> talking about virtual I meant just a function that has an entry in a
>> vtbl and is invoked virtually, while you probably had C++'s semantics in
>> mind.
>
> I'm not aware of _any_ difference between virtual functions in C++ and D except
> for the fact that all public and protected member functions are always virtual
> in D (unless the compiler is able to determine that it can make them non-
> virtual), and currently private and protected functions are never virtual.
> What differences are you referring to?
>
> - Jonathan M Davis

The main difference is that C++ has an explicit 'virtual' modifier that naturally grants override-access to private symbols.

On Tuesday, February 14, 2012 04:39:18 Timon Gehr wrote:
> On 02/14/2012 03:54 AM, Jonathan M Davis wrote:
> > On Tuesday, February 14, 2012 03:45:54 Timon Gehr wrote:
> >> On 02/14/2012 03:05 AM, Jonathan M Davis wrote:
> >>> On Tuesday, February 14, 2012 02:43:29 Timon Gehr wrote:
> >>>> On 02/14/2012 02:16 AM, Jonathan M Davis wrote:
> >>>> 
> >>>> Well, not being able override what cannot be accesses is a quite basic requirement of security. Private members cannot be overriden in a different module.
> >>> 
> >>> Have you ever read up on NVI? That's how it's supposed to work. The
> >>> whole
> >>> idea is to have derived classes override functions without being able to
> >>> use them. It makes it so that the public function is non-virtual and
> >>> contains whatever stuff you want to guarantee is called before or after
> >>> the private function, which is then virtual.
> >>> 
> >>> http://www.gotw.ca/publications/mill18.htm
> >> 
> >> That article uses C++. It is not really a fair comparison, because in D there is no way to explicitly declare a method "private virtual", and D's private is different from C++'s private, and C++ has a different sub-typing and inheritance model. I think there are different design trade-offs at play.
> >> 
> >> But I can make better sense of your point of view now: When I was talking about virtual I meant just a function that has an entry in a vtbl and is invoked virtually, while you probably had C++'s semantics in mind.
> > 
> > I'm not aware of _any_ difference between virtual functions in C++ and D except for the fact that all public and protected member functions are always virtual in D (unless the compiler is able to determine that it can make them non- virtual), and currently private and protected functions are never virtual. What differences are you referring to?
> > 
> > - Jonathan M Davis
> 
> The main difference is that C++ has an explicit 'virtual' modifier that naturally grants override-access to private symbols.

In other words, the only difference is what I said - that public and protected in D are automatically virtual, and private and protected are automatically non-virtual. So, assuming that private were made virtual, it would be the same as a private virtual function in C++ as far as virtuality goes. The only differences would be that D effectively makes all of the functions in the rest of the module friends and has no other way of declaring friends.

Regardless, I think that we agree that it would be better if private were non- virtual and access level also affected visibility so that private functions were not visible outside of the module that they're declared in.

- Jonathan M Davis

> private and protected are automatically non-virtual

You've said that twice now, I think you mean private and package.