January 23, 2012
On Mon, 23 Jan 2012 01:33:47 +0100, Timon Gehr <timon.gehr@gmx.ch> wrote:

> On 01/23/2012 01:07 AM, so wrote:
>> On Mon, 23 Jan 2012 01:39:23 +0200, so <so@so.so> wrote:
>>
>>> I have been asking that for some time now, i am afraid you won't get
>>> much of an audience.
>>> You can get rid of both additional allocation and indirection but it
>>> is not pretty. We could definitely use some help/sugar on this.
>>>
>>> http://www.artima.com/cppsource/backyard3.html
>>
>> http://www.digitalmars.com/d/archives/digitalmars/D/Implementation_hiding_139625.html
>>
>>
>> There is another issue Walter forgot to mention in the article.
>> I think there might be a way but looks like we also loose the "destructor".
>> Which means we are all the way back to the
>> http://en.wikipedia.org/wiki/Opaque_pointer.
>>
>> Walter, is there a way to get around destructor limitation?
>
> This seems to work.
>
> a.di:
>
> final class A{
>      private this();
>      static A factory();
>      T1 publicMember1(int x);
>      T2 publicMember2(float y);
>      T3 publicField;
>      // ...
> }
>
> a.d:
>
> class A{
>      static A factory(){return new A;}
>      T1 publicMember1(int x){ ... }
>      T2 publicMember2(float y){ ... }
>      T3 publicField;
>      // ...
> private:
>      T1 field1;
>      T2 field2;
> }
>
This will even work with plain new/this, as the allocation is done
in the constructor. The difficulty is that you can't build inheritable pimpls.
January 23, 2012
On Mon, 23 Jan 2012 02:49:39 +0200, Martin Nowak <dawg@dawgfoto.de> wrote:

> This will even work with plain new/this, as the allocation is done
> in the constructor. The difficulty is that you can't build inheritable pimpls.

No, you can't define a struct without providing all the fields.
Inheritance has zero importance on this one. If you will use inheritance why would you need things like this in the first place?
January 23, 2012
On 01/23/2012 01:49 AM, Martin Nowak wrote:
> On Mon, 23 Jan 2012 01:33:47 +0100, Timon Gehr <timon.gehr@gmx.ch> wrote:
>
>> On 01/23/2012 01:07 AM, so wrote:
>>> On Mon, 23 Jan 2012 01:39:23 +0200, so <so@so.so> wrote:
>>>
>>>> I have been asking that for some time now, i am afraid you won't get
>>>> much of an audience.
>>>> You can get rid of both additional allocation and indirection but it
>>>> is not pretty. We could definitely use some help/sugar on this.
>>>>
>>>> http://www.artima.com/cppsource/backyard3.html
>>>
>>> http://www.digitalmars.com/d/archives/digitalmars/D/Implementation_hiding_139625.html
>>>
>>>
>>>
>>> There is another issue Walter forgot to mention in the article.
>>> I think there might be a way but looks like we also loose the
>>> "destructor".
>>> Which means we are all the way back to the
>>> http://en.wikipedia.org/wiki/Opaque_pointer.
>>>
>>> Walter, is there a way to get around destructor limitation?
>>
>> This seems to work.
>>
>> a.di:
>>
>> final class A{
>> private this();
>> static A factory();
>> T1 publicMember1(int x);
>> T2 publicMember2(float y);
>> T3 publicField;
>> // ...
>> }
>>
>> a.d:
>>
>> class A{
>> static A factory(){return new A;}
>> T1 publicMember1(int x){ ... }
>> T2 publicMember2(float y){ ... }
>> T3 publicField;
>> // ...
>> private:
>> T1 field1;
>> T2 field2;
>> }
>>
> This will even work with plain new/this, as the allocation is done
> in the constructor. The difficulty is that you can't build inheritable
> pimpls.

Ah, good to know. One solution would be to just add some dummy space to the class declaration (private void[256] dummy=void) so that the implementation can use it for private members. (union{void[256] dummy; struct{/*private_members*/}}) Of course, this is potentially wasteful.
January 23, 2012
> 
> What you're asking for is dynamic typing. You can achieve something equivalent using opDispatch and std.variant.

I am not sure I understood you correctly. It is dynamic typing more or less, just as polymorphic types like classes implement dynamic types, because you can change the implementation at runtime, by assigning an object of another derived class.

What I am asking for is hiding the implemenation of class completely, not only to the programmer but also to the compiler.
January 23, 2012
> Ah, good to know. One solution would be to just add some dummy space to the class declaration (private void[256] dummy=void) so that the implementation can use it for private members. (union{void[256] dummy; struct{/*private_members*/}}) Of course, this is potentially wasteful.

This is what I did for the thread pimpl, but it requires you to manually
keep the size in sync with the implementation.
January 23, 2012
On Monday, January 23, 2012 02:19:35 AM Timon Gehr wrote:
> On 01/23/2012 01:49 AM, Martin Nowak wrote:
> > On Mon, 23 Jan 2012 01:33:47 +0100, Timon Gehr <timon.gehr@gmx.ch> wrote:
> >> On 01/23/2012 01:07 AM, so wrote:
> >>> On Mon, 23 Jan 2012 01:39:23 +0200, so <so@so.so> wrote:
> >>>> I have been asking that for some time now, i am afraid you won't
> >>>> get
> >>>> much of an audience.
> >>>> You can get rid of both additional allocation and indirection but
> >>>> it
> >>>> is not pretty. We could definitely use some help/sugar on this.
> >>>> 
> >>>> http://www.artima.com/cppsource/backyard3.html
> >>> 
> >>> http://www.digitalmars.com/d/archives/digitalmars/D/Implementation_h iding_139625.html
> >>> 
Thanks for the links, it was a good read.
> >>> 
> >>> 
> >>> There is another issue Walter forgot to mention in the article.
> >>> I think there might be a way but looks like we also loose the
> >>> "destructor".
> >>> Which means we are all the way back to the
> >>> http://en.wikipedia.org/wiki/Opaque_pointer.
> >>> 
> >>> Walter, is there a way to get around destructor limitation?
What's the destructor limitation?
> >> 
> >> This seems to work.
> >> 
> >> a.di:
> >> 
> >> final class A{
> >> private this();
> >> static A factory();
> >> T1 publicMember1(int x);
> >> T2 publicMember2(float y);
> >> T3 publicField;
> >> // ...
> >> }
> >> 
> >> a.d:
> >> 
> >> class A{
> >> static A factory(){return new A;}
> >> T1 publicMember1(int x){ ... }
> >> T2 publicMember2(float y){ ... }
> >> T3 publicField;
> >> // ...
> >> private:
> >> T1 field1;
> >> T2 field2;
> >> }
> > 
> > This will even work with plain new/this, as the allocation is done
> > in the constructor. The difficulty is that you can't build inheritable
> > pimpls.
Hmm, I see some problems with this approach: It is not only impossible to derive from this class without using the di file with the full implementation (which might be acceptable)but you can't even use derived classes by an A reference, because the compiler would not use the vtbl because it assumes there are no derived classes. Which is not acceptable, in my opinion.

> 
> Ah, good to know. One solution would be to just add some dummy space to the class declaration (private void[256] dummy=void) so that the implementation can use it for private members. (union{void[256] dummy; struct{/*private_members*/}}) Of course, this is potentially wasteful.

I also thought of such variants, but as you already noted, they are either wasteful, or limited, or both.

The simplest and most efficient variant I could think of, was the one I
already explained, but I am illustrating it here with some code, to make it
more clear:
// base.d
import std.container;
class Base {
	this() {
	// ...
	}
	SList!int overrideMe() {
		//....
	}
@private_impl:
	int foo;
	float bar;
}

//////

Generate two .di files, with --private-impl compiler switch: dmd -H -c --private-impl base.d

The --private-impl switch would cause the following:
 1. It creates two ".di" files, see below.
 2. It creates a hidden static "size" field in the object file:

If you don't pass the --private-impl switch, just one standard .di file is generated and the "@private_impl:" is interpreted as plain "private:".

// base.d generated code:
import std.container;
class Base {
	this() {
	// ...
	}
	SList!int overrideMe() {
		//....
	}
@private_impl:
	int foo;
	float bar;
	static size_t __size_of_object=actual_base_object_size;
}

// base.d -- This is the public interface:
import std.container;
//Non public imports are still necessary, because the compiler needs to check
//that client code uses the same SList implementation.
// but imports only needed by method bodies, should be omitted, but how to
tell this the compiler?
class Base {
	this(); // No inline, we are hiding the implementation!
	SList!int overrideMe();
@private_impl: // Omit private data, just a hint to the compiler that there
are private hidden fields (and that there exists a hidden size field)
}

// base_private.di -- the second .di file, just the normally generated one.

If I want to create a derived class and I want to be independent of the base class implementation, I would simple import the public base.di file:

// derived.d
import base;
class D : Base {
	SList!int overrideMe() {
		// override it with something
		// access  my_field:
		my_field=2;
		// compiler would generate:
		int* p=cast(int*)
((cast(void*)this)+Base.__size_of_object+offset_of_my_field);
		*p=2;
		// instead of:
		int* p=cast(int*)
((cast(void*)this)+offset_of_my_field_including_statically_known_base_size);
		*p=2;
		// A class deriving from D, would also have to add Ds size and so on,
but as these offsets don't change during the life time of a program, the
calculated offsets could easily be cached.
	}
@private_impl:
	int my_field;
}

 If I don't care about implementation hiding, I would simple import the
base_private.di file, and I would have statically calculated offsets. Derived
classes in the same library would usually do that.

// main.d

import derived;
import std.container;

void main() {
	D d=new D;
	// I am using the public .di file, thus new has to look up the objects size
at runtime in the library and allocate the needed space, then it triggers the
now non inlined initialization sequence:
	// - initialize all base members
	// - initialize all derived members
	// - call base class constructor
	// - call derived class constructor
	// -> can be optimized if derived class uses the non public interface.

	// use d.
	// d gets destroyed, simply call the destructors, and free the used space.
}

If you don't rename the private implementations, but simply use another module
lookup path, you can switch implementations by no code change at all, simply
comple main.d with the private module search path and you get the old
behaviour.
--> client code can choose, without a code change in the library, if you don't
mind the static size fields, not even a recompile of the library is necessary.

I hope it is now more clear what my basic approach is. I think the benefits are overwhelming and it seems to be easy, maybe too easy. So I am pretty sure it is not that simple, but what is it I am missing?

Best regards,

Robert


January 23, 2012
On Mon, 23 Jan 2012 18:09:58 +0200, Robert Caravani <jfanatiker@gmx.at> wrote:

> Thanks for the links, it was a good read.

I think it is the best answer to the problem.

> What's the destructor limitation?

struct S {
  static S* make(); // constructor
  static void purge(S*); // destructor - you have to provide this as well
}

Limitation is that just like constructor, we also lose the destructor.
Which means we can't use "delete", and the tools designed for it.
As a result we don't have something new. It is exactly like C impl. hiding.

struct S;
S* make();
void purge(S*);
February 10, 2012
On Monday, 23 January 2012 at 17:09:30 UTC, so wrote:
> On Mon, 23 Jan 2012 18:09:58 +0200, Robert Caravani <jfanatiker@gmx.at> wrote:
>
>> Thanks for the links, it was a good read.
>
> I think it is the best answer to the problem.
>
>> What's the destructor limitation?
>
> struct S {
>  static S* make(); // constructor
>  static void purge(S*); // destructor - you have to provide this as well
> }
>
> Limitation is that just like constructor, we also lose the destructor.
> Which means we can't use "delete", and the tools designed for it.
> As a result we don't have something new. It is exactly like C impl. hiding.
>
> struct S;
> S* make();
> void purge(S*);

I think i finally got a solution.
(Sorry for C++ code)

--- pimpl.hpp
template<typename T>
struct pimpl : noncopyable
{
 virtual ~pimpl() {}
};

--- lib.hpp
struct lib : public pimpl<lib>
{
 void fun(...);
 ...
 static lib* make();
};

--- lib.cpp
struct lib_impl : lib
{
 ... // data
};

void lib::fun(...)
{
 auto& r = *static_cast<lib_impl*>(this);
 ...
}

lib* lib::make()
{
 return new lib_impl;
}
February 19, 2012
On Friday 10 February 2012 20:23:59 so wrote:
> I think i finally got a solution.
> (Sorry for C++ code)
Never mind, it is my mother tongue.
> 
> --- pimpl.hpp
> template<typename T>
> struct pimpl : noncopyable
> {
>   virtual ~pimpl() {}
> };
> 
> --- lib.hpp
> struct lib : public pimpl<lib>
> {
>   void fun(...);
>   ...
>   static lib* make();
> };
> 
> --- lib.cpp
> struct lib_impl : lib
> {
>   ... // data
> };
> 
> void lib::fun(...)
> {
>   auto& r = *static_cast<lib_impl*>(this);
>   ...
> }
> 
> lib* lib::make()
> {
>   return new lib_impl;
> }

Well it is not bad, but how do you support derived classes with this scheme? Derived classes would have to derive from the implementation, so it would have to be public and compiled code using the derived class would break on changes of the private fields. Am I misinterpreting something?

What's the purpose of the pimpl template?

What do you think about my proposal? Is it sane and feasible at all?

Best regards,

Robert
February 19, 2012
On Sunday, 19 February 2012 at 17:25:51 UTC, Robert Caravani wrote:
> On Friday 10 February 2012 20:23:59 so wrote:
>> I think i finally got a solution.
>> (Sorry for C++ code)
> Never mind, it is my mother tongue.
>> 
>> --- pimpl.hpp
>> template<typename T>
>> struct pimpl : noncopyable
>> {
>>   virtual ~pimpl() {}
>> };
>> 
>> --- lib.hpp
>> struct lib : public pimpl<lib>
>> {
>>   void fun(...);
>>   ...
>>   static lib* make();
>> };
>> 
>> --- lib.cpp
>> struct lib_impl : lib
>> {
>>   ... // data
>> };
>> 
>> void lib::fun(...)
>> {
>>   auto& r = *static_cast<lib_impl*>(this);
>>   ...
>> }
>> 
>> lib* lib::make()
>> {
>>   return new lib_impl;
>> }
>
> Well it is not bad, but how do you support derived classes with this scheme? Derived classes would have to derive from the implementation, so it would have to be public and compiled code using the derived class would break on changes of the private fields. Am I misinterpreting something?

Whole purpose of this is hiding implementation from user with "zero" cost.
Derived classes has no places here. Using anything about derived classes means "zero" cost is not your first concern. So just use pure classes / interfaces, they are much cleaner and to the point.

> What's the purpose of the pimpl template?

template<typename T>
struct pimpl : noncopyable
{
  virtual ~pimpl() {}
};

"virtual ~pimpl() {}"

This is to get rid of leaking, you know we need to provide a "virtual destructor" for an interface otherwise a base class has no way of knowing it is a "base" class and when you delete a pointer you would free only the memory base class allocated. But i guess you are asking why it should be like this, another way is:

#define PIMPL(x) \
	private: \
		x(const x&); \
		const x& operator=(const x&); \
	protected: \
		x() {} \
	public: \
		virtual	~x() {}

now we can just use:

struct lib
{
   PIMPL(lib)
   ...
   ...
};

I don't understand why this didn't get much attention either, i am now using it in my framework and it rocks!

> What do you think about my proposal? Is it sane and feasible at all?

I am having trouble understanding the need for supporting class hierarchies.
Deriving from a private implementation feels quite wrong. A pimpl IMO should be a black box.