I've seen some discussions before about people dealing with opaque
pointers, or handle types in D.
It's a common C practise, and there seems to be uncertainty about the best
way to implement this on the D side of the fence.

So I'm looking for opinions... hit me?

I have this in C:

  struct Thing;
  Thing* MakeThing();

And in D:

  struct Thing;
  extern (C) Thing* MakeThing();

The question is, does this suck?
D currently can't allocate an array of Thing*'s for some weird reason.

Are there any concerns relating to the GC that I should be aware of when handling pointers returned to me from C code?


So a further goal, and maybe I'll add a little back-story; my 30th birthday
game jam from a few weeks back (which is still an active thread!) was
interfacing with my engine using basically an extern(C) API precisely
mirroring the C headers.
I'm planning on joining another game jam in a couple of week, using D again
(maybe I can start another thread as long and controversial as the last
one!) ;)
But this time, I want to spend some preparation time building a much more
D-styled API to interact with. Again, I'm struggling with just how I should
wrap the C calls and opaque types up.

What do people feel is conventional, and has proven to work well?

Again, wrt the opaque pointer problem. I don't really feel using an opaque
pointer on the D side is really right.
What I really want to use is something that feels more like a class
reference, and I toyed with the idea of abusing class references to hold my
handles, and methods would internally case 'this' to my handle type before
passing it back into the C code it wraps up.
This seems okay at first, but then you realise there are serious problems
with new (wants to return new memory, I can't hook the new operator?), and
there's no way to free it automatically, since D is garbage collected
rather than ref counted >_<
(Note: I'm still fairly certain that I want a ref-counting GC in D)

So... should I make it a struct with a single void* member?

struct MyThing { void* handle; }

I kinda hate this. the struct keyword totally gives the user the wrong impression, and people have a natural instinct to avoid passing structs around by value, which is exactly how these objects should be handled...

So if I can't present it as a class, and a struct is a pretty shit solution... what?

My objects are opaque handles to engine-managed resources. They are ref-counted, and there is extern(C) API to add/release references to stuff, which I would rather have automated in the D code...

One final problem with ref-counting in D. Since it's not a 1st class feature and requires you to store a ref-count in your object, you can't make your objects immutable anymore because the ref-count needs to be bumped about as the object moves around. Anyone approached this problem?

> One final problem with ref-counting in D. Since it's not a 1st class
> feature and requires you to store a ref-count in your object, you can't
> make your objects immutable anymore because the ref-count needs to be
> bumped about as the object moves around. Anyone approached this problem?

I use a shared_ptr (especially if I deal with C memory).

On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
> This seems okay at first, but then you realise there are serious problems with new (wants to return new memory, I can't hook the new operator?),

There is an override for new, but it is deprecated in d2.
http://dlang.org/memory.html#newdelete

> I kinda hate this. the struct keyword totally gives the user the wrong impression, and people have a natural instinct to avoid passing structs around by value

...just don't do that? You don't write "ref Class obj" in D, so just pretend the struct is a class and do it the same way.

The only time you'd even need to know it is a struct is if you look at the source, and there you can write

struct /* but pretend it is a class! */ whatever {

}



Here's how I might do it. Given this simple C test header:

struct Test;

struct Test* makeTest(int num);
void addTestRef(struct Test* t);
void killTest(struct Test* t); // i should have called that releaseref lol
int getNumber(struct Test* t);
void setNumber(struct Test* t, int n);



We might use it in D like this:


// this represents the struct Test in C, our opaque pointer
// (the disabled stuff is to workaround a minor dmd bug)
struct c_Test {
	@disable this();
	@disable this(this);
}

// and this is our D wrapper
// always pass this by value
struct Test {
       // the internal reference
	private c_Test* c_ptr;

	// construction can only be done publicly through the static make method
	@disable this();
	private this(c_Test* ptr) {
		c_ptr = ptr;
	}

	public static Test make(int num) {
		return Test(makeTest(num));
	}

	// the alias this lets us call the rest of the binded C functions with UFCS without manually writing wrappers
	c_Test* getCPointer() { return c_ptr; } // it does a property so you can't assign to the internal pointer
	// you CAN break the refcounting with this, but since you have to specifically ask for the uglier c_Test* to trigger
	// this, it is unlikely to happen by accident.
	alias getCPointer this;

	// refcount stuff
	~this() {
		c_ptr.killTest();
	}
	this(this) {
		c_ptr.addTestRef();
	}
}

// you might notice the above is pretty generic, perhaps it could all be a single template so you don't rewrite it for too many types.

// and the C function prototypes
extern(C) {
	c_Test* makeTest(int num);
	void addTestRef(c_Test* t);
	void killTest(c_Test* t);

	int getNumber(c_Test* t);
	void setNumber(c_Test* t, int n);
}

// test program
import core.stdc.stdio;
void foo(Test t) {
	printf("foo on %d\n", t.getNumber());
	t.setNumber(20);
}

void main() {
	auto t = Test.make(12);
	auto t2 = Test.make(24);

	printf("about to call foo\n");
	foo(t);
	printf("done calling foo\n");

	printf("main with t  == %d\n", t.getNumber());
	printf("main with t2 == %d\n", t2.getNumber());
}




You don't have my c implementation but it isn't special and running the program produced the following output:

Test made with number 12
Test made with number 24
about to call foo
aref refcount now 2 on Test 12 # refcount increased properly for the function call
foo on 12
dtor refcount now 1 on Test 20 # and properly decreased when ending
done calling foo
main with t  == 20 # setting the number in foo() correctly did it by reference
main with t2 == 24
dtor refcount now 0 on Test 24 # and main is dead, so we release the ref automatically
Test killed with number 24 # refcount == 0, so C free()'d it
dtor refcount now 0 on Test 20
Test killed with number 20




Using UFCS for the C functions might be a little bit weird, but saves tediously writing them all out again to forward inside the struct.

On 19 September 2013 02:57, Adam D. Ruppe <destructionator@gmail.com> wrote:

> On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
>
>> This seems okay at first, but then you realise there are serious problems with new (wants to return new memory, I can't hook the new operator?),
>>
>
> There is an override for new, but it is deprecated in d2. http://dlang.org/memory.html#**newdelete<http://dlang.org/memory.html#newdelete>
>
>
>  I kinda hate this. the struct keyword totally gives the user the wrong
>> impression, and people have a natural instinct to avoid passing structs around by value
>>
>
> ...just don't do that? You don't write "ref Class obj" in D, so just pretend the struct is a class and do it the same way.
>
> The only time you'd even need to know it is a struct is if you look at the source, and there you can write
>
> struct /* but pretend it is a class! */ whatever {
> }
>

Except those times when you read "struct Thing" in the documentation, or
hover over it in the IDE and it says "struct Thing", or when the syntax
hilighting makes it go the struct colour.
It's still a struct, it's obviously a struct, and that communicates some
amount of intent.
What you describe is a hack, and I don't want to build the foundation of a
performance-oriented library upon a hack. If that's as good as it gets,
then there's a deficiency in D here that we need to think about.

I also don't have faith in many compilers passing struct's-with-only-a-single-member to functions by that member's type rather than as a struct.

Here's how I might do it. Given this simple C test header:
>
> struct Test;
>
> struct Test* makeTest(int num);
> void addTestRef(struct Test* t);
> void killTest(struct Test* t); // i should have called that releaseref lol
> int getNumber(struct Test* t);
> void setNumber(struct Test* t, int n);
>
>
>
> We might use it in D like this:
>
>
> // this represents the struct Test in C, our opaque pointer
> // (the disabled stuff is to workaround a minor dmd bug)
> struct c_Test {
>         @disable this();
>         @disable this(this);
> }
>
> // and this is our D wrapper
> // always pass this by value
> struct Test {
>        // the internal reference
>         private c_Test* c_ptr;
>
>         // construction can only be done publicly through the static make
> method
>         @disable this();
>         private this(c_Test* ptr) {
>                 c_ptr = ptr;
>         }
>
>         public static Test make(int num) {
>                 return Test(makeTest(num));
>         }
>
>         // the alias this lets us call the rest of the binded C functions
> with UFCS without manually writing wrappers
>         c_Test* getCPointer() { return c_ptr; } // it does a property so
> you can't assign to the internal pointer
>         // you CAN break the refcounting with this, but since you have to
> specifically ask for the uglier c_Test* to trigger
>         // this, it is unlikely to happen by accident.
>         alias getCPointer this;
>
>         // refcount stuff
>         ~this() {
>                 c_ptr.killTest();
>         }
>         this(this) {
>                 c_ptr.addTestRef();
>         }
> }
>
> // you might notice the above is pretty generic, perhaps it could all be a single template so you don't rewrite it for too many types.
>
> // and the C function prototypes
> extern(C) {
>         c_Test* makeTest(int num);
>         void addTestRef(c_Test* t);
>         void killTest(c_Test* t);
>
>         int getNumber(c_Test* t);
>         void setNumber(c_Test* t, int n);
> }
>
> // test program
> import core.stdc.stdio;
> void foo(Test t) {
>         printf("foo on %d\n", t.getNumber());
>         t.setNumber(20);
> }
>
> void main() {
>         auto t = Test.make(12);
>         auto t2 = Test.make(24);
>
>         printf("about to call foo\n");
>         foo(t);
>         printf("done calling foo\n");
>
>         printf("main with t  == %d\n", t.getNumber());
>         printf("main with t2 == %d\n", t2.getNumber());
> }
>
>
>
>
> You don't have my c implementation but it isn't special and running the program produced the following output:
>
> Test made with number 12
> Test made with number 24
> about to call foo
> aref refcount now 2 on Test 12 # refcount increased properly for the
> function call
> foo on 12
> dtor refcount now 1 on Test 20 # and properly decreased when ending
> done calling foo
> main with t  == 20 # setting the number in foo() correctly did it by
> reference
> main with t2 == 24
> dtor refcount now 0 on Test 24 # and main is dead, so we release the ref
> automatically
> Test killed with number 24 # refcount == 0, so C free()'d it
> dtor refcount now 0 on Test 20
> Test killed with number 20
>
>
>
>
> Using UFCS for the C functions might be a little bit weird, but saves tediously writing them all out again to forward inside the struct.
>

Well, this is precisely what I've tried before, but I don't like it. It
feels like a hack.
Can you make me some promises about the ABI for passing that struct by
value across every compiler+architecture?

I don't like that it's a struct, it's NOT a struct, it's a handle.

There was some discussion about windows HANDLE types recently. I can't
remember where I saw it, but I felt like that was quite relevant.
It even dealt with the problem of HWND implicitly casting to a HANDLE, and
I also have that problem here.

On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
> One final problem with ref-counting in D. Since it's not a 1st class
> feature and requires you to store a ref-count in your object, you can't
> make your objects immutable anymore because the ref-count needs to be
> bumped about as the object moves around. Anyone approached this problem?

I'de like to here some guidelines regarding this too. I tried once to manage some resources with ref-counted structs. The problem was that I could not hold them in classes anymore, so for deterministic life-management I had to use just structs, everywhere...

On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
> struct MyThing { void* handle; }
>
> I kinda hate this. the struct keyword totally gives the user the wrong
> impression, and people have a natural instinct to avoid passing structs
> around by value, which is exactly how these objects should be handled...

You've got some wrong people, lol. How do they work with smart pointers?

On Thursday, 19 September 2013 at 09:17:09 UTC, Kagamin wrote:
> On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
>> struct MyThing { void* handle; }
>>
>> I kinda hate this. the struct keyword totally gives the user the wrong
>> impression, and people have a natural instinct to avoid passing structs
>> around by value, which is exactly how these objects should be handled...
>
> You've got some wrong people, lol. How do they work with smart pointers?

I had a similar problem not long ago and ended up using RefCount from std.typecons. It worked well enough for my needs, maybe it would be an option here.

I'm not sure of the performance though for a game situation, you'd have to test it.

On Thursday, 19 September 2013 at 09:53:28 UTC, growler wrote:
> On Thursday, 19 September 2013 at 09:17:09 UTC, Kagamin wrote:
>> On Wednesday, 18 September 2013 at 16:05:00 UTC, Manu wrote:
>>> struct MyThing { void* handle; }
>>>
>>> I kinda hate this. the struct keyword totally gives the user the wrong
>>> impression, and people have a natural instinct to avoid passing structs
>>> around by value, which is exactly how these objects should be handled...
>>
>> You've got some wrong people, lol. How do they work with smart pointers?
>
> I had a similar problem not long ago and ended up using RefCount from std.typecons. It worked well enough for my needs, maybe it would be an option here.
>
> I'm not sure of the performance though for a game situation, you'd have to test it.

Yes, that should be enough. But I, for my part, do not trust the phobos solution completely. Therefore I use my own.