Hello, I'm attempting to interface with the Steam API DLL in D and running into some trouble working with callbacks.  I'm aware there's already a project here http://derelict-steamworks.dub.pm/ but it seems to have not yet addressed the same issue.  Steam provides ways to poll for whether an asynchronous request has completed yet and retrieve the results, but what I am trying to implement is receiving actual callbacks from the API.

Just for reference, I've gotten simple callbacks working in other APIs (passing a D function pointer to a C function and having it be called back), and the other basic Steam functions that don't deal with callbacks all work fine (defined in D as extern(C)).  Here's where I'm running into trouble:

Definitions in C++ headers provided by Steam:

typedef uint64 SteamAPICall_t;
S_API void S_CALLTYPE SteamAPI_RegisterCallResult( class CCallbackBase *pCallback, SteamAPICall_t hAPICall );
S_API void S_CALLTYPE SteamAPI_UnregisterCallResult( class CCallbackBase *pCallback, SteamAPICall_t hAPICall );

class CCallbackBase
{
public:
	CCallbackBase() { m_nCallbackFlags = 0; m_iCallback = 0; }
	// don't add a virtual destructor because we export this binary interface across dll's
	virtual void Run( void *pvParam ) = 0;
	virtual void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall ) = 0;
	int GetICallback() { return m_iCallback; }
	virtual int GetCallbackSizeBytes() = 0;

protected:
	enum { k_ECallbackFlagsRegistered = 0x01, k_ECallbackFlagsGameServer = 0x02 };
	uint8 m_nCallbackFlags;
	int m_iCallback;
	friend class CCallbackMgr;

private:
	CCallbackBase( const CCallbackBase& );
	CCallbackBase& operator=( const CCallbackBase& );
};


It's actually more complex than that as it has templated classes that inherit from that base so that there is a different template for each expected struct that gets returned from various async requests.  I'm trying to get the simplest possible scenario working first and hoping I'm not missing anything, but I don't actually understand how I'm supposed to work with the C++ objects enough really to be sure.  Here's how I've defined it in D, as per https://dlang.org/spec/cpp_interface.html#classes :

alias ulong SteamAPICall_t;
extern(C) {
	void SteamAPI_RegisterCallResult(CCallbackBase pCallback, SteamAPICall_t hAPICall);
	void SteamAPI_UnregisterCallResult(CCallbackBase pCallback, SteamAPICall_t hAPICall);
}

extern(C++) {
	interface CCallbackBase {
		//this() { m_nCallbackFlags = 0; m_iCallback = 0; }
		void Run( void *pvParam );
		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall );
		int GetICallback();
		int GetCallbackSizeBytes();

		enum { k_ECallbackFlagsRegistered = 0x01, k_ECallbackFlagsGameServer = 0x02 }
		//uint8 m_nCallbackFlags;
		//int m_iCallback;
		//friend class CCallbackMgr;

		//CCallbackBase( const CCallbackBase& );
		//CCallbackBase& operator=( const CCallbackBase& );
	}
}

class CImpl : CCallbackBase {
	extern(C++) {
		this() { m_nCallbackFlags = 0; m_iCallback = 0; }
		void Run( void *pvParam ) { writeln("Run1"); }
		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall ) { writeln("Run2"); }
		int GetICallback() { return m_iCallback; }
		int GetCallbackSizeBytes() { return NumberOfCurrentPlayers_t.sizeof; } // ordinarily use templates to determine what type struct ptr to return
	}
	uint8 m_nCallbackFlags;
	int m_iCallback;
}


Which then gets called later on as:

auto cbk = new CImpl();
cbk.m_iCallback = NumberOfCurrentPlayers_t.k_iCallback;
auto hid = SteamAPI_ISteamUserStats_GetNumberOfCurrentPlayers(...);
SteamAPI_RegisterCallResult(cbk, hid);

And then every frame SteamAPI_RunCallbacks(); runs, which is supposed to fire off the callbacks.  I've also tried using abstract class instead of interface.  My expectation of what's supposed to happen is that one of those Run() functions gets called at some point.  However, it never does (even though I can poll to determine the call has actually completed).  I'm hoping the problem is I'm simply not passing the right expected type of data to the registration function and/or haven't defined the interface in D properly (I admit I have very little idea how the C++ is working anymore, including what those private methods are for), and not an issue with utilizing the API itself, since I doubt there's anyone I could ask on the Steam end of things about interfacing with D.

My gut feeling is I'm doing something very incorrect/stupid on the definition end of things, I understand passing function pointers to C++ functions that intend to call back well enough but I'm confused about this concept of passing "class SomeClass *" to a function and expecting it to call a method on that class, I don't know where everything is supposed to "exist".  If anyone has any insight to provide it would be greatly appreciated, thanks!

On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:

>
> class CImpl : CCallbackBase {
> 	extern(C++) {

> If anyone has any insight to provide it would be greatly appreciated, thanks!

I've not used any of the C++ interfacing features yet, but my understanding is the extern(C++) has to apply to the class declaration itself, not only the member functions:

extern(C++)
class CImpl : CCallbackBase {

Does that make a difference?

On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:
>
> I've defined it in D, as per https://dlang.org/spec/cpp_interface.html#classes :
>

change this to class, or even abstract class as shown in example

> extern(C++) {
> 	interface CCallbackBase {
> 		//this() { m_nCallbackFlags = 0; m_iCallback = 0; }
> 		void Run( void *pvParam );
> 		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall );
> 		int GetICallback();
> 		int GetCallbackSizeBytes();
>
> 		enum { k_ECallbackFlagsRegistered = 0x01, k_ECallbackFlagsGameServer = 0x02 }
> 		//uint8 m_nCallbackFlags;
> 		//int m_iCallback;
> 		//friend class CCallbackMgr;
>
> 		//CCallbackBase( const CCallbackBase& );
> 		//CCallbackBase& operator=( const CCallbackBase& );
> 	}
> }

add extern(C++) to class as well to

> class CImpl : CCallbackBase {
> 	extern(C++) {
> 		this() { m_nCallbackFlags = 0; m_iCallback = 0; }
> 		void Run( void *pvParam ) { writeln("Run1"); }
> 		void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall ) { writeln("Run2"); }
> 		int GetICallback() { return m_iCallback; }
> 		int GetCallbackSizeBytes() { return NumberOfCurrentPlayers_t.sizeof; } // ordinarily use templates to determine what type struct ptr to return
> 	}
> 	uint8 m_nCallbackFlags;
> 	int m_iCallback;
> }
>

you also may or may not need to mark non-virtual C++ methods as final.

Of course I haven't used D for quite some time so I can be mistaken.

But the lesson I learned the hard way is that in D for extern(C++) you don't use interface(it simply has no mapping to C++ types?), and for defining COM-interfaces use interface or it will bite you.
I hope it helps.

On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:
> On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:
>
>>
>> class CImpl : CCallbackBase {
>> 	extern(C++) {
>
>> If anyone has any insight to provide it would be greatly appreciated, thanks!
>
> I've not used any of the C++ interfacing features yet, but my understanding is the extern(C++) has to apply to the class declaration itself, not only the member functions:
>
> extern(C++)
> class CImpl : CCallbackBase {
>
> Does that make a difference?

Tried wrapping the CImpl class in extern(C++), no luck.. tried final keyword on the methods too.  Thanks though.

On Friday, 8 June 2018 at 06:59:51 UTC, cc wrote:
> On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:
>> On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:
>>
>>>
>>> class CImpl : CCallbackBase {
>>> 	extern(C++) {
>>
>>> If anyone has any insight to provide it would be greatly appreciated, thanks!
>>
>> I've not used any of the C++ interfacing features yet, but my understanding is the extern(C++) has to apply to the class declaration itself, not only the member functions:
>>
>> extern(C++)
>> class CImpl : CCallbackBase {
>>
>> Does that make a difference?
>
> Tried wrapping the CImpl class in extern(C++), no luck.. tried final keyword on the methods too.  Thanks though.

Can you upload reduced example somewhere to play with?

On Friday, 8 June 2018 at 07:32:54 UTC, evilrat wrote:
> On Friday, 8 June 2018 at 06:59:51 UTC, cc wrote:
>> On Friday, 8 June 2018 at 02:52:10 UTC, Mike Parker wrote:
>>> On Friday, 8 June 2018 at 00:55:35 UTC, cc wrote:
>>>
>>>>
>>>> class CImpl : CCallbackBase {
>>>> 	extern(C++) {
>>>
>>>> If anyone has any insight to provide it would be greatly appreciated, thanks!
>>>
>>> I've not used any of the C++ interfacing features yet, but my understanding is the extern(C++) has to apply to the class declaration itself, not only the member functions:
>>>
>>> extern(C++)
>>> class CImpl : CCallbackBase {
>>>
>>> Does that make a difference?
>>
>> Tried wrapping the CImpl class in extern(C++), no luck.. tried final keyword on the methods too.  Thanks though.
>
> Can you upload reduced example somewhere to play with?

I've put together a simplified test program here (124KB):
https://drive.google.com/uc?id=1VPxCcPShlYpGo8BggxniNlPH96iIRG8_&export=download

The official SDK can be downloaded here (including the steam_api.dll if you don't want to use the one in the zip for security):
https://partner.steamgames.com/doc/sdk
Although it may require a developer account to download.

Also I should note that Steam differentiates between "callbacks" (events that may be fired to the application that registers to listen for them at any time or multiple times) and "call results" (1:1 asynchronous requests, I believe).  This test so far is using call results.

On Saturday, 9 June 2018 at 03:07:39 UTC, cc wrote:
> I've put together a simplified test program here (124KB):

Here is a pastebin of the D source file updated with some additional comments at the end with the callback class definitions from the original header files
https://pastebin.com/M8hDXt6L

Sample output:

Initializing.
User logged on: true
Starting request.
hid: 4838393704146785693
..!!!!!!!!
Request completed: NumberOfCurrentPlayers_t(1, 5828)
Terminating.


Not present: any indication that the registered callresult was executed.

On Saturday, 9 June 2018 at 03:14:13 UTC, cc wrote:
> On Saturday, 9 June 2018 at 03:07:39 UTC, cc wrote:
>> I've put together a simplified test program here (124KB):
>
> Here is a pastebin of the D source file updated with some additional comments at the end with the callback class definitions from the original header files
> https://pastebin.com/M8hDXt6L

The proper bare minimum signature for call result callback is
------------------
extern(C++) abstract class CCallbackBase {
  abstract void Run( void *pvParam ) { writeln("Run()"); }
  abstract void Run( void *pvParam, bool bIOFailure, SteamAPICall_t hSteamAPICall );
  final int GetICallback() { return m_iCallback; } // this is actually optional
  abstract int GetCallbackSizeBytes();
 protected:
  uint8 m_nCallbackFlags; // probably can be left for some specific use cases
  int m_iCallback;
}
------------------
and the rest templates and stuff is purely for C++ convenience. Run(void*, bool, SteamAPICall_t) is what actually called for call result, then CallResult<> template overrides it to call appropriate pointer member in class with void (T::fun)(Data*, bool bIOFailure) signature.  You can mimic it on D side, or just ignore and make own.

There is one catch however, for it to be called flags should be equal 1, and iCallback must match struct enum, like NumberOfCurrentPlayers_t.k_iCallback for GetNumberOfCurrentPlayers()


However steam devs decided to shield actual pointer and return pointer sized integer when C API is used(or they just screw up?). Anyway, the pointers for subsystems returned by context calls on C++ API and mirrored C API calls are different, but they also have some mechanism for filtering this stuff, that way both integer handle and pointer calls the same underlying implementation, but C API call again is shielded so setting up CallResult and CCallback are ignored.


So my solution was just to make simple wrapper around C++ context calls and pass that real pointer to D side.

------------------- C++ code
// build as static library & link with your project
#ifdef _WIN32
// sorry, I just hardcoded it...
#pragma comment (lib, "steam_api64.lib")
#define _CRT_SECURE_NO_WARNINGS 1
#endif
  #include "public/steam/steam_api.h"
#ifdef _WIN32
#define _CRT_SECURE_NO_WARNINGS 1
#endif

extern "C" void* D_GetSteamUserStats()
{
  return SteamUserStats();
}

------------------- D code
// yes, I messed up with the signature to avoid casting, this may or may not be dangerous
extern(C) ISteamUserStats D_GetSteamUserStats();


This of course involves making C++ wrapper, but I don't really see other ways because they inlined everything ...

On Saturday, 9 June 2018 at 14:11:13 UTC, evilrat wrote:
> However steam devs decided to shield actual pointer and return pointer sized integer when C API is used(or they just screw up?). Anyway, the pointers for subsystems returned by context calls on C++ API and mirrored C API calls are different, but they also have some mechanism for filtering this stuff, that way both integer handle and pointer calls the same underlying implementation, but C API call again is shielded so setting up CallResult and CCallback are ignored.
>
>
> So my solution was just to make simple wrapper around C++ context calls and pass that real pointer to D side.

I see, thank you for checking it out.  Is it only the functions that return the interface pointers e.g. ISteamUserStats that need to be wrapped to use the class-based versions instead of the steam_api_flat versions?  Or does the entire callback system need to be handled through the wrapper?