Some DirectX methods return structs by value, but when I try calling them I either get garbage or an access violation.

Usually COM methods return structs by pointer as a parameter, but these are returning the struct as the actual return value, as in this definition:

  extern(Windows):
  struct D2D1_SIZE_F { float width, height; }

  interface ID2D1Bitmap : ID2D1Image {
    D2D1_SIZE_F GetSize();
  }

If I rewrite GetSize to return by pointer as a parameter, it appears to work and I get the correct width and height without an AV being thrown. And I can add a helper method that returns by value:

  interface ID2D1Bitmap : ID2D1Image {
    void GetSize(D2D1_SIZE_F* size);

    final D2D1_SIZE_F GetSize() {
      D2D1_SIZE_F size;
      GetSize(&size);
      return size;
    }
  }

But does anyone know why the original definition works in C++ but not D? Is it a bug? (I'm compiling with -m64.)

On Saturday, 3 December 2016 at 09:51:00 UTC, John C wrote:
> Some DirectX methods return structs by value, but when I try calling them I either get garbage or an access violation.
>
> [...]

I know for ldc that function that return struct by value actually return by a hidden return parameter pointer.

On Saturday, 3 December 2016 at 09:51:00 UTC, John C wrote:
> Some DirectX methods return structs by value, but when I try calling them I either get garbage or an access violation.
>
> Usually COM methods return structs by pointer as a parameter, but these are returning the struct as the actual return value, as in this definition:
>
>   extern(Windows):
>   struct D2D1_SIZE_F { float width, height; }
>
>   interface ID2D1Bitmap : ID2D1Image {
>     D2D1_SIZE_F GetSize();
>   }
>
> If I rewrite GetSize to return by pointer as a parameter, it appears to work and I get the correct width and height without an AV being thrown. And I can add a helper method that returns by value:
>
>   interface ID2D1Bitmap : ID2D1Image {
>     void GetSize(D2D1_SIZE_F* size);
>
>     final D2D1_SIZE_F GetSize() {
>       D2D1_SIZE_F size;
>       GetSize(&size);
>       return size;
>     }
>   }
>
> But does anyone know why the original definition works in C++ but not D? Is it a bug? (I'm compiling with -m64.)

most likely due to this
https://issues.dlang.org/show_bug.cgi?id=16527

and that is annoying. i tried ldc2 1.1.beta6 and dmd 2.072.1 both x86 and x64 and it crashes anyways, sooner or later.

btw if you make call with return pointer instead of value it can probably messed up your stack already and crash any moment later.

in my DirectX bindings i used to return ref struct(though i think this is dirty hack), and it was working until recent.

On Saturday, 3 December 2016 at 09:51:00 UTC, John C wrote:
> Some DirectX methods return structs by value, but when I try calling them I either get garbage or an access violation.
>
> Usually COM methods return structs by pointer as a parameter, but these are returning the struct as the actual return value, as in this definition:
>
>   extern(Windows):
>   struct D2D1_SIZE_F { float width, height; }
>
>   interface ID2D1Bitmap : ID2D1Image {
>     D2D1_SIZE_F GetSize();
>   }
>
> If I rewrite GetSize to return by pointer as a parameter, it appears to work and I get the correct width and height without an AV being thrown. And I can add a helper method that returns by value:
>
>   interface ID2D1Bitmap : ID2D1Image {
>     void GetSize(D2D1_SIZE_F* size);
>
>     final D2D1_SIZE_F GetSize() {
>       D2D1_SIZE_F size;
>       GetSize(&size);
>       return size;
>     }
>   }
>
> But does anyone know why the original definition works in C++ but not D? Is it a bug? (I'm compiling with -m64.)

That's rather interesting. The COM function really seems to return the struct directly, not returning an HRESULT and setting some output pointee as most COM functions I've seen so far. According to the Win64 ABI, the returned D2D1_SIZE_F struct should be returned in RAX, as the 2 floats are <= 64 bit. But your workaround seems to suggest it's using sret (struct-return via hidden pointer). To make sure this is the case, I'd suggest inspecting the C++ assembly for a trivial function. If so, we'll need to find out why the Win64 ABI isn't followed and whether COM has its own ABI.

> I know for ldc that function that return struct by value actually return by a hidden return parameter pointer.

Not always. In fact, it highly depends on the target ABI which structs are returned in registers and which ones via sret.

> https://issues.dlang.org/show_bug.cgi?id=16527

That is definitely a bug in DMD (swapping `this` and `sret` pointers) but doesn't apply to LDC, and is a separate issue.

On Monday, 19 December 2016 at 09:49:13 UTC, kinke wrote:
> On Saturday, 3 December 2016 at 09:51:00 UTC, John C wrote:
>> Some DirectX methods return structs by value, but when I try calling them I either get garbage or an access violation.
>>
>> Usually COM methods return structs by pointer as a parameter, but these are returning the struct as the actual return value, as in this definition:
>>
>>   extern(Windows):
>>   struct D2D1_SIZE_F { float width, height; }
>>
>>   interface ID2D1Bitmap : ID2D1Image {
>>     D2D1_SIZE_F GetSize();
>>   }
>>
>> If I rewrite GetSize to return by pointer as a parameter, it appears to work and I get the correct width and height without an AV being thrown. And I can add a helper method that returns by value:
>>
>>   interface ID2D1Bitmap : ID2D1Image {
>>     void GetSize(D2D1_SIZE_F* size);
>>
>>     final D2D1_SIZE_F GetSize() {
>>       D2D1_SIZE_F size;
>>       GetSize(&size);
>>       return size;
>>     }
>>   }
>>

yes it works indeed O_o

also align(8) for struct seems to work in x86 but has some issues with ESP(reserved 4 bytes for pointer, still has 4 bytes for second float?), crashes without align at later point.

x64 still has messed up 'this'(both dmd and ldc, not tested with gdc),
but with passing parameter instead of return it really keeps stack where it leaves it and even 'this' are in place so everything is seems to work.

On Monday, 19 December 2016 at 12:23:46 UTC, evilrat wrote:
> x64 still has messed up 'this'(both dmd and ldc, not tested with gdc)

Please file an LDC issue then, as I'm pretty sure it works for regular (non-COM) C++ classes.

On Monday, 19 December 2016 at 09:49:13 UTC, kinke wrote:
> That's rather interesting. The COM function really seems to return the struct directly, not returning an HRESULT and setting some output pointee as most COM functions I've seen so far. According to the Win64 ABI, the returned D2D1_SIZE_F struct should be returned in RAX, as the 2 floats are <= 64 bit. But your workaround seems to suggest it's using sret (struct-return via hidden pointer). To make sure this is the case, I'd suggest inspecting the C++ assembly for a trivial function. If so, we'll need to find out why the Win64 ABI isn't followed and whether COM has its own ABI.

I did some research myself and indeed, COM classes/interfaces are apparently subject to a separate ABI. Unfortunately, googling it hasn't turned up any official (and not even some inofficial) documentation so far. Based on the first few tests on Win64, integers are returned in RAX, floats (and I guess doubles too) in XMM0, and structs (incl. 2x int32 and 2x float) via hidden sret pointer, with `this` pointer in RCX (1st arg) and `sret` in RDX (2nd arg). Compared to the normal Win64 C++ ABI it just seems more conservative by always returning structs via hidden pointer.

https://issues.dlang.org/show_bug.cgi?id=16987
https://github.com/ldc-developers/ldc/issues/1935

On Monday, 19 December 2016 at 22:47:26 UTC, kinke wrote:
> On Monday, 19 December 2016 at 09:49:13 UTC, kinke wrote:
>> [...]
>
> I did some research myself and indeed, COM classes/interfaces are apparently subject to a separate ABI. Unfortunately, googling it hasn't turned up any official (and not even some inofficial) documentation so far. Based on the first few tests on Win64, integers are returned in RAX, floats (and I guess doubles too) in XMM0, and structs (incl. 2x int32 and 2x float) via hidden sret pointer, with `this` pointer in RCX (1st arg) and `sret` in RDX (2nd arg). Compared to the normal Win64 C++ ABI it just seems more conservative by always returning structs via hidden pointer.

Wow! That was quick. Such tests would take me few days.

I'm not sure how it may help but there is "specs" in MS .net core runtime, just search the repo for COM

like this https://github.com/dotnet/coreclr/blob/32f0f9721afb584b4a14d69135bea7ddc129f755/src/vm/amd64/GenericComCallStubs.asm#L177