May 24, 2021

On Monday, 24 May 2021 at 17:39:38 UTC, Gavin Ray wrote:

>

I'd be grateful for solid information on this

Here is a more informal report how it works in C++:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23.4735&rep=rep1&type=pdf

But in the end, I think a byte code analysis is needed to be sure. The best tools to visualize have already been proposed by you, so I think you are on a good path.

May 25, 2021

On Monday, 24 May 2021 at 20:31:18 UTC, sighoya wrote:

>

On Monday, 24 May 2021 at 17:39:38 UTC, Gavin Ray wrote:

>

Hence why I was asking how to make D structs/classes that have compatible or identical vtables to multiply inherited objects to pass as arguments to extern (C++) functions.

I think classes annotated with extern is your only high level guaranteed == type safe option to be compatible to other c++ classes.

But you seek for the general multiple inheritance case which seems not to be supported with extern, sadly.
So you stick with manual solutions like template metaprogramming or/and raw pointer fiddling.
Anyway, both solutions would require an unsafe cast in the end, so you are on your own.

The below seems to work at least, which is encouraging:

#include <cstdio>
#define DLL_EXPORT __declspec(dllexport)

class Base1 {
public:
    virtual void overrideMe1() = 0;
    const int getSomething() { return 42; }
};

class Base2 {
public:
    virtual void overrideMe2() = 0;
    const int getOtherThing() { return 99; }
};

class Derived : public Base1, public Base2 {
public:
    int someInt;
    Derived(int someIntVal) { this->someInt = someIntVal; }
    virtual void overrideMe1() override;
    virtual void overrideMe2() override;
};

DLL_EXPORT void takesADerived(Derived* derived)
{
    printf("[C++] Calling Derived::overrideMe1() \n");
    derived->overrideMe1();
    printf("[C++] Calling Derived::overrideMe2() \n");
    derived->overrideMe2();
    printf("[C++] Calling Derived::getSomething() = %d \n",
        derived->getSomething());
    printf("[C++] Calling Derived::getOtherThing() = %d \n",
        derived->getOtherThing());
    printf("[C++] Derived::someInt = %d \n", derived->someInt);
}
import core.stdc.stdio : printf;

extern (C++)
{
  void takesADerived(Derived derived);

  interface Base1 { void overrideMe1(); }
  interface Base2 { void overrideMe2(); }
  class Derived : Base1, Base2
  {
    int someInt;
    this(int someIntVal) { this.someInt = someIntVal; }
    void overrideMe1() { printf("[D] Dlang Derived overrideMe1 called \n"); }
    void overrideMe2() { printf("[D] Dlang Derived overrideMe1 called \n"); }
  }
}

void main()
{
  auto dlangDerived = new Derived(123);
  printf("[D] Calling C++ takesADerived() with D Derived* \n");
  takesADerived(dlangDerived);
}

example

Unfortunately, it does not work if I try to add final int getSomething() or the other one to the D interfaces, it throws a symbol error because the mangled names are slightly different:


unresolved external symbol "public: int __cdecl Base1::getSomething(void)"
                  (?getSomething@Base1@@QEAAHXZ)
0000000000000000 T ?getSomething@Base1@@QEAA?BHXZ # < "nm" output

If I use nm and list the symbols, and then try to manually use the mangling scheme, it almost works but because the return types differ it won't compile =/

extern class Derived : Base1, Base2
{
  int someInt;

  pragma(mangle, "?getOtherThing@Base2@@QEAA?BHXZ")
  int getOtherThing();
}
main.d(29): Error: Function type does not match previously declared function with the same mangled name: `?getOtherThing@Base2@@QEAA?BHXZ`
main.d(29):        Previous IR type: i32 (%main.Base2*)
main.d(29):        New IR type:      i32 (%main.Derived*)
May 25, 2021

On Tuesday, 25 May 2021 at 02:47:19 UTC, Gavin Ray wrote:

>

Unfortunately, it does not work if I try to add final int getSomething() or the other one to the D interfaces, it throws a symbol error because the mangled names are slightly different:


unresolved external symbol "public: int __cdecl Base1::getSomething(void)"
                  (?getSomething@Base1@@QEAAHXZ)
0000000000000000 T ?getSomething@Base1@@QEAA?BHXZ # < "nm" output

If I use nm and list the symbols, and then try to manually use the mangling scheme, it almost works but because the return types differ it won't compile =/

extern class Derived : Base1, Base2
{
  int someInt;

  pragma(mangle, "?getOtherThing@Base2@@QEAA?BHXZ")
  int getOtherThing();
}
main.d(29): Error: Function type does not match previously declared function with the same mangled name: `?getOtherThing@Base2@@QEAA?BHXZ`
main.d(29):        Previous IR type: i32 (%main.Base2*)
main.d(29):        New IR type:      i32 (%main.Derived*)

That's just LDC thing, should work with DMD.
Are you sure getOtherThing marked final? Because in your C++ class it is not virtual and in your example it is not final as well.
In C++ having multiple bases with final method means that both Base1 and Base2 have their own instances of that method.

Anyway in order to call it you'll have to cast manually to target base.
i.e. if you want to call Base1::getOtherThing and you have Derived you'll have to cast to Base1.

Derived d;
// d.getOtherThing(); // likely won't link
(cast(Base1)d) .getOtherThing(); // will call Base1 variant
(cast(Base2)d) .getOtherThing(); // will call Base2 variant

It is also possible that you'll have to put them in using mixin template so it will create scope for them that doesn't collide with each other, though I think it's more of a hack than a feature.

Something like

class Derived
{
   mixin base1; // has final getOtherThing
   mixin base2; // has final getOtherThing
}

then you can call it almost like in C++

d.base1.getOtherThing();
d.base2.getOtherThing();

Anyway all this stuff requires thorough research & testing as such ABI tinkering is very easy to mess up and very hard to debug, for example if you mess up the order(functions layout) it can land on another final method call that seemingly work but in debugger you'll see that you can't hit breakpoint in that method.
Happy debugging time!

May 25, 2021

On Tuesday, 25 May 2021 at 02:47:19 UTC, Gavin Ray wrote:

>

The below seems to work at least, which is encouraging:

Awesome!
At least, it becomes problematic with fields in base classes, it would be nice if we could map them to @property annotated functions in D interfaces.

May 25, 2021

On Tuesday, 25 May 2021 at 08:10:25 UTC, sighoya wrote:

>

On Tuesday, 25 May 2021 at 02:47:19 UTC, Gavin Ray wrote:

>

The below seems to work at least, which is encouraging:

Awesome!
At least, it becomes problematic with fields in base classes, it would be nice if we could map them to @property annotated functions in D interfaces.

I did some basic testing with code above, it seems class layout is recursively linear at least on Windows, and D follows C++ rules close enough, at least it works if one comments out all but the first base, and the rest can be hand crafted using structs and templates.

With this code it is possible to pass back C++ class received from C++ code and call it.
So it seems support for multiple inheritance shouldn't be that hard to implement (for Windows), no idea how it works on Linux though.


// replacement for second base
template iBase2() {
    // final is needed or it will call overrideMe1 instead of this
    final void overrideMe2() {
        static struct _layout {
            void* b1;
            void* b2;
            int someInt;
        }

        _layout* base = cast(_layout*) cast(void*) this;
        alias fn = extern(C++) void function(void*);

        fn _fn = *cast(fn*) base.b2; // vtable for Base2, vtable[0] is overrideMe2
        _fn(cast(void*)this);
    }
}

class Derived : Base1 //, Base2 {
    mixin iBase2 __base2;
    alias overrideMe2 = __base2.overrideMe2;


    static assert(Derived.someInt.offsetof == 16); // that's important otherwise D ctor will mess up everything
}

Maybe I'll be able to create templated base class that does all this stuff with mixins and templates.

like this

extern(C++)
abstract class MultipleInheritance(T...)
{
  // implementation //
}

class Derived : MultipleInheritance!(Base1, Base2)
{ ... }
May 25, 2021

On Tuesday, 25 May 2021 at 06:02:55 UTC, evilrat wrote:

>

Anyway all this stuff requires thorough research & testing as such ABI tinkering is very easy to mess up and very hard to debug, for example if you mess up the order(functions layout) it can land on another final method call that seemingly work but in debugger you'll see that you can't hit breakpoint in that method.
Happy debugging time!

Ahh, that works!

It also works without the cast(BaseType) if you use alias in the class:

import core.stdc.stdio : printf;
extern (C++)
{
  void takesADerived(Derived derived);
  Derived createTestDerivedCPP();

  extern interface Base1
  {
    void overrideMe1();

    pragma(mangle, "?getSomething@Base1@@QEAA?BHXZ")
    final int getSomething();

    pragma(mangle, "?inheritedFunctionWithArgs@Base1@@QEAAHHH@Z")
    final int inheritedFunctionWithArgs(int x, int y);
  }

  extern interface Base2
  {
    void overrideMe2();

    pragma(mangle "?getOtherThing@Base2@@QEAA?BHXZ")
    final int getOtherThing();
  }

  extern class Derived : Base1, Base2
  {
    int someInt;

    alias getSomething = Base1.getSomething;
    alias inheritedFunctionWithArgs = Base1.inheritedFunctionWithArgs;

    this(int someIntVal) { this.someInt = someIntVal; }

    void overrideMe1() { printf("[D] Dlang Derived overrideMe1 called \n"); }
    void overrideMe2() { printf("[D] Dlang Derived overrideMe1 called \n"); }
  }
}

void main()
{
  Derived dlangDerived = new Derived(123);
  printf("[D] Derived.Base1::getSomething() = %d \n", dlangDerived.getSomething());
  printf("[D] Derived.Base2::getOtherThing() = %d \n", dlangDerived.getOtherThing());
  printf("[D] Derived.Base1::inheritedFunctionWithArgs(5, 10) = %d \n",
      dlangDerived.inheritedFunctionWithArgs(5, 10));
  printf("[D] Calling C++ takesADerived() with D Derived* \n");
  takesADerived(dlangDerived);
}

output

Just you wait, I'll learn enough to be of any use contributing to OMG yet! haha

May 25, 2021

On Tuesday, 25 May 2021 at 17:52:14 UTC, Gavin Ray wrote:

>
void main()
{
  Derived dlangDerived = new Derived(123);
  printf("[D] Derived.Base1::getSomething() = %d \n", dlangDerived.getSomething());
  printf("[D] Derived.Base2::getOtherThing() = %d \n", dlangDerived.getOtherThing());
  printf("[D] Derived.Base1::inheritedFunctionWithArgs(5, 10) = %d \n",
      dlangDerived.inheritedFunctionWithArgs(5, 10));
  printf("[D] Calling C++ takesADerived() with D Derived* \n");
  takesADerived(dlangDerived);
}

output

That last one with someInt is what I warned about. D ctor messed up class layout, in this simple case where class data isn't used it almost works, but will be practically unusable in real scenarios.

May 25, 2021

On Tuesday, 25 May 2021 at 11:38:03 UTC, evilrat wrote:

>

I did some basic testing with code above, it seems class layout is recursively linear at least on Windows, and D follows C++ rules close enough, at least it works if one comments out all but the first base, and the rest can be hand crafted using structs and templates.

With this code it is possible to pass back C++ class received from C++ code and call it.
So it seems support for multiple inheritance shouldn't be that hard to implement (for Windows), no idea how it works on Linux though.

Maybe I'll be able to create templated base class that does all this stuff with mixins and templates.

like this

extern(C++)
abstract class MultipleInheritance(T...)
{
  // implementation //
}

class Derived : MultipleInheritance!(Base1, Base2)
{ ... }

Oh this is brilliant! Absolutely incredible work + findings!

Is this conceptually similar at all to this bit of code that adr had given to me?

// Same thing repeated for Abstract2
interface Abstract1 {
    /* virtual */ void overrideMe1();
    void sayHello();
    mixin template Abstract1Impl() {
        void sayHello() { import std.stdio; writeln("Hello"); }
    }
}

class MyClass : Abstract1, Abstract2 {
  mixin Abstract1Impl; mixin Abstract2Impl;
  override void overrideMe1() { writeln("overrode 1"); }
  override void overrideMe2() { writeln("overrode 2"); }
}

void main() {
    auto it = new MyClass();
    it.sayHello(); it.sayGoodbye();
    it.overrideMe1(); it.overrideMe2();
}

Also, I wonder if there's a good way to build a quick CLI visualization + debugging tool for this using D's .vtble[] method. I would find it really useful for learning =D

There are also some automatic ABI compliance checker tools/libraries, which could be good for programmatically verifying that the class produced by D is valid with the ABI of a method/class from a C++ library file.

(I know for example, that Qt does this with their Python codegen tool, Shiboken, for auto-testing the generated code)

>

... libabigail provides facilities to perform deep comparisons of two ABIs. That is, it can compare the types of two sets of functions or variables and represents the result in a way that allows it to emit textual reports about the differences.

>

This allows us to write tools like abidiff that can compare the ABI of two shared libraries and represent the result in a meaningful enough way to help us spot ABI incompatibilities. There are several other tools that are built using the Abigail framwork.

$ g++ -g -Wall -shared -o libtest-v0.so test-v0.cc
$ g++ -g -Wall -shared -o libtest-v1.so test-v1.cc
$
$ ../build/tools/abidiff libtest-v0.so libtest-v1.so
Functions changes summary: 0 Removed, 1 Changed, 0 Added function
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

1 function with some indirect sub-type change:

  [C]'function void foo(S0&)' has some indirect sub-type changes:
        parameter 0 of type 'S0&' has sub-type changes:
          in referenced type 'struct S0':
            size changed from 32 to 64 bits
            1 data member insertion:
              'char S0::inserted_member', at offset 0 (in bits)
            1 data member change:
             'int S0::m0' offset changed from 0 to 32
May 25, 2021

On Tuesday, 25 May 2021 at 18:03:00 UTC, evilrat wrote:

>

That last one with someInt is what I warned about. D ctor messed up class layout, in this simple case where class data isn't used it almost works, but will be practically unusable in real scenarios.

Ah =/
You have this in your code example:

static assert(Derived.someInt.offsetof == 16); // that's important otherwise D ctor will mess up everything

Would this fix it, or is it just not super viable to hack around C++ multiple inheritance in D?

Maybe generating some wrapper C++ code to be linked could help too?
I'm not sure though/don't really know enough to say

May 25, 2021

On Tuesday, 25 May 2021 at 18:12:27 UTC, Gavin Ray wrote:

>

Would this fix it, or is it just not super viable to hack around C++ multiple inheritance in D?

You can do anything you want with structs, raw memory, and casting, so it is viable, if you have a strong interest for this.

But if you are not a low level programmer you might find it tedious.