Hi everyone,

I have the pleasure to announce to you all the existence of a modified LDC able to interface directly to C++ libraries, wiping out the need to write bindings:

 https://github.com/Syniurge/Calypso

It's at a prototype stage, but its C++ support is pretty wide already:

 • Global variables
 • Functions
 • Structs
 • Unions (symbol only)
 • Enums
 • Typedefs
 • C++ class creation with the correct calls to ctors (destruction is disabled for now)
 • Virtual function calls
 • Static casts between C++ base and derived classes (incl. multiple inheritance offsets)
 • Mapping template implicit and explicit specializations already in the PCH to DMD ones, no new specialization on the D side yet
 • D classes inheriting from C++ ones, including the correct vtable generation for the C++ part of the class

So what is this sorcery? Let's remind ourselves that this isn't supposed to be feasible:

>Being 100% compatible with C++ means more or less adding a fully functional C++ compiler front end to D. Anecdotal evidence suggests that writing such is a minimum of a 10 man-year project, essentially making a D compiler with such capability unimplementable.
http://dlang.org/cpp_interface.html

Well.. it is :D
Calypso introduces the modmap keyword, as in:

  modmap (C++) "cppheader.h";

to generate with the help of Clang libraries a "virtual" tree of C++ modules. Then after making Clang generate a PCH for all the headers, the PCH is loaded and classes, structs, enums are placed inside modules named after them, while global variables and functions are in a special module named "_". For example:

  import (C++) Namespace.SomeClass;  // imports Namespace::SomeClass
  import (C++) Namespace._;  // imports all the global variables and functions in Namespace
  import (C++) _ : myCfunc, myGlobalVar;  // importing the global namespace = bad idea, but selective imports work

Being a prototype, I didn't really pay attention to code conventions or elegance and instead focused on getting things working.
And being tied to LDC and Clang (I have no idea how feasible a GCC version would be), it's going to stay like this for some time until I get feedback from the contributors on how this all should really be implemented,. For example Calypso introduces "language plugins", to minimize the amount of code specific to C++ and to make support of foreign languages cleaner and less intrusive, although it of course needs numerous hooks here and there in DMD and LDC.

Calypso is still WIP, but it's in pretty good shape and already works in a lot of test cases (see tests/calypso/), and is almost ready to use for C++ libraries at least. Since C libraries are in the global namespace, it's not a convenient replacement yet for bindings until I implement the Clang module map format. More info this blog post detailing some of the history behind Calypso:

http://syniurge.blogspot.com/2013/08/calypso-to-mars-first-contact.html

So.. Merry Christmas dear D community? :)


My take on the current talks of "feature freezing" D: the strength of D is its sophistication. The core reason why D fails to attract more users isn't the frequent compiler bugs or regressions, but the huge amount of time needed to get something done because neither equivalent nor bindings exist for most big and widespread C++ libraries like Qt. All these talks about making D a more minimalist language won't solve much and will only result in holding back D, which has the potential to become a superset of all the good in other system languages, as well as bringing its own powerful unique features such as metaprogramming done right.
By removing the main reason why D wasn't a practical choice, this will hopefully unlock the situation and make D gain momentum as well as attract more contributors to the compilers to fix bugs and regressions before releases.

On 23/12/2014 12:14 p.m., Elie Morisse wrote:
> Hi everyone,
>
> I have the pleasure to announce to you all the existence of a modified
> LDC able to interface directly to C++ libraries, wiping out the need to
> write bindings:
>
>   https://github.com/Syniurge/Calypso
>
> It's at a prototype stage, but its C++ support is pretty wide already:
>
>   • Global variables
>   • Functions
>   • Structs
>   • Unions (symbol only)
>   • Enums
>   • Typedefs
>   • C++ class creation with the correct calls to ctors (destruction is
> disabled for now)
>   • Virtual function calls
>   • Static casts between C++ base and derived classes (incl. multiple
> inheritance offsets)
>   • Mapping template implicit and explicit specializations already in
> the PCH to DMD ones, no new specialization on the D side yet
>   • D classes inheriting from C++ ones, including the correct vtable
> generation for the C++ part of the class
>
> So what is this sorcery? Let's remind ourselves that this isn't supposed
> to be feasible:
>
>> Being 100% compatible with C++ means more or less adding a fully
>> functional C++ compiler front end to D. Anecdotal evidence suggests
>> that writing such is a minimum of a 10 man-year project, essentially
>> making a D compiler with such capability unimplementable.
> http://dlang.org/cpp_interface.html
>
> Well.. it is :D
> Calypso introduces the modmap keyword, as in:
>
>    modmap (C++) "cppheader.h";

That really should be a pragma.
pragma(modmap, C++, "cppheader.h");
Since pragma's are the way to instruct the compiler to do something.

> to generate with the help of Clang libraries a "virtual" tree of C++
> modules. Then after making Clang generate a PCH for all the headers, the
> PCH is loaded and classes, structs, enums are placed inside modules
> named after them, while global variables and functions are in a special
> module named "_". For example:
>
>    import (C++) Namespace.SomeClass;  // imports Namespace::SomeClass
>    import (C++) Namespace._;  // imports all the global variables and
> functions in Namespace
>    import (C++) _ : myCfunc, myGlobalVar;  // importing the global
> namespace = bad idea, but selective imports work
>
> Being a prototype, I didn't really pay attention to code conventions or
> elegance and instead focused on getting things working.
> And being tied to LDC and Clang (I have no idea how feasible a GCC
> version would be), it's going to stay like this for some time until I
> get feedback from the contributors on how this all should really be
> implemented,. For example Calypso introduces "language plugins", to
> minimize the amount of code specific to C++ and to make support of
> foreign languages cleaner and less intrusive, although it of course
> needs numerous hooks here and there in DMD and LDC.
>
> Calypso is still WIP, but it's in pretty good shape and already works in
> a lot of test cases (see tests/calypso/), and is almost ready to use for
> C++ libraries at least. Since C libraries are in the global namespace,
> it's not a convenient replacement yet for bindings until I implement the
> Clang module map format. More info this blog post detailing some of the
> history behind Calypso:
>
> http://syniurge.blogspot.com/2013/08/calypso-to-mars-first-contact.html
>
> So.. Merry Christmas dear D community? :)
>
>
> My take on the current talks of "feature freezing" D: the strength of D
> is its sophistication. The core reason why D fails to attract more users
> isn't the frequent compiler bugs or regressions, but the huge amount of
> time needed to get something done because neither equivalent nor
> bindings exist for most big and widespread C++ libraries like Qt. All
> these talks about making D a more minimalist language won't solve much
> and will only result in holding back D, which has the potential to
> become a superset of all the good in other system languages, as well as
> bringing its own powerful unique features such as metaprogramming done
> right.
> By removing the main reason why D wasn't a practical choice, this will
> hopefully unlock the situation and make D gain momentum as well as
> attract more contributors to the compilers to fix bugs and regressions
> before releases.

Will you be upstreaming this? Or maintaining this completely yourself?

On Tuesday, 23 December 2014 at 00:01:30 UTC, Rikki Cattermole wrote:
> Will you be upstreaming this? Or maintaining this completely yourself?

The ultimate goal is upstream, but first I need to agree with the main DMD and LDC contributors about how this should really be done. I.e atm the Calypso code coexists with the vanilla C++ support which has a different coding philosophy and is more intertwined with the rest of the code.

So I expect that I'll have to maintain it myself quite some time before this happens. And of course I'll make Calypso catch up with upstream LDC frequently.

On Monday, 22 December 2014 at 23:14:44 UTC, Elie Morisse wrote:
>>Being 100% compatible with C++ means more or less adding a fully functional C++ compiler front end to D. Anecdotal evidence suggests that writing such is a minimum of a 10 man-year project, essentially making a D compiler with such capability unimplementable.
> http://dlang.org/cpp_interface.html
>
> Well.. it is :D

Well, technically speaking you DO include fully functional C++ compiler front end with much more than 10 man-years of time invested - it just happens that it is already available and called clang :)

Project itself is very cool but I am in doubts about possibility of merging this upstream. Doing so would make full D implementation effectively impossible without some C++ compiler already available as a library on same platform - quite a restriction!

I think it is better suited as LDC extension and would discourage its usage in public open-source projects sticking to old way of C binding generation instead. For more in-house projects it looks like an absolute killer and exactly the thing Facebook guys wanted :)

On Tuesday, 23 December 2014 at 07:18:01 UTC, Dicebot wrote:
> Project itself is very cool but I am in doubts about possibility of merging this upstream. Doing so would make full D implementation effectively impossible without some C++ compiler already available as a library on same platform - quite a restriction!

Just to be clear, you're talking about needing the C++ compiler-as-library on the platform where the compiler is being run, right?

I don't see how that prevents upstreaming, so long as it's made clear that it's platform-dependent. Are there any particular platforms you are concerned about?

> I think it is better suited as LDC extension and would discourage its usage in public open-source projects sticking to old way of C binding generation instead. For more in-house projects it looks like an absolute killer and exactly the thing Facebook guys wanted :)

Agree that it's best suited as an extension, but that doesn't prevent it being an official part of the LDC codebase. The most problematic aspect is that it divides the compilers in terms of features supported. As you suggest, for public  projects, it's not nice to bind people to a single compiler.

On 2014-12-23 00:14, Elie Morisse wrote:
> Hi everyone,
>
> I have the pleasure to announce to you all the existence of a modified
> LDC able to interface directly to C++ libraries, wiping out the need to
> write bindings:

This is pretty cool, but I'm not so sure if this will be accepted in upstream.

I did something similar myself [1] with the help of DStep [2]. I basically linked DStep with DMD which called DStep to generate a temporary D file with the bindings. DMD would then read the temporary file as if it was there before starting the compilation.

[1] http://forum.dlang.org/thread/ks3kir$1ubq$1@digitalmars.com
[2] https://github.com/jacob-carlborg/dstep

-- 
/Jacob Carlborg

On Tuesday, 23 December 2014 at 10:52:58 UTC, Joseph Rushton Wakeling wrote:
> On Tuesday, 23 December 2014 at 07:18:01 UTC, Dicebot wrote:
> Just to be clear, you're talking about needing the C++ compiler-as-library on the platform where the compiler is being run, right?
>
> I don't see how that prevents upstreaming, so long as it's made clear that it's platform-dependent. Are there any particular platforms you are concerned about?

Consider both things like embedded/MIPS and Windows64 - LLVM tooling is not as strong on those right now, GCC does not provide such easy way to reuse C++ frontend and with DMD/MSVC it is simply beyond feasibility.

By upstream I don't mean LDC upstream but D upstream - we don't want it to become part of full language spec if implementation is so reliable on specific toolchain. It has been already discussed when Jacob proposed dstep integration AFAIR.

On Tuesday, 23 December 2014 at 11:53:38 UTC, Dicebot wrote:
> 
> By upstream I don't mean LDC upstream but D upstream - we don't want it to become part of full language spec if implementation is so reliable on specific toolchain. It has been already discussed when Jacob proposed dstep integration AFAIR.

Actually it's meant to be optional and Calypso should ultimately go in a separate shared library.

As I implemented it I added this virtual function to Dsymbol:

   virtual LangPlugin *langPlugin() { return NULL; }

The import (C++) create cpp::****Decl which derive from the DMD ****Decl and override some of its parent methods. E.g:

   bool isBaseOf(::ClassDeclaration* cd, int* poffset) override;

makes isBaseOf take into account multiple inheritance and ask Clang to compute the right offsets.

Then in gen/, there is a different kind of hook which checks if a langPlugin is attached to the declaration:

   if (auto lp = s->langPlugin()) // yeah C++11, it's a prototype
     return lp->codeGen()->toCallFunction(s, ...);

And that way neither DMD nor LDC has to depend on Clang. The intrusions are the methods made virtual and the hooks, but if no language extension is present LDC would work as usual.

On Tuesday, 23 December 2014 at 11:53:38 UTC, Dicebot wrote:
> On Tuesday, 23 December 2014 at 10:52:58 UTC, Joseph Rushton Wakeling wrote:
>> clip
>
> Consider both things like embedded/MIPS and Windows64 - LLVM tooling is not as strong on those right now, GCC does not provide such easy way to reuse C++ frontend and with DMD/MSVC it is simply beyond feasibility.
>
> clip

Regarding GCC C++ frontend, will this be of help (also recently posted on this list):

http://forum.dlang.org/thread/weodkqwxrqetvolhbghb@forum.dlang.org

On Tuesday, 23 December 2014 at 07:18:01 UTC, Dicebot wrote:
> Project itself is very cool but I am in doubts about possibility of merging this upstream. Doing so would make full D implementation effectively impossible without some C++ compiler already available as a library on same platform - quite a restriction!

Why C++? Any language can be plugged in.

On Tuesday, 23 December 2014 at 11:53:38 UTC, Dicebot wrote:
> Consider both things like embedded/MIPS and Windows64 - LLVM tooling is not as strong on those right now

Win64 is nearing completion. LDC can compile and run code for quite some time already.

> provide such easy way to reuse C++ frontend and with DMD/MSVC it is simply beyond feasibility.

AFAIK, clang features MSVC compatibility.

> By upstream I don't mean LDC upstream but D upstream - we don't want it to become part of full language spec if implementation is so reliable on specific toolchain.

Another reason is to ease integration with C++. Language plugin is an obvious solution.