Hi everyone,
recently I was working on .NET embedding, and I have done simple library for hosting .NET Core in D, it is accompanied with small C# library that takes care of some limitations of current .NET implementation regarding native delegates.
It allows to embed .NET runtime in your program and call .NET code, for example it might be some important library implemented only in C# or maybe you want to build your own game engine and use it as scripting* engine like Unity does.
*Oh, and this is also interesting because .NET Core 3.1 size is roughly 70 MB with libs, which is nearly the same size as CPython, but has advantage of performance AND security features.

It is still in quite early stage and lacks some important features, so I don't recommend using it in production.
However I think some might find it useful, others might like to evaluate the possibilities.


I have quite fuzzy roadmap for future related projects ATM and I can't give any precise dates, but anyway here is 4 projects that I have in mind:

    1)  host utility library (WIP, already has some crucial features, prerequisite for all future work on this list)
    2)  D .NET host library (WIP, .NET Core low level stuff done, high level .NET wrapper generator is still not designed though)
    3)  utility tool that outputs proper declarations to use in D (planned)
    4)  D to .NET compiler, this ultimately requires all previous steps done, and I still haven't sorted out what D features could be problematic to implement (not yet started)

Wrt. to compiler my biggest concern is actually not the language features itself, since .NET has almost everything D can do, but rather D runtime and standard library. This is however is so distant future that I can't even say I will ever do this.
The good thing is that I can work on compiler itself thanks to dmd-as-a-library dub package.


Now of course the question is why I am even bother doing this?
- .NET 5 is going to release later in 2020 finally unifying both MS .NET and Mono VM's
- it will have many improvements, features and stuff, all this is cross platform (you can find more info on the net) (oh crap, sounds like some BS ad)
- AOT/JIT (D lacks this, can benefit REPL's, and can also in some cases give you that last drops of performance)
- so far no other "serious" language have targeted .NET so this niche is quite empty, this could give D enormous advantage and recognition especially in enterprise (don't quote me on this though)
- ???
- profit (oh, I hope so)


source code
https://github.com/Superbelko/dotnethost-d

On Tuesday, 11 February 2020 at 05:21:20 UTC, evilrat wrote:
> Hi everyone,
> recently I was working on .NET embedding, and I have done simple library for hosting .NET Core in D, it is accompanied with small C# library that takes care of some limitations of current .NET implementation regarding native delegates.
> It allows to embed .NET runtime in your program and call .NET code, for example it might be some important library implemented only in C# or maybe you want to build your own game engine and use it as scripting* engine like Unity does.
> *Oh, and this is also interesting because .NET Core 3.1 size is roughly 70 MB with libs, which is nearly the same size as CPython, but has advantage of performance AND security features.
>
> [...]

I started a project just over a month ago that fills the same role:

https://github.com/marler8997/clrbridge

I started with the dflat project found here (https://github.com/thewilsonator/dflat) and eventually changed the design to create this new one.  I'll have to take a look at your project and see if we can share/mutually benefit from our approaches.

On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler wrote:
>
> I started a project just over a month ago that fills the same role:
>
> https://github.com/marler8997/clrbridge
>
> I started with the dflat project found here (https://github.com/thewilsonator/dflat) and eventually changed the design to create this new one.  I'll have to take a look at your project and see if we can share/mutually benefit from our approaches.

Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved.

Your project seems to aim to be a complete solution, it looks far more robust but also much more complex.

While my library is basically a .NET Core API wrapper over 5 native functions with extra 7 or so C# functions. No external tools are required, no patching, no intermediate steps. Just provide type declarations in D and that's it.
The most complex part is IL generator that weaves in custom marshaling logic that avoids current CLR implementation limitations (which may or may not be resolved in .NET 5), but it also means delegates generated on demand, and the next biggest chunk is huge D template to glue this together and hide memory management details.
I also keep it pretty independent, one is free to make their own wrapper generator, also feel free to use delegate emitter library in other projects or even languages.

When designing it I tried to keep it minimalistic as possible, I also think it nicely fits the position as a temporary solution until I do real compiler (if ever). Meanwhile it might be useful on occasion to call few methods here and there.

On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:
> On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler wrote:
>>
>> I started a project just over a month ago that fills the same role:
>>
>> https://github.com/marler8997/clrbridge
>>
>> I started with the dflat project found here (https://github.com/thewilsonator/dflat) and eventually changed the design to create this new one.  I'll have to take a look at your project and see if we can share/mutually benefit from our approaches.
>
> Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved.

After spending some time working on dflat I came to the same conclusion.  Since coreclr only supports static functions, Dflat was using Cecil to generate wrapper assemblies in order to access non-static functions.  However, it's simpler to have just one library that can do this for any assembly rather than generating a new unique wrapper assembly for each one.

>
> Your project seems to aim to be a complete solution, it looks far more robust but also much more complex.
>
> While my library is basically a .NET Core API wrapper over 5 native functions with extra 7 or so C# functions. No external tools are required, no patching, no intermediate steps. Just provide type declarations in D and that's it.

The code generation side adds complexity.  However, all the generated code ends up calling into a small simple library called ClrBridge.dll.

https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs

It defines a handful static functions to load assemblies, get types, methods, call methods, marshal values, box values and create arrays.  The bare minimum you need to have access to the entire Clr if you're only able to call static methods through the coreclr library.

You can have access to the full CLR with this library without any code generation as well.  Here's an example that doesn't use any code generation:

https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d


> The most complex part is IL generator that weaves in custom marshaling logic that avoids current CLR implementation limitations (which may or may not be resolved in .NET 5), but it also means delegates generated on demand, and the next biggest chunk is huge D template to glue this together and hide memory management details.

Interesting.  I haven't found a case yet where I've needed to generate IL code to marshal types.  I don't anticipate needing to either as I already have a design that should work for marshaling any type.  What types are you needing to generate IL to marshal for?

On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:
> On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler wrote:
>>
>> I started a project just over a month ago that fills the same role:
>>
>> https://github.com/marler8997/clrbridge
>>
>> I started with the dflat project found here (https://github.com/thewilsonator/dflat) and eventually changed the design to create this new one.  I'll have to take a look at your project and see if we can share/mutually benefit from our approaches.
>
> Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved.

After spending some time working on dflat I came to the same conclusion.  Since coreclr only supports static functions, Dflat was using Cecil to generate wrapper assemblies in order to access non-static functions.  However, it's simpler to have just one library that can do this for any assembly rather than generating a new unique wrapper assembly for each one.

>
> Your project seems to aim to be a complete solution, it looks far more robust but also much more complex.
>
> While my library is basically a .NET Core API wrapper over 5 native functions with extra 7 or so C# functions. No external tools are required, no patching, no intermediate steps. Just provide type declarations in D and that's it.

The code generation side adds complexity.  However, all the generated code ends up calling into a small simple library called ClrBridge.dll.

https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs

It defines a handful static functions to load assemblies, get types, methods, call methods, marshal values, box values and create arrays.  The bare minimum you need to have access to the entire Clr if you're only able to call static methods through the coreclr library.

You can have access to the full CLR with this library without any code generation as well.  Here's an example that doesn't use any code generation:

https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d


> The most complex part is IL generator that weaves in custom marshaling logic that avoids current CLR implementation limitations (which may or may not be resolved in .NET 5), but it also means delegates generated on demand, and the next biggest chunk is huge D template to glue this together and hide memory management details.

Interesting.  I haven't found a case yet where I've needed to generate IL code to marshal types.  I don't anticipate needing to either as I already have a design that should work for marshaling any type.  What types are you needing to generate IL to marshal for?

On Tuesday, 11 February 2020 at 08:47:48 UTC, Jonathan Marler wrote:
>
> The code generation side adds complexity.  However, all the generated code ends up calling into a small simple library called ClrBridge.dll.
>
> https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs

Yes, but it is moved from user to library maintainer. That way users are happy, they just don't care about these details. And it only adds single DLL with few dependencies to redistribute. In total 12 files less than 1 MB in size, including 4 files for Mono.Cecil which is a leftover from my compiler experiments that I forgot to exclude.

>
> It defines a handful static functions to load assemblies, get types, methods, call methods, marshal values, box values and create arrays.  The bare minimum you need to have access to the entire Clr if you're only able to call static methods through the coreclr library.
>

Most of this is accessible without extra helpers.
That's why I did delegate handling first. It creates static delegate at runtime that can be called as normal function pointer, and rewrites incompatible types to pass handles instead, then before calling former method it obtains actual objects using that handles.

System.Activator - is 'new' operator.
System.Array - all things for arrays.
System.Type - basically all reflection
System.Delegate - call methods, etc..

From that list I only need static helpers for Type.GetType(), Type.GetMethod(), Delegate.CreateDelegate() and Marshal.GetFunctionPointerForDelegate(), this provides access to specific types and methods and after creating delegate and then obtaining callable function pointer with these helpers the rest should be accessible.


> You can have access to the full CLR with this library without any code generation as well.  Here's an example that doesn't use any code generation:
>
> https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d
>

ok, will take a second look.


>
> Interesting.  I haven't found a case yet where I've needed to generate IL code to marshal types.  I don't anticipate needing to either as I already have a design that should work for marshaling any type.  What types are you needing to generate IL to marshal for?

There is not much going on.
My conclusion is that they just not yet implemented full handling for delegates so far.
So all I did was just take reference types in method params/return instead to be rewritten as IntPtr and handled using GCHandle, simply it just calls GCHandle.Alloc(ret)/GCHandle.FromIntPtr(param).Target on return value and parameters.
(hamburger analogy where meat is the real method and bread is that extra handling)

Though this can change later as it is still at the very early stage right now. This does not account for structs with reference types, etc...
Also CLR does marshal arrays just fine, while my current solution currently takes the route of using object handle for that(which might be slower or even incorrect in some cases?).