View mode: basic / threaded / horizontal-split · Log in · Help
February 21, 2012
Custom calling conventions
So I was thinking about this extern(language) thing, the obvious ones are
supported, but it would be really nice to be able to implement custom
conventions for other languages/scripting languages.

For instance, I'm thinking about Android, I have JNI binding code
everywhere, it's really ugly.
I'd love to be able to declare:
 extern(Java) int someJavaFunc(int x, float y)

And then use my function like any regular function, with the 'extern(Java)'
bit handling the JNI business behind the scenes.
I also regularly interact with javascript, lua, C#/mono, and these could
all be implemented the same way.

I'm imaging some mechanism to declare a calling convention (which would be
resolved within the extern(...) statement), and define it with a template,
something like:

callconv Java
{
 R call(T...)
 {
   // process tuple of args, make the call, return something?
 }

 R thisCall(Class, T...)
 {
   // also need a way to implementing methods... this might be enough.
 }
}

Some fancy code in there could conceivably call into any foreign language,
and this would be great!
Now when I: import java.jni;
I have the jni interface, but I also have access to extern(Java), and
that's awesome! :)

The main benefit over using a template, for
instance: jniCall!"functionName"(args...), would be the function name is
not a string, or require custom code construct (facilitating later
re-factoring or delegation to script without changing masses of existing
code, something I have done often), and if it was seen by the language as a
regular function call, you can mark it with all the usual stuff,
const/pure/etc, and apply the expected set of optimisations to the call.

I'm sure this has been discussed before... so go on, tear it apart :)
February 21, 2012
Re: Custom calling conventions
Why can't you do that with existing language features?

alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);

Where the templates generate the wrapper code/calling convention arg shuffle 
for each function.
February 21, 2012
Re: Custom calling conventions
On 21 February 2012 14:13, Daniel Murphy <yebblies@nospamgmail.com> wrote:

> Why can't you do that with existing language features?
>
> alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
> mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);
>
> Where the templates generate the wrapper code/calling convention arg
> shuffle
> for each function.
>

You could, but that's really ugly. I did make the point in my OP that it
can be hacked right now, but I also listed the benefits of making it a
proper feature:
1. It would be very neat and tidy. Clarity and readability is important
for code longevity.
2. Ease of refactoring/delegation to scripts without refactoring all the
calling code throughout your app (this is real, I have done this on many
occasions)
3. Declaring functions in the regular way, and annotating them with
various properties (const/pure/etc) allows all the usual type safety and
optimisation potentials available to the language.
    As far as the compiler is concerned, it is a real and normal function
call. All surrounding code gen/optimisations are still applicable, the
language will simply substitute the marshalling function at the final step
of the call.
4-ish. It actually seems relatively simple to implement... and it's not a
breaking change. I can't really see any down sides.
February 21, 2012
Re: Custom calling conventions
On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com> said:

> 
> So I was thinking about this extern(language) thing, the obvious ones are
> supported, but it would be really nice to be able to implement custom
> conventions for other languages/scripting languages.
> 
> For instance, I'm thinking about Android, I have JNI binding code
> everywhere, it's really ugly.
> I'd love to be able to declare:
>   extern(Java) int someJavaFunc(int x, float y)
> 
> And then use my function like any regular function, with the 'extern(Java)'
> bit handling the JNI business behind the scenes.
> I also regularly interact with javascript, lua, C#/mono, and these could
> all be implemented the same way.
> 
> I'm imaging some mechanism to declare a calling convention (which would be
> resolved within the extern(...) statement), and define it with a template,
> something like:
> 
> callconv Java
> {
>   R call(T...)
>   {
>     // process tuple of args, make the call, return something?
>   }
> 
>   R thisCall(Class, T...)
>   {
>     // also need a way to implementing methods... this might be enough.
>   }
> }
> 
> Some fancy code in there could conceivably call into any foreign language,
> and this would be great!
> Now when I: import java.jni;
> I have the jni interface, but I also have access to extern(Java), and
> that's awesome! :)

In all currently existing cases, extern(Lang) offers not only a way to 
declare and call foreign-language functions, but also a way to define 
functions to be called from the foreign language. Basically, you have 
two-way compatibility. How would that work with Java?

- - -

Also, I see you're thinking about methods, which means you're thinking 
about declaring Java classes. How is that supposed to work exactly?

I have some experience bridging Objective-C and D. I once built a 
complete wrapper system for Objective-C objects, each object was 
wrapped by a D one. It worked very well, but it generated so much bloat 
that it became unusable as soon as I started defining enough classes 
for it to be useful. See the D/Objective-C bridge: 
<http://michelf.com/projects/d-objc-bridge/>.

Maybe you'll have more luck, but for me wrappers just couldn't work. So 
I decided making the compiler understand the Objective-C runtime and 
object model was a much better solution. It was fun to do and wasn't 
that hard actually. Here's the result: 
<http://michelf.com/projects/d-objc/>. Unfortunately I don't have much 
time to work on it anymore.

More background:
<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/
February 21, 2012
Re: Custom calling conventions
On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com> wrote:

> On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com> said:
>
>
>> So I was thinking about this extern(language) thing, the obvious ones are
>> supported, but it would be really nice to be able to implement custom
>> conventions for other languages/scripting languages.
>>
>> For instance, I'm thinking about Android, I have JNI binding code
>> everywhere, it's really ugly.
>> I'd love to be able to declare:
>>  extern(Java) int someJavaFunc(int x, float y)
>>
>> And then use my function like any regular function, with the
>> 'extern(Java)'
>> bit handling the JNI business behind the scenes.
>> I also regularly interact with javascript, lua, C#/mono, and these could
>> all be implemented the same way.
>>
>> I'm imaging some mechanism to declare a calling convention (which would be
>> resolved within the extern(...) statement), and define it with a template,
>> something like:
>>
>> callconv Java
>> {
>>  R call(T...)
>>  {
>>    // process tuple of args, make the call, return something?
>>  }
>>
>>  R thisCall(Class, T...)
>>  {
>>    // also need a way to implementing methods... this might be enough.
>>  }
>> }
>>
>> Some fancy code in there could conceivably call into any foreign language,
>> and this would be great!
>> Now when I: import java.jni;
>> I have the jni interface, but I also have access to extern(Java), and
>> that's awesome! :)
>>
>
> In all currently existing cases, extern(Lang) offers not only a way to
> declare and call foreign-language functions, but also a way to define
> functions to be called from the foreign language. Basically, you have
> two-way compatibility. How would that work with Java?
>

Interesting point.
Java (or any VM based language) can't hard link to the binary, so it never
could try and call back in through an extern-ed function directly, it must
be explicitly supplied the pointer in the first place. In that case, the
API that supplies the callback to Java can produce the appropriate incoming
call wrapper.

I can imagine a way to doing a hard-extern under my proposal if you were
hard linking against an unsupported language language... you might need to
provide a name mangler in the definition block I describe to match the
foreign language, and an incoming call template would also need to be
defined as a naked function with the responsibility of managing the
args/stack appropriately, and calling the externed function as a regular
D-call (this would probably only be some light register/stack management, I
don't think it would bloat a lot). This would allow D code to also call the
externed function, since the physical definition is still a D-call, you
just call it directly internally.

Anyway, while I can imagine a solution, I don't think it would be needed
immediately. It'll just open a further can of worms like foreign language
struct layout as people try to use it in complex scenarios ;)

I don't think this reduces the usefulness of my proposal at all though, ie.
being able to apply it to outgoing calls for any foreign language.

- - -
>
> Also, I see you're thinking about methods, which means you're thinking
> about declaring Java classes. How is that supposed to work exactly?
>

Well I was thinking about that with respect to calling functions on classes
that I have imported from a foreign language, not declared locally for
export to a foreign language.
I clearly didn't think that through though, I'll try and clarify my
thoughts to that end... I get the feeling that idea was a dead-end though.


> I have some experience bridging Objective-C and D. I once built a complete
> wrapper system for Objective-C objects, each object was wrapped by a D one.
> It worked very well, but it generated so much bloat that it became unusable
> as soon as I started defining enough classes for it to be useful. See the
> D/Objective-C bridge: <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>
> >.
>

What was the primary cause of the bloat? I can't imagine my proposal
causing any more bloat than the explicit jni call (or equivalent) woudl
have otherwise.


> Maybe you'll have more luck, but for me wrappers just couldn't work. So I
> decided making the compiler understand the Objective-C runtime and object
> model was a much better solution. It was fun to do and wasn't that hard
> actually. Here's the result: <http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.
> Unfortunately I don't have much time to work on it anymore.
>

That sounds appropriate for Obj-C, but I don't think it applies to VM based
languages however you slice it, since there's not actually a direct binary
linkage. There's no way to know about the Java/.NET/Lua/etc object model
internally, the VM owns that entirely, so you're forced to marshal through
API's. That's okay though, you have to do it anyway. I'm proposing a way to
unify the concept of calling functions without obscuring the actual call
with junk, and still allowing the D language to annotate and optimise to
its fullest extent.

More background:
> <http://michelf.com/weblog/**2010/dobjc-dead-end-start-**anew/<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>
> >
>
>
> --
> Michel Fortin
> michel.fortin@michelf.com
> http://michelf.com/
>
>
February 21, 2012
Re: Custom calling conventions
On 21-02-2012 18:03, Manu wrote:
> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
> <mailto:michel.fortin@michelf.com>> wrote:
>
>     On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com
>     <mailto:turkeyman@gmail.com>> said:
>
>
>         So I was thinking about this extern(language) thing, the obvious
>         ones are
>         supported, but it would be really nice to be able to implement
>         custom
>         conventions for other languages/scripting languages.
>
>         For instance, I'm thinking about Android, I have JNI binding code
>         everywhere, it's really ugly.
>         I'd love to be able to declare:
>           extern(Java) int someJavaFunc(int x, float y)
>
>         And then use my function like any regular function, with the
>         'extern(Java)'
>         bit handling the JNI business behind the scenes.
>         I also regularly interact with javascript, lua, C#/mono, and
>         these could
>         all be implemented the same way.
>
>         I'm imaging some mechanism to declare a calling convention
>         (which would be
>         resolved within the extern(...) statement), and define it with a
>         template,
>         something like:
>
>         callconv Java
>         {
>           R call(T...)
>           {
>             // process tuple of args, make the call, return something?
>           }
>
>           R thisCall(Class, T...)
>           {
>             // also need a way to implementing methods... this might be
>         enough.
>           }
>         }
>
>         Some fancy code in there could conceivably call into any foreign
>         language,
>         and this would be great!
>         Now when I: import java.jni;
>         I have the jni interface, but I also have access to
>         extern(Java), and
>         that's awesome! :)
>
>
>     In all currently existing cases, extern(Lang) offers not only a way
>     to declare and call foreign-language functions, but also a way to
>     define functions to be called from the foreign language. Basically,
>     you have two-way compatibility. How would that work with Java?
>
>
> Interesting point.
> Java (or any VM based language) can't hard link to the binary, so it
> never could try and call back in through an extern-ed function directly,

Sure it could. Why wouldn't it be able to? This is how extern + 
DllImport works in C#.

> it must be explicitly supplied the pointer in the first place. In that
> case, the API that supplies the callback to Java can produce the
> appropriate incoming call wrapper.
>
> I can imagine a way to doing a hard-extern under my proposal if you were
> hard linking against an unsupported language language... you might need
> to provide a name mangler in the definition block I describe to match
> the foreign language, and an incoming call template would also need to
> be defined as a naked function with the responsibility of managing the
> args/stack appropriately, and calling the externed function as a regular
> D-call (this would probably only be some light register/stack
> management, I don't think it would bloat a lot). This would allow D code
> to also call the externed function, since the physical definition is
> still a D-call, you just call it directly internally.
>
> Anyway, while I can imagine a solution, I don't think it would be needed
> immediately. It'll just open a further can of worms like foreign
> language struct layout as people try to use it in complex scenarios ;)
>
> I don't think this reduces the usefulness of my proposal at all though,
> ie. being able to apply it to outgoing calls for any foreign language.
>
>     - - -
>
>     Also, I see you're thinking about methods, which means you're
>     thinking about declaring Java classes. How is that supposed to work
>     exactly?
>
>
> Well I was thinking about that with respect to calling functions on
> classes that I have imported from a foreign language, not declared
> locally for export to a foreign language.
> I clearly didn't think that through though, I'll try and clarify my
> thoughts to that end... I get the feeling that idea was a dead-end though.
>
>     I have some experience bridging Objective-C and D. I once built a
>     complete wrapper system for Objective-C objects, each object was
>     wrapped by a D one. It worked very well, but it generated so much
>     bloat that it became unusable as soon as I started defining enough
>     classes for it to be useful. See the D/Objective-C bridge:
>     <http://michelf.com/projects/__d-objc-bridge/
>     <http://michelf.com/projects/d-objc-bridge/>>.
>
>
> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.
>
>     Maybe you'll have more luck, but for me wrappers just couldn't
>     work.. So I decided making the compiler understand the Objective-C
>     runtime and object model was a much better solution. It was fun to
>     do and wasn't that hard actually. Here's the result:
>     <http://michelf.com/projects/__d-objc/
>     <http://michelf.com/projects/d-objc/>>. Unfortunately I don't have
>     much time to work on it anymore.
>
>
> That sounds appropriate for Obj-C, but I don't think it applies to VM
> based languages however you slice it, since there's not actually a
> direct binary linkage. There's no way to know about the
> Java/.NET/Lua/etc object model internally, the VM owns that entirely, so
> you're forced to marshal through API's. That's okay though, you have to
> do it anyway. I'm proposing a way to unify the concept of calling
> functions without obscuring the actual call with junk, and still
> allowing the D language to annotate and optimise to its fullest extent.
>
>     More background:
>     <http://michelf.com/weblog/__2010/dobjc-dead-end-start-__anew/
>     <http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>>
>
>
>     --
>     Michel Fortin
>     michel.fortin@michelf.com <mailto:michel.fortin@michelf.com>
>     http://michelf.com/
>
>

-- 
- Alex
February 21, 2012
Re: Custom calling conventions
On 2012-02-21 18:03, Manu wrote:
> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
>     I have some experience bridging Objective-C and D. I once built a
>     complete wrapper system for Objective-C objects, each object was
>     wrapped by a D one. It worked very well, but it generated so much
>     bloat that it became unusable as soon as I started defining enough
>     classes for it to be useful. See the D/Objective-C bridge:
>     <http://michelf.com/projects/__d-objc-bridge/
>     <http://michelf.com/projects/d-objc-bridge/>>.
>
>
> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.

Template bloat. Every call bridging D/Objective-C is made throw a series 
of templates. This is for making it possible (less verbose) to create 
bindings.

It might be possible to decrease the template bloat by having a tool 
that automatically generates the bindings and outputs what the templates 
do inline.

-- 
/Jacob Carlborg
February 21, 2012
Re: Custom calling conventions
I think this fails in the same trap as extern "language" in C++.

Besides C and C++, I think the only variation I saw so far for the 
language part has been fortran, and cannot recall anylonger in what
compiler it was.

In C++'s case, since it is implementation defined, most compiler vendors
did not bother to support more than what the standard specifies (C and C++).

In you proposal, how to keep the feature portable across D implementations?

If you make it part of D language specification, which languages would 
be the chosen ones to be made available in any D compiler?

How to garantee correct interoperability (ABI) with other languages? 
After deciding the set of chosen languages, which compilers/runtimes 
would be the lucky ones?

--
Paulo

Am 21.02.2012 13:29, schrieb Manu:
> On 21 February 2012 14:13, Daniel Murphy <yebblies@nospamgmail.com
> <mailto:yebblies@nospamgmail.com>> wrote:
>
>     Why can't you do that with existing language features?
>
>     alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
>     mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);
>
>     Where the templates generate the wrapper code/calling convention arg
>     shuffle
>     for each function.
>
>
> You could, but that's really ugly. I did make the point in my OP that it
> can be hacked right now, but I also listed the benefits of making it a
> proper feature:
>   1. It would be very neat and tidy. Clarity and readability is
> important for code longevity.
>   2. Ease of refactoring/delegation to scripts without refactoring all
> the calling code throughout your app (this is real, I have done this on
> many occasions)
>   3. Declaring functions in the regular way, and annotating them with
> various properties (const/pure/etc) allows all the usual type safety and
> optimisation potentials available to the language.
>       As far as the compiler is concerned, it is a real and normal
> function call. All surrounding code gen/optimisations are still
> applicable, the language will simply substitute the marshalling function
> at the final step of the call.
>   4-ish. It actually seems relatively simple to implement... and it's
> not a breaking change. I can't really see any down sides.
February 21, 2012
Re: Custom calling conventions
On 2012-02-21 17:03:56 +0000, Manu <turkeyman@gmail.com> said:

> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.

I had one D class for every corresponding Objective-C class, each 
method wrapped in both direction, each wrapper doing automatic mapping 
of D and Objective-C wrapper objects and translating exceptions. I 
could actually override a method in a derived class of the D wrapper 
and it'd effectively override the method on the Objective-C side. But 
with all those D classes having virtual tables referring to all these 
wrapper functions, even if you use only 1% of all the bindings nothing 
can be stripped of the final executable.

Quoting from a blog post of mine:
"""
But the real show stopper is the size of the generated code for the 
bindings. The system I created needs to generate a lot of code for each 
and every one of the functions and classes you define bindings for. 
This makes the final executable awfully big (and might also explain in 
part the slowness while compiling). The small test application included 
with the bridge takes about 60 Mb of code, thats for something that’d 
be only a few kilobytes when written in Objective-C.

This wasn’t the case at first. This bloat appeared as I added bindings 
for more and more classes in Cocoa. It is a real problem because no one 
in his right mind will want to ship a 60 Mb application that contains 
so much binding code that stay unused most of the time. A solution 
could be that everyone would have to write bindings that only contains 
the classes and methods he needs, but this is of rather limited 
interest.
"""

>> Maybe you'll have more luck, but for me wrappers just couldn't work. So I
>> decided making the compiler understand the Objective-C runtime and object
>> model was a much better solution. It was fun to do and wasn't that hard
>> actually. Here's the result: 
>> <http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.
Unfortunately 
>> 
>> I don't have much time to work on it anymore.
> 
> That sounds appropriate for Obj-C, but I don't think it applies to VM based
> languages however you slice it, since there's not actually a direct binary
> linkage. There's no way to know about the Java/.NET/Lua/etc object model
> internally, the VM owns that entirely, so you're forced to marshal through
> API's.

That's not actually very different from what I'm doing for Objective-C. 
Call to Objective-C methods are all done through low-level API 
functions in the Objective-C runtime. Class definitions are static 
data, but could be done dynamically through API calls in module 
constructors.

The most interesting bit however is that class *declarations* require 
nothing to be compiled, the compiler will just emit the right code when 
calling a method, using objc_msgSend(). This has a major impact on 
compilation time if you have thousands of such functions.


> That's okay though, you have to do it anyway. I'm proposing a way to
> unify the concept of calling functions without obscuring the actual call
> with junk, and still allowing the D language to annotate and optimise to
> its fullest extent.

The goal is certainly worthwhile. I think your proposal has merit if 
you goal is just to call functions and not define function or classes, 
although I'm skeptical overloading extern() is the right way to do that 
(perhaps a pragma?). But I'd suggest you define your goals more 
throughly before deciding on an implementation.

Some questions you should try to have an answer for regarding OO: if 
you're interacting with classes, how should it work in an ideal world? 
Then, do you need to define new classes that'd be available in the 
foreign language? Do you need to override functions on existing 
classes? How do you want code using foreign objects to look? Then ask 
yourself what would be an acceptable compromise, and start implementing 
it.

For my part, I came to the conclusion that you couldn't really use 
Objective-C APIs in a meaningful way without the ability to create 
subclasses and override functions… which led me to the current design 
where you can use Objective-C classes as if they were D classes.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/
February 21, 2012
Re: Custom calling conventions
On 21 February 2012 19:10, Alex Rønne Petersen <xtzgzorex@gmail.com> wrote:

> On 21-02-2012 18:03, Manu wrote:
>
>> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
>> <mailto:michel.fortin@michelf.**com <michel.fortin@michelf.com>>> wrote:
>>
>>    On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com
>>    <mailto:turkeyman@gmail.com>> said:
>>
>> Interesting point.
>> Java (or any VM based language) can't hard link to the binary, so it
>> never could try and call back in through an extern-ed function directly,
>>
>
> Sure it could. Why wouldn't it be able to? This is how extern + DllImport
> works in C#.
>

But that only works because C# has DLL support, which knows about the C
calling convention, so you just need to export an extern(C) in that case,
like any regular DLL.
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home