So I was thinking about this extern(language) thing, the obvious ones are supported, but it would be really nice to be able to implement custom conventions for other languages/scripting languages.

For instance, I'm thinking about Android, I have JNI binding code
everywhere, it's really ugly.
I'd love to be able to declare:
  extern(Java) int someJavaFunc(int x, float y)

And then use my function like any regular function, with the 'extern(Java)'
bit handling the JNI business behind the scenes.
I also regularly interact with javascript, lua, C#/mono, and these could
all be implemented the same way.

I'm imaging some mechanism to declare a calling convention (which would be
resolved within the extern(...) statement), and define it with a template,
something like:

callconv Java
{
  R call(T...)
  {
    // process tuple of args, make the call, return something?
  }

  R thisCall(Class, T...)
  {
    // also need a way to implementing methods... this might be enough.
  }
}

Some fancy code in there could conceivably call into any foreign language,
and this would be great!
Now when I: import java.jni;
I have the jni interface, but I also have access to extern(Java), and
that's awesome! :)

The main benefit over using a template, for
instance: jniCall!"functionName"(args...), would be the function name is
not a string, or require custom code construct (facilitating later
re-factoring or delegation to script without changing masses of existing
code, something I have done often), and if it was seen by the language as a
regular function call, you can mark it with all the usual stuff,
const/pure/etc, and apply the expected set of optimisations to the call.

I'm sure this has been discussed before... so go on, tear it apart :)

Why can't you do that with existing language features?

alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);

Where the templates generate the wrapper code/calling convention arg shuffle for each function.

On 21 February 2012 14:13, Daniel Murphy <yebblies@nospamgmail.com> wrote:

> Why can't you do that with existing language features?
>
> alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
> mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);
>
> Where the templates generate the wrapper code/calling convention arg
> shuffle
> for each function.
>

You could, but that's really ugly. I did make the point in my OP that it
can be hacked right now, but I also listed the benefits of making it a
proper feature:
 1. It would be very neat and tidy. Clarity and readability is important
for code longevity.
 2. Ease of refactoring/delegation to scripts without refactoring all the
calling code throughout your app (this is real, I have done this on many
occasions)
 3. Declaring functions in the regular way, and annotating them with
various properties (const/pure/etc) allows all the usual type safety and
optimisation potentials available to the language.
     As far as the compiler is concerned, it is a real and normal function
call. All surrounding code gen/optimisations are still applicable, the
language will simply substitute the marshalling function at the final step
of the call.
 4-ish. It actually seems relatively simple to implement... and it's not a
breaking change. I can't really see any down sides.

On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com> said:

> 
> So I was thinking about this extern(language) thing, the obvious ones are
> supported, but it would be really nice to be able to implement custom
> conventions for other languages/scripting languages.
> 
> For instance, I'm thinking about Android, I have JNI binding code
> everywhere, it's really ugly.
> I'd love to be able to declare:
>   extern(Java) int someJavaFunc(int x, float y)
> 
> And then use my function like any regular function, with the 'extern(Java)'
> bit handling the JNI business behind the scenes.
> I also regularly interact with javascript, lua, C#/mono, and these could
> all be implemented the same way.
> 
> I'm imaging some mechanism to declare a calling convention (which would be
> resolved within the extern(...) statement), and define it with a template,
> something like:
> 
> callconv Java
> {
>   R call(T...)
>   {
>     // process tuple of args, make the call, return something?
>   }
> 
>   R thisCall(Class, T...)
>   {
>     // also need a way to implementing methods... this might be enough.
>   }
> }
> 
> Some fancy code in there could conceivably call into any foreign language,
> and this would be great!
> Now when I: import java.jni;
> I have the jni interface, but I also have access to extern(Java), and
> that's awesome! :)

In all currently existing cases, extern(Lang) offers not only a way to declare and call foreign-language functions, but also a way to define functions to be called from the foreign language. Basically, you have two-way compatibility. How would that work with Java?

- - -

Also, I see you're thinking about methods, which means you're thinking about declaring Java classes. How is that supposed to work exactly?

I have some experience bridging Objective-C and D. I once built a complete wrapper system for Objective-C objects, each object was wrapped by a D one. It worked very well, but it generated so much bloat that it became unusable as soon as I started defining enough classes for it to be useful. See the D/Objective-C bridge: <http://michelf.com/projects/d-objc-bridge/>.

Maybe you'll have more luck, but for me wrappers just couldn't work. So I decided making the compiler understand the Objective-C runtime and object model was a much better solution. It was fun to do and wasn't that hard actually. Here's the result: <http://michelf.com/projects/d-objc/>. Unfortunately I don't have much time to work on it anymore.

More background:
<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com> wrote:

> On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com> said:
>
>
>> So I was thinking about this extern(language) thing, the obvious ones are supported, but it would be really nice to be able to implement custom conventions for other languages/scripting languages.
>>
>> For instance, I'm thinking about Android, I have JNI binding code
>> everywhere, it's really ugly.
>> I'd love to be able to declare:
>>  extern(Java) int someJavaFunc(int x, float y)
>>
>> And then use my function like any regular function, with the
>> 'extern(Java)'
>> bit handling the JNI business behind the scenes.
>> I also regularly interact with javascript, lua, C#/mono, and these could
>> all be implemented the same way.
>>
>> I'm imaging some mechanism to declare a calling convention (which would be
>> resolved within the extern(...) statement), and define it with a template,
>> something like:
>>
>> callconv Java
>> {
>>  R call(T...)
>>  {
>>    // process tuple of args, make the call, return something?
>>  }
>>
>>  R thisCall(Class, T...)
>>  {
>>    // also need a way to implementing methods... this might be enough.
>>  }
>> }
>>
>> Some fancy code in there could conceivably call into any foreign language,
>> and this would be great!
>> Now when I: import java.jni;
>> I have the jni interface, but I also have access to extern(Java), and
>> that's awesome! :)
>>
>
> In all currently existing cases, extern(Lang) offers not only a way to declare and call foreign-language functions, but also a way to define functions to be called from the foreign language. Basically, you have two-way compatibility. How would that work with Java?
>

Interesting point.
Java (or any VM based language) can't hard link to the binary, so it never
could try and call back in through an extern-ed function directly, it must
be explicitly supplied the pointer in the first place. In that case, the
API that supplies the callback to Java can produce the appropriate incoming
call wrapper.

I can imagine a way to doing a hard-extern under my proposal if you were hard linking against an unsupported language language... you might need to provide a name mangler in the definition block I describe to match the foreign language, and an incoming call template would also need to be defined as a naked function with the responsibility of managing the args/stack appropriately, and calling the externed function as a regular D-call (this would probably only be some light register/stack management, I don't think it would bloat a lot). This would allow D code to also call the externed function, since the physical definition is still a D-call, you just call it directly internally.

Anyway, while I can imagine a solution, I don't think it would be needed immediately. It'll just open a further can of worms like foreign language struct layout as people try to use it in complex scenarios ;)

I don't think this reduces the usefulness of my proposal at all though, ie. being able to apply it to outgoing calls for any foreign language.

- - -
>
> Also, I see you're thinking about methods, which means you're thinking about declaring Java classes. How is that supposed to work exactly?
>

Well I was thinking about that with respect to calling functions on classes
that I have imported from a foreign language, not declared locally for
export to a foreign language.
I clearly didn't think that through though, I'll try and clarify my
thoughts to that end... I get the feeling that idea was a dead-end though.


> I have some experience bridging Objective-C and D. I once built a complete
> wrapper system for Objective-C objects, each object was wrapped by a D one.
> It worked very well, but it generated so much bloat that it became unusable
> as soon as I started defining enough classes for it to be useful. See the
> D/Objective-C bridge: <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>
> >.
>

What was the primary cause of the bloat? I can't imagine my proposal causing any more bloat than the explicit jni call (or equivalent) woudl have otherwise.


> Maybe you'll have more luck, but for me wrappers just couldn't work. So I
> decided making the compiler understand the Objective-C runtime and object
> model was a much better solution. It was fun to do and wasn't that hard
> actually. Here's the result: <http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.
> Unfortunately I don't have much time to work on it anymore.
>

That sounds appropriate for Obj-C, but I don't think it applies to VM based languages however you slice it, since there's not actually a direct binary linkage. There's no way to know about the Java/.NET/Lua/etc object model internally, the VM owns that entirely, so you're forced to marshal through API's. That's okay though, you have to do it anyway. I'm proposing a way to unify the concept of calling functions without obscuring the actual call with junk, and still allowing the D language to annotate and optimise to its fullest extent.

More background:
> <http://michelf.com/weblog/**2010/dobjc-dead-end-start-**anew/<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>
> >
>
>
> --
> Michel Fortin
> michel.fortin@michelf.com
> http://michelf.com/
>
>

On 21-02-2012 18:03, Manu wrote:
> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
> <mailto:michel.fortin@michelf.com>> wrote:
>
>     On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com
>     <mailto:turkeyman@gmail.com>> said:
>
>
>         So I was thinking about this extern(language) thing, the obvious
>         ones are
>         supported, but it would be really nice to be able to implement
>         custom
>         conventions for other languages/scripting languages.
>
>         For instance, I'm thinking about Android, I have JNI binding code
>         everywhere, it's really ugly.
>         I'd love to be able to declare:
>           extern(Java) int someJavaFunc(int x, float y)
>
>         And then use my function like any regular function, with the
>         'extern(Java)'
>         bit handling the JNI business behind the scenes.
>         I also regularly interact with javascript, lua, C#/mono, and
>         these could
>         all be implemented the same way.
>
>         I'm imaging some mechanism to declare a calling convention
>         (which would be
>         resolved within the extern(...) statement), and define it with a
>         template,
>         something like:
>
>         callconv Java
>         {
>           R call(T...)
>           {
>             // process tuple of args, make the call, return something?
>           }
>
>           R thisCall(Class, T...)
>           {
>             // also need a way to implementing methods... this might be
>         enough.
>           }
>         }
>
>         Some fancy code in there could conceivably call into any foreign
>         language,
>         and this would be great!
>         Now when I: import java.jni;
>         I have the jni interface, but I also have access to
>         extern(Java), and
>         that's awesome! :)
>
>
>     In all currently existing cases, extern(Lang) offers not only a way
>     to declare and call foreign-language functions, but also a way to
>     define functions to be called from the foreign language. Basically,
>     you have two-way compatibility. How would that work with Java?
>
>
> Interesting point.
> Java (or any VM based language) can't hard link to the binary, so it
> never could try and call back in through an extern-ed function directly,

Sure it could. Why wouldn't it be able to? This is how extern + DllImport works in C#.

> it must be explicitly supplied the pointer in the first place. In that
> case, the API that supplies the callback to Java can produce the
> appropriate incoming call wrapper.
>
> I can imagine a way to doing a hard-extern under my proposal if you were
> hard linking against an unsupported language language... you might need
> to provide a name mangler in the definition block I describe to match
> the foreign language, and an incoming call template would also need to
> be defined as a naked function with the responsibility of managing the
> args/stack appropriately, and calling the externed function as a regular
> D-call (this would probably only be some light register/stack
> management, I don't think it would bloat a lot). This would allow D code
> to also call the externed function, since the physical definition is
> still a D-call, you just call it directly internally.
>
> Anyway, while I can imagine a solution, I don't think it would be needed
> immediately. It'll just open a further can of worms like foreign
> language struct layout as people try to use it in complex scenarios ;)
>
> I don't think this reduces the usefulness of my proposal at all though,
> ie. being able to apply it to outgoing calls for any foreign language.
>
>     - - -
>
>     Also, I see you're thinking about methods, which means you're
>     thinking about declaring Java classes. How is that supposed to work
>     exactly?
>
>
> Well I was thinking about that with respect to calling functions on
> classes that I have imported from a foreign language, not declared
> locally for export to a foreign language.
> I clearly didn't think that through though, I'll try and clarify my
> thoughts to that end... I get the feeling that idea was a dead-end though.
>
>     I have some experience bridging Objective-C and D. I once built a
>     complete wrapper system for Objective-C objects, each object was
>     wrapped by a D one. It worked very well, but it generated so much
>     bloat that it became unusable as soon as I started defining enough
>     classes for it to be useful. See the D/Objective-C bridge:
>     <http://michelf.com/projects/__d-objc-bridge/
>     <http://michelf.com/projects/d-objc-bridge/>>.
>
>
> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.
>
>     Maybe you'll have more luck, but for me wrappers just couldn't
>     work.. So I decided making the compiler understand the Objective-C
>     runtime and object model was a much better solution. It was fun to
>     do and wasn't that hard actually. Here's the result:
>     <http://michelf.com/projects/__d-objc/
>     <http://michelf.com/projects/d-objc/>>. Unfortunately I don't have
>     much time to work on it anymore.
>
>
> That sounds appropriate for Obj-C, but I don't think it applies to VM
> based languages however you slice it, since there's not actually a
> direct binary linkage. There's no way to know about the
> Java/.NET/Lua/etc object model internally, the VM owns that entirely, so
> you're forced to marshal through API's. That's okay though, you have to
> do it anyway. I'm proposing a way to unify the concept of calling
> functions without obscuring the actual call with junk, and still
> allowing the D language to annotate and optimise to its fullest extent.
>
>     More background:
>     <http://michelf.com/weblog/__2010/dobjc-dead-end-start-__anew/
>     <http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>>
>
>
>     --
>     Michel Fortin
>     michel.fortin@michelf.com <mailto:michel.fortin@michelf.com>
>     http://michelf.com/
>
>

-- 
- Alex

On 2012-02-21 18:03, Manu wrote:
> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
>     I have some experience bridging Objective-C and D. I once built a
>     complete wrapper system for Objective-C objects, each object was
>     wrapped by a D one. It worked very well, but it generated so much
>     bloat that it became unusable as soon as I started defining enough
>     classes for it to be useful. See the D/Objective-C bridge:
>     <http://michelf.com/projects/__d-objc-bridge/
>     <http://michelf.com/projects/d-objc-bridge/>>.
>
>
> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.

Template bloat. Every call bridging D/Objective-C is made throw a series of templates. This is for making it possible (less verbose) to create bindings.

It might be possible to decrease the template bloat by having a tool that automatically generates the bindings and outputs what the templates do inline.

-- 
/Jacob Carlborg

I think this fails in the same trap as extern "language" in C++.

Besides C and C++, I think the only variation I saw so far for the language part has been fortran, and cannot recall anylonger in what
compiler it was.

In C++'s case, since it is implementation defined, most compiler vendors
did not bother to support more than what the standard specifies (C and C++).

In you proposal, how to keep the feature portable across D implementations?

If you make it part of D language specification, which languages would be the chosen ones to be made available in any D compiler?

How to garantee correct interoperability (ABI) with other languages? After deciding the set of chosen languages, which compilers/runtimes would be the lucky ones?

--
Paulo

Am 21.02.2012 13:29, schrieb Manu:
> On 21 February 2012 14:13, Daniel Murphy <yebblies@nospamgmail.com
> <mailto:yebblies@nospamgmail.com>> wrote:
>
>     Why can't you do that with existing language features?
>
>     alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
>     mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);
>
>     Where the templates generate the wrapper code/calling convention arg
>     shuffle
>     for each function.
>
>
> You could, but that's really ugly. I did make the point in my OP that it
> can be hacked right now, but I also listed the benefits of making it a
> proper feature:
>   1. It would be very neat and tidy. Clarity and readability is
> important for code longevity.
>   2. Ease of refactoring/delegation to scripts without refactoring all
> the calling code throughout your app (this is real, I have done this on
> many occasions)
>   3. Declaring functions in the regular way, and annotating them with
> various properties (const/pure/etc) allows all the usual type safety and
> optimisation potentials available to the language.
>       As far as the compiler is concerned, it is a real and normal
> function call. All surrounding code gen/optimisations are still
> applicable, the language will simply substitute the marshalling function
> at the final step of the call.
>   4-ish. It actually seems relatively simple to implement... and it's
> not a breaking change. I can't really see any down sides.

On 2012-02-21 17:03:56 +0000, Manu <turkeyman@gmail.com> said:

> What was the primary cause of the bloat? I can't imagine my proposal
> causing any more bloat than the explicit jni call (or equivalent) woudl
> have otherwise.

I had one D class for every corresponding Objective-C class, each method wrapped in both direction, each wrapper doing automatic mapping of D and Objective-C wrapper objects and translating exceptions. I could actually override a method in a derived class of the D wrapper and it'd effectively override the method on the Objective-C side. But with all those D classes having virtual tables referring to all these wrapper functions, even if you use only 1% of all the bindings nothing can be stripped of the final executable.

Quoting from a blog post of mine:
"""
But the real show stopper is the size of the generated code for the bindings. The system I created needs to generate a lot of code for each and every one of the functions and classes you define bindings for. This makes the final executable awfully big (and might also explain in part the slowness while compiling). The small test application included with the bridge takes about 60 Mb of code, thats for something that’d be only a few kilobytes when written in Objective-C.

This wasn’t the case at first. This bloat appeared as I added bindings for more and more classes in Cocoa. It is a real problem because no one in his right mind will want to ship a 60 Mb application that contains so much binding code that stay unused most of the time. A solution could be that everyone would have to write bindings that only contains the classes and methods he needs, but this is of rather limited interest.
"""

>> Maybe you'll have more luck, but for me wrappers just couldn't work. So I
>> decided making the compiler understand the Objective-C runtime and object
>> model was a much better solution. It was fun to do and wasn't that hard
>> actually. Here's the result: <http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.
Unfortunately 
>> 
>> I don't have much time to work on it anymore.
> 
> That sounds appropriate for Obj-C, but I don't think it applies to VM based
> languages however you slice it, since there's not actually a direct binary
> linkage. There's no way to know about the Java/.NET/Lua/etc object model
> internally, the VM owns that entirely, so you're forced to marshal through
> API's.

That's not actually very different from what I'm doing for Objective-C. Call to Objective-C methods are all done through low-level API functions in the Objective-C runtime. Class definitions are static data, but could be done dynamically through API calls in module constructors.

The most interesting bit however is that class *declarations* require nothing to be compiled, the compiler will just emit the right code when calling a method, using objc_msgSend(). This has a major impact on compilation time if you have thousands of such functions.


> That's okay though, you have to do it anyway. I'm proposing a way to
> unify the concept of calling functions without obscuring the actual call
> with junk, and still allowing the D language to annotate and optimise to
> its fullest extent.

The goal is certainly worthwhile. I think your proposal has merit if you goal is just to call functions and not define function or classes, although I'm skeptical overloading extern() is the right way to do that (perhaps a pragma?). But I'd suggest you define your goals more throughly before deciding on an implementation.

Some questions you should try to have an answer for regarding OO: if you're interacting with classes, how should it work in an ideal world? Then, do you need to define new classes that'd be available in the foreign language? Do you need to override functions on existing classes? How do you want code using foreign objects to look? Then ask yourself what would be an acceptable compromise, and start implementing it.

For my part, I came to the conclusion that you couldn't really use Objective-C APIs in a meaningful way without the ability to create subclasses and override functions… which led me to the current design where you can use Objective-C classes as if they were D classes.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

On 21 February 2012 19:10, Alex Rønne Petersen <xtzgzorex@gmail.com> wrote:

> On 21-02-2012 18:03, Manu wrote:
>
>> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com <mailto:michel.fortin@michelf.**com <michel.fortin@michelf.com>>> wrote:
>>
>>    On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com
>>    <mailto:turkeyman@gmail.com>> said:
>>
>> Interesting point.
>> Java (or any VM based language) can't hard link to the binary, so it
>> never could try and call back in through an extern-ed function directly,
>>
>
> Sure it could. Why wouldn't it be able to? This is how extern + DllImport works in C#.
>

But that only works because C# has DLL support, which knows about the C calling convention, so you just need to export an extern(C) in that case, like any regular DLL.