On 21 February 2012 20:12, Jacob Carlborg <doob@me.com> wrote:

> On 2012-02-21 18:03, Manu wrote:
>
>> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
>>    I have some experience bridging Objective-C and D. I once built a
>>    complete wrapper system for Objective-C objects, each object was
>>    wrapped by a D one. It worked very well, but it generated so much
>>    bloat that it became unusable as soon as I started defining enough
>>    classes for it to be useful. See the D/Objective-C bridge:
>>    <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>>    <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>
>> >>.
>>
>>
>> What was the primary cause of the bloat? I can't imagine my proposal causing any more bloat than the explicit jni call (or equivalent) woudl have otherwise.
>>
>
> Template bloat. Every call bridging D/Objective-C is made throw a series of templates. This is for making it possible (less verbose) to create bindings.
>
> It might be possible to decrease the template bloat by having a tool that automatically generates the bindings and outputs what the templates do inline.


Why aren't the templates inline themselves? Although if the templates do a
lot of work, wouldn't that INCREASE the code volume?
I can't really imagine how Obj-C linkage could bloat so much, what was
involved? What did you have to do in addition to what a regular Obj-C
function call would have done?

On 2012-02-21 19:45:37 +0000, Manu <turkeyman@gmail.com> said:

> On 21 February 2012 20:12, Jacob Carlborg <doob@me.com> wrote:
> 
>> On 2012-02-21 18:03, Manu wrote:
>> 
>>> On 21 February 2012 16:59, Michel Fortin <michel.fortin@michelf.com
>>> What was the primary cause of the bloat? I can't imagine my proposal
>>> causing any more bloat than the explicit jni call (or equivalent) woudl
>>> have otherwise.
>> 
>> Template bloat. Every call bridging D/Objective-C is made throw a series
>> of templates. This is for making it possible (less verbose) to create
>> bindings.
>> 
>> It might be possible to decrease the template bloat by having a tool that
>> automatically generates the bindings and outputs what the templates do
>> inline.
> 
> Why aren't the templates inline themselves? Although if the templates do a
> lot of work, wouldn't that INCREASE the code volume?
> I can't really imagine how Obj-C linkage could bloat so much, what was
> involved? What did you have to do in addition to what a regular Obj-C
> function call would have done?

I answered in my other post. In short: virtual functions you can override to override Objective-C methods.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

On 21-02-2012 20:43, Manu wrote:
> On 21 February 2012 19:10, Alex Rønne Petersen <xtzgzorex@gmail.com
> <mailto:xtzgzorex@gmail.com>> wrote:
>
>     On 21-02-2012 18:03, Manu wrote:
>
>         On 21 February 2012 16:59, Michel Fortin
>         <michel.fortin@michelf.com <mailto:michel.fortin@michelf.com>
>         <mailto:michel.fortin@michelf.__com
>         <mailto:michel.fortin@michelf.com>>> wrote:
>
>             On 2012-02-21 11:03:09 +0000, Manu <turkeyman@gmail.com
>         <mailto:turkeyman@gmail.com>
>         <mailto:turkeyman@gmail.com <mailto:turkeyman@gmail.com>>> said:
>
>         Interesting point.
>         Java (or any VM based language) can't hard link to the binary, so it
>         never could try and call back in through an extern-ed function
>         directly,
>
>
>     Sure it could. Why wouldn't it be able to? This is how extern +
>     DllImport works in C#.
>
>
> But that only works because C# has DLL support, which knows about the C
> calling convention, so you just need to export an extern(C) in that
> case, like any regular DLL.

Surely Java 'native' methods require the native functions to conform to some calling convention?

-- 
- Alex

On 21 February 2012 21:03, Paulo Pinto <pjmlp@progtools.org> wrote:

> I think this fails in the same trap as extern "language" in C++.
>
> Besides C and C++, I think the only variation I saw so far for the language part has been fortran, and cannot recall anylonger in what compiler it was.
>
> In C++'s case, since it is implementation defined, most compiler vendors did not bother to support more than what the standard specifies (C and C++).
>
> In you proposal, how to keep the feature portable across D implementations?
>

Well since the languages I'm using as examples are VM based, the interaction is through a standardised API, which is most likely part of the same module that defines the custom extern type, both would be presented in the same package.


If you make it part of D language specification, which languages would be
> the chosen ones to be made available in any D compiler?
>

I'm not making any language part of the D specification, I'm suggesting a system to integrate custom calling code, which can be introduced in libraries. No intrinsic support in D for any particular language.


How to garantee correct interoperability (ABI) with other languages? After
> deciding the set of chosen languages, which compilers/runtimes would be the lucky ones?
>

VM based languages have a strictly defined API, this is guaranteed, no
problem.
Extern to hard-linking languages requires the extra bits I detailed in a
prior post (name mangler, call receiver), and probably some version() mess
to support different compilers just like C/C++ does all over the place.
Many C compilers can't link against each others binaries, but with a mature
lib, you'd be able to support the calling convention of popular compilers
for your language of interest I would think.
t's probably quite do-able, but I'm not proposing to take the system that
far initially. Custom calling conventions to integrate with VM's seems more
useful, and much simpler to start with.

On 2012-02-21 20:45, Manu wrote:
> On 21 February 2012 20:12, Jacob Carlborg <doob@me.com
> <mailto:doob@me.com>> wrote:
>
>     On 2012-02-21 18:03, Manu wrote:
>
>         On 21 February 2012 16:59, Michel Fortin
>         <michel.fortin@michelf.com <mailto:michel.fortin@michelf.com>
>             I have some experience bridging Objective-C and D. I once
>         built a
>             complete wrapper system for Objective-C objects, each object was
>             wrapped by a D one. It worked very well, but it generated so
>         much
>             bloat that it became unusable as soon as I started defining
>         enough
>             classes for it to be useful. See the D/Objective-C bridge:
>         <http://michelf.com/projects/____d-objc-bridge/
>         <http://michelf.com/projects/__d-objc-bridge/>
>         <http://michelf.com/projects/__d-objc-bridge/
>         <http://michelf.com/projects/d-objc-bridge/>>>.
>
>
>         What was the primary cause of the bloat? I can't imagine my proposal
>         causing any more bloat than the explicit jni call (or
>         equivalent) woudl
>         have otherwise.
>
>
>     Template bloat. Every call bridging D/Objective-C is made throw a
>     series of templates. This is for making it possible (less verbose)
>     to create bindings.
>
>     It might be possible to decrease the template bloat by having a tool
>     that automatically generates the bindings and outputs what the
>     templates do inline.
>
>
> Why aren't the templates inline themselves? Although if the templates do
> a lot of work, wouldn't that INCREASE the code volume?
> I can't really imagine how Obj-C linkage could bloat so much, what was
> involved? What did you have to do in addition to what a regular Obj-C
> function call would have done?

Michel Fortin explained this better in two of his answers. You can also read the documentation of my implementation, very similar to Michel's:

http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals

But to give a quick example:

class Foo : NSObject
{		
    Foo foo ()
    {
        return invokeObjcSelf!(Foo, "foo");
    }

    Foo bar ()
    {
        return invokeObjcSelf!(Foo, "bar");
    }
}

"invokeObjcSelf" is a template function that calls an Objective-C method. Basically each time "invokeObjcSelf" is called a new instantiation of the template is created and that is put in the symbol table. "invokeObjcSelf" then calls several more template functions making the template bloat increase exponentially.

-- 
/Jacob Carlborg

On 21 February 2012 22:35, Jacob Carlborg <doob@me.com> wrote:

> On 2012-02-21 20:45, Manu wrote:
>
>> On 21 February 2012 20:12, Jacob Carlborg <doob@me.com <mailto:doob@me.com>> wrote:
>>
>>    On 2012-02-21 18:03, Manu wrote:
>>
>>        On 21 February 2012 16:59, Michel Fortin
>>        <michel.fortin@michelf.com <mailto:michel.fortin@michelf.**com<michel.fortin@michelf.com>
>> >
>>
>>            I have some experience bridging Objective-C and D. I once
>>        built a
>>            complete wrapper system for Objective-C objects, each object
>> was
>>            wrapped by a D one. It worked very well, but it generated so
>>        much
>>            bloat that it became unusable as soon as I started defining
>>        enough
>>            classes for it to be useful. See the D/Objective-C bridge:
>>        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
>>        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>> >
>>
>>        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>>        <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>
>> >>>.
>>
>>
>>        What was the primary cause of the bloat? I can't imagine my
>> proposal
>>        causing any more bloat than the explicit jni call (or
>>        equivalent) woudl
>>        have otherwise.
>>
>>
>>    Template bloat. Every call bridging D/Objective-C is made throw a
>>    series of templates. This is for making it possible (less verbose)
>>    to create bindings.
>>
>>    It might be possible to decrease the template bloat by having a tool
>>    that automatically generates the bindings and outputs what the
>>    templates do inline.
>>
>>
>> Why aren't the templates inline themselves? Although if the templates do
>> a lot of work, wouldn't that INCREASE the code volume?
>> I can't really imagine how Obj-C linkage could bloat so much, what was
>> involved? What did you have to do in addition to what a regular Obj-C
>> function call would have done?
>>
>
> Michel Fortin explained this better in two of his answers. You can also read the documentation of my implementation, very similar to Michel's:
>
> http://www.dsource.org/**projects/dstep/wiki/**ObjcBridge/BridgeInternals<http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>
>
> But to give a quick example:
>
> class Foo : NSObject
> {
>    Foo foo ()
>    {
>        return invokeObjcSelf!(Foo, "foo");
>    }
>
>    Foo bar ()
>    {
>        return invokeObjcSelf!(Foo, "bar");
>    }
> }
>
> "invokeObjcSelf" is a template function that calls an Objective-C method. Basically each time "invokeObjcSelf" is called a new instantiation of the template is created and that is put in the symbol table. "invokeObjcSelf" then calls several more template functions making the template bloat increase exponentially.


But they should all be inlined, and the symbol table should be stripped, which shouldn't leave anything in the end other than the inlined function calling code, and in my examples, this will be basically the exact same code that you'd have to write anyway to call through some vm's API...

On 2012-02-21 22:01, Manu wrote:
> On 21 February 2012 22:35, Jacob Carlborg <doob@me.com
> <mailto:doob@me.com>> wrote:
>
>     On 2012-02-21 20:45, Manu wrote:
>
>         On 21 February 2012 20:12, Jacob Carlborg <doob@me.com
>         <mailto:doob@me.com>
>         <mailto:doob@me.com <mailto:doob@me.com>>> wrote:
>
>             On 2012-02-21 18:03, Manu wrote:
>
>                 On 21 February 2012 16:59, Michel Fortin
>         <michel.fortin@michelf.com <mailto:michel.fortin@michelf.com>
>         <mailto:michel.fortin@michelf.__com
>         <mailto:michel.fortin@michelf.com>>
>
>                     I have some experience bridging Objective-C and D. I
>         once
>                 built a
>                     complete wrapper system for Objective-C objects,
>         each object was
>                     wrapped by a D one. It worked very well, but it
>         generated so
>                 much
>                     bloat that it became unusable as soon as I started
>         defining
>                 enough
>                     classes for it to be useful. See the D/Objective-C
>         bridge:
>         <http://michelf.com/projects/______d-objc-bridge/
>         <http://michelf.com/projects/____d-objc-bridge/>
>         <http://michelf.com/projects/____d-objc-bridge/
>         <http://michelf.com/projects/__d-objc-bridge/>>
>
>         <http://michelf.com/projects/____d-objc-bridge/
>         <http://michelf.com/projects/__d-objc-bridge/>
>         <http://michelf.com/projects/__d-objc-bridge/
>         <http://michelf.com/projects/d-objc-bridge/>>>>.
>
>
>                 What was the primary cause of the bloat? I can't imagine
>         my proposal
>                 causing any more bloat than the explicit jni call (or
>                 equivalent) woudl
>                 have otherwise.
>
>
>             Template bloat. Every call bridging D/Objective-C is made
>         throw a
>             series of templates. This is for making it possible (less
>         verbose)
>             to create bindings.
>
>             It might be possible to decrease the template bloat by
>         having a tool
>             that automatically generates the bindings and outputs what the
>             templates do inline.
>
>
>         Why aren't the templates inline themselves? Although if the
>         templates do
>         a lot of work, wouldn't that INCREASE the code volume?
>         I can't really imagine how Obj-C linkage could bloat so much,
>         what was
>         involved? What did you have to do in addition to what a regular
>         Obj-C
>         function call would have done?
>
>
>     Michel Fortin explained this better in two of his answers. You can
>     also read the documentation of my implementation, very similar to
>     Michel's:
>
>     http://www.dsource.org/__projects/dstep/wiki/__ObjcBridge/BridgeInternals
>     <http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>
>
>     But to give a quick example:
>
>     class Foo : NSObject
>     {
>         Foo foo ()
>         {
>             return invokeObjcSelf!(Foo, "foo");
>         }
>
>         Foo bar ()
>         {
>             return invokeObjcSelf!(Foo, "bar");
>         }
>     }
>
>     "invokeObjcSelf" is a template function that calls an Objective-C
>     method. Basically each time "invokeObjcSelf" is called a new
>     instantiation of the template is created and that is put in the
>     symbol table. "invokeObjcSelf" then calls several more template
>     functions making the template bloat increase exponentially.
>
>
> But they should all be inlined, and the symbol table should be stripped,
> which shouldn't leave anything in the end other than the inlined
> function calling code, and in my examples, this will be basically the
> exact same code that you'd have to write anyway to call through some
> vm's API...

Well, that's not what happen with templates.

-- 
/Jacob Carlborg

On 21 February 2012 23:06, Jacob Carlborg <doob@me.com> wrote:

> On 2012-02-21 22:01, Manu wrote:
>
>> On 21 February 2012 22:35, Jacob Carlborg <doob@me.com
>>
>> <mailto:doob@me.com>> wrote:
>>
>>    On 2012-02-21 20:45, Manu wrote:
>>
>>        On 21 February 2012 20:12, Jacob Carlborg <doob@me.com
>>        <mailto:doob@me.com>
>>        <mailto:doob@me.com <mailto:doob@me.com>>> wrote:
>>
>>            On 2012-02-21 18:03, Manu wrote:
>>
>>                On 21 February 2012 16:59, Michel Fortin
>>        <michel.fortin@michelf.com <mailto:michel.fortin@michelf.**com<michel.fortin@michelf.com>
>> >
>>        <mailto:michel.fortin@michelf.**__com
>>
>>        <mailto:michel.fortin@michelf.**com <michel.fortin@michelf.com>>>
>>
>>                    I have some experience bridging Objective-C and D. I
>>        once
>>                built a
>>                    complete wrapper system for Objective-C objects,
>>        each object was
>>                    wrapped by a D one. It worked very well, but it
>>        generated so
>>                much
>>                    bloat that it became unusable as soon as I started
>>        defining
>>                enough
>>                    classes for it to be useful. See the D/Objective-C
>>        bridge:
>>        <http://michelf.com/projects/_**_____d-objc-bridge/<http://michelf.com/projects/______d-objc-bridge/>
>>        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
>> >
>>
>>        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
>>        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>> >>
>>
>>        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
>>        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>> >
>>        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
>>        <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>
>> >>>>.
>>
>>
>>                What was the primary cause of the bloat? I can't imagine
>>        my proposal
>>                causing any more bloat than the explicit jni call (or
>>                equivalent) woudl
>>                have otherwise.
>>
>>
>>            Template bloat. Every call bridging D/Objective-C is made
>>        throw a
>>            series of templates. This is for making it possible (less
>>        verbose)
>>            to create bindings.
>>
>>            It might be possible to decrease the template bloat by
>>        having a tool
>>            that automatically generates the bindings and outputs what the
>>            templates do inline.
>>
>>
>>        Why aren't the templates inline themselves? Although if the
>>        templates do
>>        a lot of work, wouldn't that INCREASE the code volume?
>>        I can't really imagine how Obj-C linkage could bloat so much,
>>        what was
>>        involved? What did you have to do in addition to what a regular
>>        Obj-C
>>        function call would have done?
>>
>>
>>    Michel Fortin explained this better in two of his answers. You can
>>    also read the documentation of my implementation, very similar to
>>    Michel's:
>>
>>    http://www.dsource.org/__**projects/dstep/wiki/__**
>> ObjcBridge/BridgeInternals<http://www.dsource.org/__projects/dstep/wiki/__ObjcBridge/BridgeInternals>
>>
>>    <http://www.dsource.org/**projects/dstep/wiki/**
>> ObjcBridge/BridgeInternals<http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>
>> >
>>
>>    But to give a quick example:
>>
>>    class Foo : NSObject
>>    {
>>        Foo foo ()
>>        {
>>            return invokeObjcSelf!(Foo, "foo");
>>        }
>>
>>        Foo bar ()
>>        {
>>            return invokeObjcSelf!(Foo, "bar");
>>        }
>>    }
>>
>>    "invokeObjcSelf" is a template function that calls an Objective-C
>>    method. Basically each time "invokeObjcSelf" is called a new
>>    instantiation of the template is created and that is put in the
>>    symbol table. "invokeObjcSelf" then calls several more template
>>    functions making the template bloat increase exponentially.
>>
>>
>> But they should all be inlined, and the symbol table should be stripped, which shouldn't leave anything in the end other than the inlined function calling code, and in my examples, this will be basically the exact same code that you'd have to write anyway to call through some vm's API...
>>
>
> Well, that's not what happen with templates.


... really? why?
You've just made me very very scared for my simd module :/

I've had a serious concern about D's lack of a force-inline for quite a while...

On 2012-02-21 23:13:31 +0000, Manu <turkeyman@gmail.com> said:

> 
> On 21 February 2012 23:06, Jacob Carlborg <doob@me.com> wrote:
> 
>> On 2012-02-21 22:01, Manu wrote:
>> 
>>> On 21 February 2012 22:35, Jacob Carlborg <doob@me.com
>>>> But to give a quick example:
>>>> 
>>>> class Foo : NSObject
>>>> {
>>>> Foo foo ()
>>>> {
>>>> return invokeObjcSelf!(Foo, "foo");
>>>> }
>>>> 
>>>> Foo bar ()
>>>> {
>>>> return invokeObjcSelf!(Foo, "bar");
>>>> }
>>>> }
>>>> 
>>>> "invokeObjcSelf" is a template function that calls an Objective-C
>>>> method. Basically each time "invokeObjcSelf" is called a new
>>>> instantiation of the template is created and that is put in the
>>>> symbol table. "invokeObjcSelf" then calls several more template
>>>> functions making the template bloat increase exponentially.
>>> 
>>> But they should all be inlined, and the symbol table should be stripped,
>>> which shouldn't leave anything in the end other than the inlined
>>> function calling code, and in my examples, this will be basically the
>>> exact same code that you'd have to write anyway to call through some
>>> vm's API...
>> 
>> Well, that's not what happen with templates.
> 
> ... really? why?
> You've just made me very very scared for my simd module :/
> 
> I've had a serious concern about D's lack of a force-inline for quite a
> while...

I'm not sure what Jacob is referring to.

But here's what I can tell: Foo's methods can't be stripped because they're referenced by the virtual table. invokeObjcSelf could be inlined in theory, but it probably isn't because it is not that small considering it actually does exception bridging and argument/return value conversions as needed. Also, for each method there's also a trampoline function generated (by a mixin which is not in his example) which is referenced by the corresponding Objective-C reverse wrapper class (which is added dynamically on demand), so the Objective-C side can call the D method if you subclass the D wrapper and override one of its methods.

The problem is that if you write bindings for a significant part of Cocoa, you end up with thousands of those functions, and none can be stripped because they're all referenced somehow because they're needed for dynamic dispatch. Sure, you could inline what is inside the virtual function (invokeObjcSelf), but beside saving a few function prologues I don't expect it'll have a very noticeable effect. Even if you saved 25% of the symbols it won't bring things to a reasonable size or a reasonable compilation time.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

On 2012-02-22 05:38, Michel Fortin wrote:
> On 2012-02-21 23:13:31 +0000, Manu <turkeyman@gmail.com> said:
>
>>
>> On 21 February 2012 23:06, Jacob Carlborg <doob@me.com> wrote:
>>
>>> On 2012-02-21 22:01, Manu wrote:
>>>
>>>> On 21 February 2012 22:35, Jacob Carlborg <doob@me.com
>>>>> But to give a quick example:
>>>>>
>>>>> class Foo : NSObject
>>>>> {
>>>>> Foo foo ()
>>>>> {
>>>>> return invokeObjcSelf!(Foo, "foo");
>>>>> }
>>>>>
>>>>> Foo bar ()
>>>>> {
>>>>> return invokeObjcSelf!(Foo, "bar");
>>>>> }
>>>>> }
>>>>>
>>>>> "invokeObjcSelf" is a template function that calls an Objective-C
>>>>> method. Basically each time "invokeObjcSelf" is called a new
>>>>> instantiation of the template is created and that is put in the
>>>>> symbol table. "invokeObjcSelf" then calls several more template
>>>>> functions making the template bloat increase exponentially.
>>>>
>>>> But they should all be inlined, and the symbol table should be
>>>> stripped,
>>>> which shouldn't leave anything in the end other than the inlined
>>>> function calling code, and in my examples, this will be basically the
>>>> exact same code that you'd have to write anyway to call through some
>>>> vm's API...
>>>
>>> Well, that's not what happen with templates.
>>
>> ... really? why?
>> You've just made me very very scared for my simd module :/
>>
>> I've had a serious concern about D's lack of a force-inline for quite a
>> while...
>
> I'm not sure what Jacob is referring to.

I referring to that every instantiation of a template function is put in the symbol table. If this is an implementation issue with DMD not being able to properly optimize this I don't know. I thought that this was one of the major reason for the executable size when using the Objective-C/D bridge. For example:

void foo () {}
void main () {}

$ dmd -inline -O -release main.d
$ nm main | grep foo

00000001000012b4 T _D4main3fooFZv

Outputs one symbol as expected

---------------------------------------

void foo (T) () {}
void main () {}

$ dmd -inline -O -release main.d
$ nm main | grep foo

Doesn't output anything since the template functions is not instantiated, it won't even be fully compiled (only lexed and perhaps parsed).

--------------------------------------------------------------

void foo (T) () {}
void main ()
{
    foo!(int);
    foo!(char);
}

$ dmd -inline -O -release main.d
$ nm main | grep foo

00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
00000001000012b0 T _D4main10__T3fooTiZ3fooFZv

Outputs two symbols as expected, one for each instantiation.

-- 
/Jacob Carlborg