August 17, 2012
Paulo -

Surely there are ways to work around the OS's native ABI, though, even from within compiled code?

Rewriting object vtables (etc.) might be a bit more involved than simple name mangling, but I'm almost certain it would be possible to implement it as part of druntime. The blog post which I linked describes one way to go about it.

Michael -

As far as I can tell, most of the performance cost in your idea comes from the additional level of indirection, since you store vtable offsets in a global variable which must be read before each function call.

This might just be wishful thinking, but would it be possible for druntime to write new vtable offsets directly into the program's machine code, at dynamic-link time? That would remove all of the run-time performance overhead, but, as I say, I'm not sure whether it's actually possible.

I think the biggest obstacle is that it would require the .text section (or equivalent) of the executable to be writable. I know that's possible on Linux, but I'm not sure whether the same is true for most other operating systems. I guess it might also be a security risk?
August 17, 2012
On 2012-08-17 12:36:56 +0000, "R Grocott" <rgrocottbugzilla@gmail.com> said:

> Michael -
> 
> As far as I can tell, most of the performance cost in your idea comes from the additional level of indirection, since you store vtable offsets in a global variable which must be read before each function call.
> 
> This might just be wishful thinking, but would it be possible for druntime to write new vtable offsets directly into the program's machine code, at dynamic-link time? That would remove all of the run-time performance overhead, but, as I say, I'm not sure whether it's actually possible.
> 
> I think the biggest obstacle is that it would require the .text section (or equivalent) of the executable to be writable. I know that's possible on Linux, but I'm not sure whether the same is true for most other operating systems. I guess it might also be a security risk?

It's certainly doable if you put it at the right place. But druntime is not the one in charge of dynamic linking: the dynamic linker from the OS is, so that's the ideal place to do such a thing. If you want to do it in druntime it'll be a huge hassle, if doable at all it'll probably break easily. But on the OS side you hit a rock because of different approach to dynamic linking (Windows does not really have a dynamic linker).

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 17, 2012
On Friday, 17 August 2012 at 12:36:57 UTC, R Grocott wrote:
> Paulo -
>
> Surely there are ways to work around the OS's native ABI, though, even from within compiled code?
>
> Rewriting object vtables (etc.) might be a bit more involved than simple name mangling, but I'm almost certain it would be possible to implement it as part of druntime. The blog post which I linked describes one way to go about it.
>
> Michael -
>
> As far as I can tell, most of the performance cost in your idea comes from the additional level of indirection, since you store vtable offsets in a global variable which must be read before each function call.
>
> This might just be wishful thinking, but would it be possible for druntime to write new vtable offsets directly into the program's machine code, at dynamic-link time? That would remove all of the run-time performance overhead, but, as I say, I'm not sure whether it's actually possible.
>
> I think the biggest obstacle is that it would require the .text section (or equivalent) of the executable to be writable. I know that's possible on Linux, but I'm not sure whether the same is true for most other operating systems. I guess it might also be a security risk?

It's possible to modify the vtable at runtime. Just access through .__vtable in ClassInfo (I think) and treat it like a regular array. Don't know if that's what you mean.

--
/Jacob Carlborg
August 17, 2012
On Thursday, 16 August 2012 at 20:19:27 UTC, R Grocott wrote:

> Likewise, WinRT actually hides a huge amount of complexity inside its language bindings. Inheriting from a WinRT object using only the underlying COM interfaces involves a lot of hassle, and is a prime example of what I'm talking about. See here:
>
> http://www.interact-sw.co.uk/iangblog/2011/09/25/native-winrt-inheritance

Someone has written a piece of code making it easy to use WinRT from D. I don't have the link but it should be possible to find in some of the newsgroups.

--
/Jacob Carlborg
August 17, 2012
On Friday, 17 August 2012 at 13:42:27 UTC, Michel Fortin wrote:
> On 2012-08-17 12:36:56 +0000, "R Grocott" [..] (Windows does not really have a dynamic linker).

Why? What is it lacking?

From my experience, I don't have much to complain about it.

Just wanting to know, not trolling.

--
Paulo


August 17, 2012
On Friday, 17 August 2012 at 13:42:27 UTC, Michel Fortin wrote:
> It's certainly doable if you put it at the right place. But druntime is not the one in charge of dynamic linking: the dynamic linker from the OS is, so that's the ideal place to do such a thing. If you want to do it in druntime it'll be a huge hassle, if doable at all it'll probably break easily. But on the OS side you hit a rock because of different approach to dynamic linking (Windows does not really have a dynamic linker).

So basically, any system to create a more sensible ABI needs to built on top of the C ABI, name mangling, editable vtables, and nothing else? Ouch.

If druntime can't detect whether symbols were dynamically or statically linked, does that mean that your system would add an overhead to *all* virtual function calls, not just those which call into a dynamic library? If so, that'd be a pretty strong argument against implementing anything like it, unless it's something which has to be explicitly switched on (using, say, an "extern(D_FlexibleABI)" statement).
August 17, 2012
On 2012-08-17 14:13:10 +0000, "R Grocott" <rgrocottbugzilla@gmail.com> said:

> On Friday, 17 August 2012 at 13:42:27 UTC, Michel Fortin wrote:
>> It's certainly doable if you put it at the right place. But druntime is not the one in charge of dynamic linking: the dynamic linker from the OS is, so that's the ideal place to do such a thing. If you want to do it in druntime it'll be a huge hassle, if doable at all it'll probably break easily. But on the OS side you hit a rock because of different approach to dynamic linking (Windows does not really have a dynamic linker).
> 
> So basically, any system to create a more sensible ABI needs to built on top of the C ABI, name mangling, editable vtables, and nothing else? Ouch.

That's pretty much it yes.

> If druntime can't detect whether symbols were dynamically or statically linked, does that mean that your system would add an overhead to *all* virtual function calls, not just those which call into a dynamic library?

Yes. Neither druntime nor the compiler knows whether you're creating an executable. The typical compilation process is to convert D code to an object file containing machine code. Then that object file is linked either inside an executable or inside a shared library. But the machine code is written before the compiler knows where it'll go.

> If so, that'd be a pretty strong argument against implementing anything like it, unless it's something which has to be explicitly switched on (using, say, an "extern(D_FlexibleABI)" statement).

Exactly my thoughts.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 17, 2012
On 2012-08-17 14:00:42 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:

> On Friday, 17 August 2012 at 13:42:27 UTC, Michel Fortin wrote:
>> On 2012-08-17 12:36:56 +0000, "R Grocott" [..] (Windows does not really have a dynamic linker).
> 
> Why? What is it lacking?
> 
>  From my experience, I don't have much to complain about it.
> 
> Just wanting to know, not trolling.

Quote from <http://xenophilia.org/winvunix.html>:

> In Unix, a shared object (.so) file contains code to be used by the program, and also the names of functions and data that it expects to find in the program. When the file is joined to the program, all references to those functions and data in the file's code are changed to point to the actual locations in the program where the functions and data are placed in memory. This is basically a link operation.
> In Windows, a dynamic-link library (.dll) file has no dangling references. Instead, an access to functions or data goes through a lookup table. So the DLL code does not have to be fixed up at runtime to refer to the program's memory; instead, the code already uses the DLL's lookup table, and the lookup table is modified at runtime to point to the functions and data.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 17, 2012
On Friday, 17 August 2012 at 17:42:24 UTC, Michel Fortin wrote:
> On 2012-08-17 14:00:42 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:
>
>> On Friday, 17 August 2012 at 13:42:27 UTC, Michel Fortin wrote:
>>> On 2012-08-17 12:36:56 +0000, "R Grocott" [..] (Windows does not really have a dynamic linker).
>> 
>> Why? What is it lacking?
>> 
>> From my experience, I don't have much to complain about it.
>> 
>> Just wanting to know, not trolling.
>
> Quote from <http://xenophilia.org/winvunix.html>:
>
>> In Unix, a shared object (.so) file contains code to be used by the program, and also the names of functions and data that it expects to find in the program. When the file is joined to the program, all references to those functions and data in the file's code are changed to point to the actual locations in the program where the functions and data are placed in memory. This is basically a link operation.
>> In Windows, a dynamic-link library (.dll) file has no dangling references. Instead, an access to functions or data goes through a lookup table. So the DLL code does not have to be fixed up at runtime to refer to the program's memory; instead, the code already uses the DLL's lookup table, and the lookup table is modified at runtime to point to the functions and data.

For this just looks at two different ways to implement dynamic loading.

Having Windows experience since the Windows 3.1 days, alongside with
many other OS besides the typical Linux fan, I always find sad this vision
of Linux == UNIX, right, Windows wrong.

The author also recognises that Aix, which is UNIX, also has strange
dynamic loading rules, according to his understanding what dynamic loading
is.

--
Paulo

August 18, 2012
On 2012-08-17 20:49:21 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:

> On Friday, 17 August 2012 at 17:42:24 UTC, Michel Fortin wrote:
>> Quote from <http://xenophilia.org/winvunix.html>:
>> 
>>> In Unix, a shared object (.so) file contains code to be used by the program, and also the names of functions and data that it expects to find in the program. When the file is joined to the program, all references to those functions and data in the file's code are changed to point to the actual locations in the program where the functions and data are placed in memory. This is basically a link operation.
>>> In Windows, a dynamic-link library (.dll) file has no dangling references. Instead, an access to functions or data goes through a lookup table. So the DLL code does not have to be fixed up at runtime to refer to the program's memory; instead, the code already uses the DLL's lookup table, and the lookup table is modified at runtime to point to the functions and data.
> 
> For this just looks at two different ways to implement dynamic loading.
> 
> Having Windows experience since the Windows 3.1 days, alongside with
> many other OS besides the typical Linux fan, I always find sad this vision
> of Linux == UNIX, right, Windows wrong.

I never intended to mean Windows == wrong. What I am saying is that Windows does not have a dynamic linker capable of changing pointers to symbols in the code it loads (which is what a linker does). Code compiled for Windows need instead to refer to a separate lookup table when accessing symbols from others DLLs. Neither approach is wrong; each has different tradeoffs.

Implementing a non-fragile ABI would be possible using a lookup table of runtime calculated values for field and vtable offsets, and those could be implemented both through a dynamic linker with DLLs, but accessing the lookup table adds some runtime overhead each time you need to access this symbol in the code because of the indirection.

In the original linked article (which I wrote) what was proposed was to have the dynamic linker calculate offsets for fields and vtable entries and insert those offsets directly in the code (just like a linker does when it resolves symbols). But for that you'd need a custom linker (both static and dynamic), and probably a custom shared library format. So it's a huge task, especially when you consider that it should run on multiple platforms. But this same approach could make the C++ ABI non-fragile too.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/