August 18, 2012
On 8/18/12, Michel Fortin <michel.fortin@michelf.ca> wrote:
> But for that you'd need a custom linker
> (both static and dynamic), and probably a custom shared library format.
> So it's a huge task, especially when you consider that it should run on
> multiple platforms.

Isn't this what the DDL project was all about? I don't know what state it was left in though..
August 18, 2012
On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:
> On 2012-08-17 20:49:21 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:
>
>> On Friday, 17 August 2012 at 17:42:24 UTC, Michel Fortin wrote:
>>> Quote from <http://xenophilia.org/winvunix.html>:
>>> 
>>>> In Unix, a shared object (.so) file contains code to be used by the program, and also the names of functions and data that it expects to find in the program. When the file is joined to the program, all references to those functions and data in the file's code are changed to point to the actual locations in the program where the functions and data are placed in memory. This is basically a link operation.
>>>> In Windows, a dynamic-link library (.dll) file has no dangling references. Instead, an access to functions or data goes through a lookup table. So the DLL code does not have to be fixed up at runtime to refer to the program's memory; instead, the code already uses the DLL's lookup table, and the lookup table is modified at runtime to point to the functions and data.
>> 
>> For this just looks at two different ways to implement dynamic loading.
>> 
>> Having Windows experience since the Windows 3.1 days, alongside with
>> many other OS besides the typical Linux fan, I always find sad this vision
>> of Linux == UNIX, right, Windows wrong.
>
> I never intended to mean Windows == wrong. What I am saying is that Windows does not have a dynamic linker capable of changing pointers to symbols in the code it loads (which is what a linker does). Code compiled for Windows need instead to refer to a separate lookup table when accessing symbols from others DLLs. Neither approach is wrong; each has different tradeoffs.
>
> Implementing a non-fragile ABI would be possible using a lookup table of runtime calculated values for field and vtable offsets, and those could be implemented both through a dynamic linker with DLLs, but accessing the lookup table adds some runtime overhead each time you need to access this symbol in the code because of the indirection.
>
> In the original linked article (which I wrote) what was proposed was to have the dynamic linker calculate offsets for fields and vtable entries and insert those offsets directly in the code (just like a linker does when it resolves symbols). But for that you'd need a custom linker (both static and dynamic), and probably a custom shared library format. So it's a huge task, especially when you consider that it should run on multiple platforms. But this same approach could make the C++ ABI non-fragile too.

Wouldn't DLL redirection with Toolhelp32 help here?

But you're right the main problem is the portability across OS,
specially since most tend to have a C like linker, instead of a more
intelligent one.

This is can only be improved with higher level linkers, which is a
bit what happens with the solutions where the binaries have bytecode
which is only compiled at load time, like .NET and JVM in their default offerings. Or even older Lillith system, or Native Oberon with bytecode modules.

So, a solution would be either something as you describe, or having everyone use a D based OS. :)

--
Paulo

August 18, 2012
On Friday, 17 August 2012 at 17:42:18 UTC, Michel Fortin wrote:

> Yes. Neither druntime nor the compiler knows whether you're creating an executable. The typical compilation process is to convert D code to an object file containing machine code. Then that object file is linked either inside an executable or inside a shared library. But the machine code is written before the compiler knows where it'll go.

In most cases that is true. But in some cases the compiler do know what will be compiled. DMD has the -shared and -lib flags. They indicate that a shared library or static library should be built.

--
/Jacob Carlborg
August 18, 2012
On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:

> In the original linked article (which I wrote) what was proposed was to have the dynamic linker calculate offsets for fields and vtable entries and insert those offsets directly in the code (just like a linker does when it resolves symbols). But for that you'd need a custom linker (both static and dynamic), and probably a custom shared library format. So it's a huge task, especially when you consider that it should run on multiple platforms. But this same approach could make the C++ ABI non-fragile too.

I'm having a hard time to see why a regular application couldn't do this, i.e. druntime. I'm mostly familiar with Mac OS X and seems pretty easy just to access the running executable and change what you want in it. That's what the dynamic linker is doing anyway. There's even a flag for object files indicating it's a dynamic linker (don't know if that is used any more). Sure it would probably break easily if the runtime of the OS changed (new version of the dynamic linker, something changing the object format).

--
/Jacob Carlborg


August 18, 2012
On 2012-08-18 05:29:18 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:

> Wouldn't DLL redirection with Toolhelp32 help here?

No idea what toolhelp32 is.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 18, 2012
On 2012-08-18 10:20:17 +0000, "Jacob Carlborg" <doob@me.com> said:

> On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:
> 
>> In the original linked article (which I wrote) what was proposed was to have the dynamic linker calculate offsets for fields and vtable entries and insert those offsets directly in the code (just like a linker does when it resolves symbols). But for that you'd need a custom linker (both static and dynamic), and probably a custom shared library format. So it's a huge task, especially when you consider that it should run on multiple platforms. But this same approach could make the C++ ABI non-fragile too.
> 
> I'm having a hard time to see why a regular application couldn't do this, i.e. druntime. I'm mostly familiar with Mac OS X and seems pretty easy just to access the running executable and change what you want in it. That's what the dynamic linker is doing anyway. There's even a flag for object files indicating it's a dynamic linker (don't know if that is used any more). Sure it would probably break easily if the runtime of the OS changed (new version of the dynamic linker, something changing the object format).

Using a lookup table it could be done.

But if you're going to patch the code as a dynamic linker does but after the dynamic linking stage, then you'll have to play around with no-execute flags as well as address layout randomization, and this is going to be ugly.

Speaking of OS X, if your app is sandboxed I think it won't be able to do anything like that. Given that sandboxing is the beginning of a trend on many platforms, I'm not sure implementing all that would be worthwhile: all it'd accomplish is make processes that can't be sandboxed run a little faster.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 18, 2012
Hey Michel, posted this question earlier in the thread, but I think you might have missed it:

> The blog post I linked only talks about rewriting vtables at dynamic-link-time. Would it be possible to implement something similar for the sizes of structs and scope-classes, and the relative address of member variables? dsimcha's post has led me to realize that without these additional features, the ABI would still be quite fragile.

Long story short: Could your proposal realistically be extended to support flexible sizeofs and member-variable offsets? I figure that such a system might as well stabilize the entire class ABI, rather than just vtables.

I would guess that flexible sizeofs would increase the runtime cost only very slightly, but flexible member-variables might be more of an issue. The cost could be minimized by looking up offsets only when crossing a library boundary (say, when calling a template method on an extern (D_FlexibleABI) class, or accessing a protected member variable of same), but would that actually be possible?
August 18, 2012
On 2012-08-18 14:50:54 +0000, "R Grocott" <rgrocottbugzilla@gmail.com> said:

> Hey Michel, posted this question earlier in the thread, but I think you might have missed it:
> 
>> The blog post I linked only talks about rewriting vtables at dynamic-link-time. Would it be possible to implement something similar for the sizes of structs and scope-classes, and the relative address of member variables? dsimcha's post has led me to realize that without these additional features, the ABI would still be quite fragile.
> 
> Long story short: Could your proposal realistically be extended to support flexible sizeofs and member-variable offsets? I figure that such a system might as well stabilize the entire class ABI, rather than just vtables.
> 
> I would guess that flexible sizeofs would increase the runtime cost only very slightly, but flexible member-variables might be more of an issue. The cost could be minimized by looking up offsets only when crossing a library boundary (say, when calling a template method on an extern (D_FlexibleABI) class, or accessing a protected member variable of same), but would that actually be possible?

Apple's Modern Objective-C runtime has a non-fragile ABI that extends to member variables which works very well. It could be implemented pretty much the same way in D. There would a very small runtime overhead for accessing member variables though: each time you access a member you need need first to read a runtime-initialized global variable giving you its offset.

As for structs, you could do it the same way, but I don't think you'd get enough benefit to compensate the drawback in performance and elsewhere. To truly have non-fragile structs you'd need to disable almost all compile-time introspection too. That would be very disruptive.

So I don't think it makes sense to have non-fragile structs.

By the way, did you take a look at the benchmarks?
<http://michelf.ca/blog/2009/vtable-benchmarking/>

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

August 18, 2012
On Saturday, 18 August 2012 at 17:55:22 UTC, Michel Fortin wrote:
> As for structs, you could do it the same way, but I don't think you'd get enough benefit to compensate the drawback in performance and elsewhere. To truly have non-fragile structs you'd need to disable almost all compile-time introspection too. That would be very disruptive.
>
> So I don't think it makes sense to have non-fragile structs.

Can't D allocate classes on the stack, in some circumstances? Would this lead to the same problems that you just described, or is the problem caused by how structs and classes are used, rather than how they're implemented?

> By the way, did you take a look at the benchmarks?
> <http://michelf.ca/blog/2009/vtable-benchmarking/>

Those are some pretty interesting numbers - seems as though the overhead is only about 10% - 20% over that of a regular vtable dispatch? I would have guessed that it would almost double the overhead, but I suppose that most of the overhead of a function call comes from saving registers, managing the stack, etc.

The overhead is so small, in fact, that I'd feel quite comfortable using it in a very performance-sensitive situation (like, say, a rasterizer library), presuming that your benchmarks were accurate. Another strong argument in favour of trying to have this system implemented :)
August 18, 2012
On Thursday, 16 August 2012 at 15:54:12 UTC, Piotr Szturmaj wrote:
> Kagamin wrote:
>> You can also use C++ to develop COM components which have standardized ABI.
>
> ...only on Windows.

Code compiled with VC has a different vtable layout than code compiled with say GCC, so even on Windows there is no standard.