View mode: basic / threaded / horizontal-split · Log in · Help
August 18, 2012
Re: Fragile ABI
On 8/18/12, Michel Fortin <michel.fortin@michelf.ca> wrote:
> But for that you'd need a custom linker
> (both static and dynamic), and probably a custom shared library format.
> So it's a huge task, especially when you consider that it should run on
> multiple platforms.

Isn't this what the DDL project was all about? I don't know what state
it was left in though..
August 18, 2012
Re: Fragile ABI
On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:
> On 2012-08-17 20:49:21 +0000, "Paulo Pinto" 
> <pjmlp@progtools.org> said:
>
>> On Friday, 17 August 2012 at 17:42:24 UTC, Michel Fortin wrote:
>>> Quote from <http://xenophilia.org/winvunix.html>:
>>> 
>>>> In Unix, a shared object (.so) file contains code to be used 
>>>> by the program, and also the names of functions and data 
>>>> that it expects to find in the program. When the file is 
>>>> joined to the program, all references to those functions and 
>>>> data in the file's code are changed to point to the actual 
>>>> locations in the program where the functions and data are 
>>>> placed in memory. This is basically a link operation.
>>>> In Windows, a dynamic-link library (.dll) file has no 
>>>> dangling references. Instead, an access to functions or data 
>>>> goes through a lookup table. So the DLL code does not have 
>>>> to be fixed up at runtime to refer to the program's memory; 
>>>> instead, the code already uses the DLL's lookup table, and 
>>>> the lookup table is modified at runtime to point to the 
>>>> functions and data.
>> 
>> For this just looks at two different ways to implement dynamic 
>> loading.
>> 
>> Having Windows experience since the Windows 3.1 days, 
>> alongside with
>> many other OS besides the typical Linux fan, I always find sad 
>> this vision
>> of Linux == UNIX, right, Windows wrong.
>
> I never intended to mean Windows == wrong. What I am saying is 
> that Windows does not have a dynamic linker capable of changing 
> pointers to symbols in the code it loads (which is what a 
> linker does). Code compiled for Windows need instead to refer 
> to a separate lookup table when accessing symbols from others 
> DLLs. Neither approach is wrong; each has different tradeoffs.
>
> Implementing a non-fragile ABI would be possible using a lookup 
> table of runtime calculated values for field and vtable 
> offsets, and those could be implemented both through a dynamic 
> linker with DLLs, but accessing the lookup table adds some 
> runtime overhead each time you need to access this symbol in 
> the code because of the indirection.
>
> In the original linked article (which I wrote) what was 
> proposed was to have the dynamic linker calculate offsets for 
> fields and vtable entries and insert those offsets directly in 
> the code (just like a linker does when it resolves symbols). 
> But for that you'd need a custom linker (both static and 
> dynamic), and probably a custom shared library format. So it's 
> a huge task, especially when you consider that it should run on 
> multiple platforms. But this same approach could make the C++ 
> ABI non-fragile too.

Wouldn't DLL redirection with Toolhelp32 help here?

But you're right the main problem is the portability across OS,
specially since most tend to have a C like linker, instead of a 
more
intelligent one.

This is can only be improved with higher level linkers, which is a
bit what happens with the solutions where the binaries have 
bytecode
which is only compiled at load time, like .NET and JVM in their 
default offerings. Or even older Lillith system, or Native Oberon 
with bytecode modules.

So, a solution would be either something as you describe, or 
having everyone use a D based OS. :)

--
Paulo
August 18, 2012
Re: Fragile ABI
On Friday, 17 August 2012 at 17:42:18 UTC, Michel Fortin wrote:

> Yes. Neither druntime nor the compiler knows whether you're 
> creating an executable. The typical compilation process is to 
> convert D code to an object file containing machine code. Then 
> that object file is linked either inside an executable or 
> inside a shared library. But the machine code is written before 
> the compiler knows where it'll go.

In most cases that is true. But in some cases the compiler do 
know what will be compiled. DMD has the -shared and -lib flags. 
They indicate that a shared library or static library should be 
built.

--
/Jacob Carlborg
August 18, 2012
Re: Fragile ABI
On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:

> In the original linked article (which I wrote) what was 
> proposed was to have the dynamic linker calculate offsets for 
> fields and vtable entries and insert those offsets directly in 
> the code (just like a linker does when it resolves symbols). 
> But for that you'd need a custom linker (both static and 
> dynamic), and probably a custom shared library format. So it's 
> a huge task, especially when you consider that it should run on 
> multiple platforms. But this same approach could make the C++ 
> ABI non-fragile too.

I'm having a hard time to see why a regular application couldn't 
do this, i.e. druntime. I'm mostly familiar with Mac OS X and 
seems pretty easy just to access the running executable and 
change what you want in it. That's what the dynamic linker is 
doing anyway. There's even a flag for object files indicating 
it's a dynamic linker (don't know if that is used any more). Sure 
it would probably break easily if the runtime of the OS changed 
(new version of the dynamic linker, something changing the object 
format).

--
/Jacob Carlborg
August 18, 2012
Re: Fragile ABI
On 2012-08-18 05:29:18 +0000, "Paulo Pinto" <pjmlp@progtools.org> said:

> Wouldn't DLL redirection with Toolhelp32 help here?

No idea what toolhelp32 is.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/
August 18, 2012
Re: Fragile ABI
On 2012-08-18 10:20:17 +0000, "Jacob Carlborg" <doob@me.com> said:

> On Saturday, 18 August 2012 at 01:06:10 UTC, Michel Fortin wrote:
> 
>> In the original linked article (which I wrote) what was proposed was to 
>> have the dynamic linker calculate offsets for fields and vtable entries 
>> and insert those offsets directly in the code (just like a linker does 
>> when it resolves symbols). But for that you'd need a custom linker 
>> (both static and dynamic), and probably a custom shared library format. 
>> So it's a huge task, especially when you consider that it should run on 
>> multiple platforms. But this same approach could make the C++ ABI 
>> non-fragile too.
> 
> I'm having a hard time to see why a regular application couldn't do 
> this, i.e. druntime. I'm mostly familiar with Mac OS X and seems pretty 
> easy just to access the running executable and change what you want in 
> it. That's what the dynamic linker is doing anyway. There's even a flag 
> for object files indicating it's a dynamic linker (don't know if that 
> is used any more). Sure it would probably break easily if the runtime 
> of the OS changed (new version of the dynamic linker, something 
> changing the object format).

Using a lookup table it could be done.

But if you're going to patch the code as a dynamic linker does but 
after the dynamic linking stage, then you'll have to play around with 
no-execute flags as well as address layout randomization, and this is 
going to be ugly.

Speaking of OS X, if your app is sandboxed I think it won't be able to 
do anything like that. Given that sandboxing is the beginning of a 
trend on many platforms, I'm not sure implementing all that would be 
worthwhile: all it'd accomplish is make processes that can't be 
sandboxed run a little faster.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/
August 18, 2012
Re: Fragile ABI
Hey Michel, posted this question earlier in the thread, but I 
think you might have missed it:

> The blog post I linked only talks about rewriting vtables at 
> dynamic-link-time. Would it be possible to implement something 
> similar for the sizes of structs and scope-classes, and the 
> relative address of member variables? dsimcha's post has led me 
> to realize that without these additional features, the ABI 
> would still be quite fragile.

Long story short: Could your proposal realistically be extended 
to support flexible sizeofs and member-variable offsets? I figure 
that such a system might as well stabilize the entire class ABI, 
rather than just vtables.

I would guess that flexible sizeofs would increase the runtime 
cost only very slightly, but flexible member-variables might be 
more of an issue. The cost could be minimized by looking up 
offsets only when crossing a library boundary (say, when calling 
a template method on an extern (D_FlexibleABI) class, or 
accessing a protected member variable of same), but would that 
actually be possible?
August 18, 2012
Re: Fragile ABI
On 2012-08-18 14:50:54 +0000, "R Grocott" <rgrocottbugzilla@gmail.com> said:

> Hey Michel, posted this question earlier in the thread, but I think you 
> might have missed it:
> 
>> The blog post I linked only talks about rewriting vtables at 
>> dynamic-link-time. Would it be possible to implement something similar 
>> for the sizes of structs and scope-classes, and the relative address of 
>> member variables? dsimcha's post has led me to realize that without 
>> these additional features, the ABI would still be quite fragile.
> 
> Long story short: Could your proposal realistically be extended to 
> support flexible sizeofs and member-variable offsets? I figure that 
> such a system might as well stabilize the entire class ABI, rather than 
> just vtables.
> 
> I would guess that flexible sizeofs would increase the runtime cost 
> only very slightly, but flexible member-variables might be more of an 
> issue. The cost could be minimized by looking up offsets only when 
> crossing a library boundary (say, when calling a template method on an 
> extern (D_FlexibleABI) class, or accessing a protected member variable 
> of same), but would that actually be possible?

Apple's Modern Objective-C runtime has a non-fragile ABI that extends 
to member variables which works very well. It could be implemented 
pretty much the same way in D. There would a very small runtime 
overhead for accessing member variables though: each time you access a 
member you need need first to read a runtime-initialized global 
variable giving you its offset.

As for structs, you could do it the same way, but I don't think you'd 
get enough benefit to compensate the drawback in performance and 
elsewhere. To truly have non-fragile structs you'd need to disable 
almost all compile-time introspection too. That would be very 
disruptive.

So I don't think it makes sense to have non-fragile structs.

By the way, did you take a look at the benchmarks?
<http://michelf.ca/blog/2009/vtable-benchmarking/>

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/
August 18, 2012
Re: Fragile ABI
On Saturday, 18 August 2012 at 17:55:22 UTC, Michel Fortin wrote:
> As for structs, you could do it the same way, but I don't think 
> you'd get enough benefit to compensate the drawback in 
> performance and elsewhere. To truly have non-fragile structs 
> you'd need to disable almost all compile-time introspection 
> too. That would be very disruptive.
>
> So I don't think it makes sense to have non-fragile structs.

Can't D allocate classes on the stack, in some circumstances? 
Would this lead to the same problems that you just described, or 
is the problem caused by how structs and classes are used, rather 
than how they're implemented?

> By the way, did you take a look at the benchmarks?
> <http://michelf.ca/blog/2009/vtable-benchmarking/>

Those are some pretty interesting numbers - seems as though the 
overhead is only about 10% - 20% over that of a regular vtable 
dispatch? I would have guessed that it would almost double the 
overhead, but I suppose that most of the overhead of a function 
call comes from saving registers, managing the stack, etc.

The overhead is so small, in fact, that I'd feel quite 
comfortable using it in a very performance-sensitive situation 
(like, say, a rasterizer library), presuming that your benchmarks 
were accurate. Another strong argument in favour of trying to 
have this system implemented :)
August 18, 2012
Re: Fragile ABI
On Thursday, 16 August 2012 at 15:54:12 UTC, Piotr Szturmaj wrote:
> Kagamin wrote:
>> You can also use C++ to develop COM components which have 
>> standardized ABI.
>
> ...only on Windows.

Code compiled with VC has a different vtable layout than code 
compiled with say GCC, so even on Windows there is no standard.
1 2 3 4 5
Top | Discussion index | About this forum | D home