To make an already long story short, I'm looking for a way to dynamically resolve symbols between dll/so files and their host executable.  This way, one could unify the TypeInfo tree at runtime (and more) to achive a sort of runtime-binding similar to what Java does with .class files.*

With portability in mind, I went about researching how to perform a dll detour (also called a trampoline) in linux using a Shared Object.

The general idea, is to perform a fixup on the dll after its loaded into memory, as to circumvent the original call hook for the library.  I know that ELF shared objects can have external symbols resolved via the ELF loader itself (which searches for matching symbols provided by in-memory tables); which is kind of backwards, but may be workable.

With that in mind, it left me with the following questions:

- Is there an ELF, COFF or OMF loader workable under windows out there in F/OSS
land?
- Is there any way to generate an ELF shared object under windows, as an analog
to a dll file (possibly via a coff to elf converter)?
- ELF loading is grand, but isn't there a guide to hacking away at the shared
object after its been loaded into memory (like what we do in windows with dll
detours)?
- Would anyone like more detail or background on the problem and why custom
library loading is important?

(* Java .class files contain a symbol table of what external symbols are needed to run the class, and what symbols are available to the calling program.  The exhange of this information, or binding is performed by the classloader at runtime when the .class file is loaded.  This is in stark contrast to D or C++ where type information is held statically in a dll/so or executable and is not shared with anything at runtime.)

Thanks in advance,

- EricAnderton at yahoo

"pragma" <pragma_member@pathlink.com> wrote in message news:ddbn9q$143n$1@digitaldaemon.com...
> To make an already long story short, I'm looking for a way to dynamically
> resolve symbols between dll/so files and their host executable.  This way,
> one
> could unify the TypeInfo tree at runtime (and more) to achive a sort of
> runtime-binding similar to what Java does with .class files.*

I was playing around with similar goals with dflect (I think... I'm not exactly sure what your goals are but I'm guessing). The work-flow was as follows: run dflect on the classes/modules you wish to expose (this requires some special coding practices like marking ctors as "export") to generate MetaInfo code. Compile the MetaInfo classes together with your actual classes and make the dll. In the main app load the dll and call the exported function getMetaClassCtors() in the dll which returns an AA indexed by class name and with value the ctor for the MetaInfo class for the class in question. Once you construct the MetaClass you can call it's methods to construct instances and get fields and methods etc etc. The AA mapping class names to ctors is initialized in the module ctors so the main app needs to be able to run the module ctors before getting the AA of MetaInfo.

Dflect also has files for interfacing with C/C++ - like the ability to get a global ref to an object so that it doesn't get garbage collected while some C code owns the only reference.

Dflect is on the back-burner now, though. I was hoping someone would redo std.loader first.

> With portability in mind, I went about researching how to perform a dll
> detour
> (also called a trampoline) in linux using a Shared Object.
>
> The general idea, is to perform a fixup on the dll after its loaded into
> memory,
> as to circumvent the original call hook for the library.  I know that ELF
> shared
> objects can have external symbols resolved via the ELF loader itself
> (which
> searches for matching symbols provided by in-memory tables); which is kind
> of
> backwards, but may be workable.
>
> With that in mind, it left me with the following questions:
>
> - Is there an ELF, COFF or OMF loader workable under windows out there in
> F/OSS
> land?
> - Is there any way to generate an ELF shared object under windows, as an
> analog
> to a dll file (possibly via a coff to elf converter)?
> - ELF loading is grand, but isn't there a guide to hacking away at the
> shared
> object after its been loaded into memory (like what we do in windows with
> dll
> detours)?
> - Would anyone like more detail or background on the problem and why
> custom
> library loading is important?
>
> (* Java .class files contain a symbol table of what external symbols are
> needed
> to run the class, and what symbols are available to the calling program.
> The
> exhange of this information, or binding is performed by the classloader at
> runtime when the .class file is loaded.  This is in stark contrast to D or
> C++
> where type information is held statically in a dll/so or executable and is
> not
> shared with anything at runtime.)

I don't know anything about trampolines so I can't help but it sounds interesting. What exactly would the trampoline do? You mention unify TypeInfos - is this so that, for example, int's TypeInfo is the same address in all the dlls and main executable? Can this stuff be built into std.loader?

> Thanks in advance,
>
> - EricAnderton at yahoo

In article <ddbrve$1bkm$1@digitaldaemon.com>, Ben Hinkle says...
>
>
>"pragma" <pragma_member@pathlink.com> wrote in message news:ddbn9q$143n$1@digitaldaemon.com...
>> To make an already long story short, I'm looking for a way to dynamically
>> resolve symbols between dll/so files and their host executable.  This way,
>> one
>> could unify the TypeInfo tree at runtime (and more) to achive a sort of
>> runtime-binding similar to what Java does with .class files.*
>
>I was playing around with similar goals with dflect (I think... I'm not exactly sure what your goals are but I'm guessing). The work-flow was as follows: run dflect on the classes/modules you wish to expose (this requires some special coding practices like marking ctors as "export") to generate MetaInfo code. Compile the MetaInfo classes together with your actual classes and make the dll. In the main app load the dll and call the exported function getMetaClassCtors() in the dll which returns an AA indexed by class name and with value the ctor for the MetaInfo class for the class in question. Once you construct the MetaClass you can call it's methods to construct instances and get fields and methods etc etc. The AA mapping class names to ctors is initialized in the module ctors so the main app needs to be able to run the module ctors before getting the AA of MetaInfo.
>
>Dflect also has files for interfacing with C/C++ - like the ability to get a global ref to an object so that it doesn't get garbage collected while some C code owns the only reference.
>
>Dflect is on the back-burner now, though. I was hoping someone would redo std.loader first.

Excellent.  That sounds like a worthwhile goal.  I've been very keen on deflect since you first introduced it, and I'll be eager to see where this goes.


>> <snip my discussion about trampolines and ELF loaders>
>I don't know anything about trampolines so I can't help but it sounds interesting. What exactly would the trampoline do?

Basically the concept is to load the dll into memory, treat it as data, rewrite portions of it and then treat it as you normally would.  A trampoline, also known as a "detour" is explained here:

http://research.microsoft.com/~galenh/Publications/HuntUsenixNt99.pdf

>You mention unify TypeInfos - is this so that, for example, int's TypeInfo is the same address in all the dlls and main executable? Can this stuff be built into std.loader?

You hit the nail right on the head.  Although I'd like to see it extended to all TypeInfo classes, standard and custom.  In a perfect world, yes, it'd be written directly into std.loader or something similar.

Such a custom loader would take advantage of the concepts outlined by Detours/Trampolines, so that it may overwrite pointers in the in-memory-dll so that they point to the 'correct' values.  The catch is: how do we coerce a dll to create such a table of all in-code references to TypeInfo structures?

Its either that, or somehow rewrite DMD to output code that changes indirect
references to TypeInfo (say in typeid() and cast() calls) to instead call
"getTypeinfo(foo)", which would then use a lookup table.  The table could be
overwritten on load to the correct set of values.

FYI, I'm curious about custom loading COFF or OMF directly, simply to bypass all the cruft that win32 places on dlls in general.  It be nice since these formats (by definition) already have all of their external references exposed for linking;  we'd just be doing it at run-time instead of at compile-time.  As a side-effect, it would do more than helping the current TypeInfo problem as it would naturally extend to every kind of symbol.  (AFAIK, this is like emulating ELF loading under win32)

- EricAnderton at yahoo

"pragma" <pragma_member@pathlink.com> wrote in message

> To make an already long story short, I'm looking for a way to dynamically resolve symbols between dll/so files and their host executable.

http://edll.sourceforge.net/index.html

In article <ddcpuo$2m16$1@digitaldaemon.com>, Carlos Smith says...
>
>"pragma" <pragma_member@pathlink.com> wrote in message
>
>> To make an already long story short, I'm looking for a way to dynamically resolve symbols between dll/so files and their host executable.
>
>http://edll.sourceforge.net/index.html
>

Carlos, you are my hero.  That's one heck of a starting point, even if it depends on the GCC/MinGW toolchain.

- EricAnderton at yahoo

"pragma" <pragma_member@pathlink.com> wrote in message

> Carlos, you are my hero.  That's one heck of a starting point, even if it depends on the GCC/MinGW toolchain.

for( i=1.0 ; i > 0 ; i += 1.0 )
    LOL();

The greatest things about Free/OpenSource software, are:

It promote/encourage collaboration instead of competition,

It's "Un pour Tous, Tous pour Un", instead of "We will beat them all",

It's the realisation of "L'Union fait la Force",

It's a great way to make friends,

And, when Bill Gates will try this, humanity will enter
a new world...

In article <ddd5h5$453$1@digitaldaemon.com>, Carlos Smith says...
>
>
>"pragma" <pragma_member@pathlink.com> wrote in message
>
>> Carlos, you are my hero.  That's one heck of a starting point, even if it depends on the GCC/MinGW toolchain.
>
>for( i=1.0 ; i > 0 ; i += 1.0 )
>    LOL();

You're welcome.

>
>The greatest things about Free/OpenSource software, are:
>
>It promote/encourage collaboration instead of competition,
>
>It's "Un pour Tous, Tous pour Un", instead of "We will beat them all",
>
>It's the realisation of "L'Union fait la Force",
>
>It's a great way to make friends,
>
>And, when Bill Gates will try this, humanity will enter
>a new world...
>

I couldn't have said it better myself.

In my experience, when I can't just google for answers it means that either it's right under my nose or its rediculously obscure and possibly offline someplace.


That's why I posted the question here since my research was going nowhere fast. If I could work with someone, drum up interest in a new project or add to an existing project in the process: all the better.

Also, I'm long overdue for posting this to the DNG as I've been writing about it on my DSP project board for a while now.

Anyway edll will give me a much needed foothold: it uses BFD (object file loader library for linux) under windows.  I'm sure that it can be ported to DMC/DMD, but it lacks OMF support.  Otherwise, I'll have to find a way to convert OMF to COFF or OMF to ELF.

So, is anyone out there willing to donate an OMF loader before I try to write one? ;)

- EricAnderton at yahoo