February 03, 2012
As time goes by the D runtime will have it's place on Unix systems next to the C runtime. The upcoming support for PIC is the next step. Now I just want to quickly raise awareness for "sonames". For any library, there may be several incompatible versions installed, since new features are added and old features a deprecated. This happens in Phobos as well, no doubt :). So the solution is to put the version in the library name as well, so they can coexist in the same directory (/usr/lib).

Here is an example for the two versions of libunique I have installed:
	/usr/lib64/libunique-1.0.so         (link to /usr/lib64/libunique-1.0.so.0.100.6)
	/usr/lib64/libunique-1.0.so.0       (link to /usr/lib64/libunique-1.0.so.0.100.6)
	/usr/lib64/libunique-1.0.so.0.100.6
	/usr/lib64/libunique-3.0.so         (link to /usr/lib64/libunique-3.0.so.0.0.2)
	/usr/lib64/libunique-3.0.so.0       (link to /usr/lib64/libunique-3.0.so.0.0.2)
	/usr/lib64/libunique-3.0.so.0.0.2
As you can see there is actually the full version down to the tiniest minor version appended to the file name and several layers of coarser versioning. An application can now link against libunique-1.0.so to get the old API and /usr/lib64/libunique-3.0.so to get the new API.

The same has to happen with druntime and Phobos2 or otherwise our programs will break with every new release that deprecates or changes non-template functions. That would probably be *every* release at the moment, so it could look like this:
/usr/lib64/libphobos2.so     (link to /usr/lib64/libphobos2.so.060)
/usr/lib64/libphobos2.so.058
/usr/lib64/libphobos2.so.059
/usr/lib64/libphobos2.so.060
/usr/lib64/libdruntime.so     (link to /usr/lib64/libdruntime.so.060)
/usr/lib64/libdruntime.so.058
/usr/lib64/libdruntime.so.059
/usr/lib64/libdruntime.so.060

There are two steps involved in getting this out of the door now:
1) I'm not an expert with these things, but from the looks of it, I think the Makefile should handle appending the version string
2) The runtime should be downloadable as a separate package (like the famous MSVC Runtime Redistributables)
Developers have three choices then:
- static linking
- packaging the so/dll with their application (always using the tested-works version)
- use the system installation of druntime/Phobos2 (benefit from patches (as far as WinSxS doesn't intervene))

-- Marco
February 03, 2012
On Friday, February 03, 2012 02:29:20 Marco Leise wrote:
> As time goes by the D runtime will have it's place on Unix systems next to the C runtime. The upcoming support for PIC is the next step. Now I just want to quickly raise awareness for "sonames". For any library, there may be several incompatible versions installed, since new features are added and old features a deprecated. This happens in Phobos as well, no doubt
> 
> :). So the solution is to put the version in the library name as well, so
> 
> they can coexist in the same directory (/usr/lib).
> 
> Here is an example for the two versions of libunique I have installed:
> 	/usr/lib64/libunique-1.0.so         (link to
> /usr/lib64/libunique-1.0.so.0.100.6)
> 	/usr/lib64/libunique-1.0.so.0       (link to
> /usr/lib64/libunique-1.0.so.0.100.6)
> 	/usr/lib64/libunique-1.0.so.0.100.6
> 	/usr/lib64/libunique-3.0.so         (link to
> /usr/lib64/libunique-3.0.so.0.0.2)
> 	/usr/lib64/libunique-3.0.so.0       (link to
> /usr/lib64/libunique-3.0.so.0.0.2)
> 	/usr/lib64/libunique-3.0.so.0.0.2
> As you can see there is actually the full version down to the tiniest
> minor version appended to the file name and several layers of coarser
> versioning. An application can now link against libunique-1.0.so to get
> the old API and /usr/lib64/libunique-3.0.so to get the new API.
> 
> The same has to happen with druntime and Phobos2 or otherwise our programs
> will break with every new release that deprecates or changes non-template
> functions. That would probably be *every* release at the moment, so it
> could look like this:
> /usr/lib64/libphobos2.so     (link to /usr/lib64/libphobos2.so.060)
> /usr/lib64/libphobos2.so.058
> /usr/lib64/libphobos2.so.059
> /usr/lib64/libphobos2.so.060
> /usr/lib64/libdruntime.so     (link to /usr/lib64/libdruntime.so.060)
> /usr/lib64/libdruntime.so.058
> /usr/lib64/libdruntime.so.059
> /usr/lib64/libdruntime.so.060
> 
> There are two steps involved in getting this out of the door now:
> 1) I'm not an expert with these things, but from the looks of it, I think
> the Makefile should handle appending the version string
> 2) The runtime should be downloadable as a separate package (like the
> famous MSVC Runtime Redistributables)
> Developers have three choices then:
> - static linking
> - packaging the so/dll with their application (always using the
> tested-works version)
> - use the system installation of druntime/Phobos2 (benefit from patches
> (as far as WinSxS doesn't intervene))

I would point out that druntime is bundled with libphobos.a. I wouldn't expect libdruntime to be on the system.

- Jonathan M Davis
February 03, 2012
On Thu, Feb 02, 2012 at 05:33:49PM -0800, Jonathan M Davis wrote: [...]
> I would point out that druntime is bundled with libphobos.a. I wouldn't expect libdruntime to be on the system.
[...]

Are there any *good* reasons why druntime and libphobos are not dynamically linked? In the long run, we need to support that, since otherwise D binaries will be unnecessarily large and the OS won't be able to optimize memory usage by sharing library images with multiple processes.


T

-- 
BREAKFAST.COM halted...Cereal Port Not Responding. -- YHL
February 03, 2012
On Thursday, February 02, 2012 18:34:34 H. S. Teoh wrote:
> On Thu, Feb 02, 2012 at 05:33:49PM -0800, Jonathan M Davis wrote: [...]
> 
> > I would point out that druntime is bundled with libphobos.a. I wouldn't expect libdruntime to be on the system.
> 
> [...]
> 
> Are there any *good* reasons why druntime and libphobos are not dynamically linked? In the long run, we need to support that, since otherwise D binaries will be unnecessarily large and the OS won't be able to optimize memory usage by sharing library images with multiple processes.

Because dmd does not yet support shared libraries on Linux.

- Jonathan M Davis
February 03, 2012
Am 03.02.2012, 03:34 Uhr, schrieb H. S. Teoh <hsteoh@quickfur.ath.cx>:

> Are there any *good* reasons why druntime and libphobos are not
> dynamically linked? In the long run, we need to support that, since
> otherwise D binaries will be unnecessarily large and the OS won't be
> able to optimize memory usage by sharing library images with multiple
> processes.
>
>
> T

No fear, the people in charge know about all that, it was technical reasons that held back the support. That said, there are people who prefer static linking. May they speak for themselves...
February 03, 2012
On Friday, February 03, 2012 04:27:37 Marco Leise wrote:
> Am 03.02.2012, 03:34 Uhr, schrieb H. S. Teoh <hsteoh@quickfur.ath.cx>:
> > Are there any *good* reasons why druntime and libphobos are not dynamically linked? In the long run, we need to support that, since otherwise D binaries will be unnecessarily large and the OS won't be able to optimize memory usage by sharing library images with multiple processes.
> > 
> > 
> > T
> 
> No fear, the people in charge know about all that, it was technical reasons that held back the support. That said, there are people who prefer static linking. May they speak for themselves...

Dynamic linking is evil. Static linking is _way_ better when you can do it. The problem is, of course, that you often need dynamic linking for a variety of reasons (saving memory being one of them).

- Jonathan M Davis
February 03, 2012
On Friday, 3 February 2012 at 03:40:07 UTC, Jonathan M Davis wrote:
> Dynamic linking is evil.

Amen!

> dynamic linking for a variety of reasons (saving memory being one of them).

Smart operating systems don't even need it for this;
they can do de-duplication of identical pages in both
memory and on the drive.
February 03, 2012
Am 03.02.2012, 04:42 Uhr, schrieb Adam D. Ruppe <destructionator@gmail.com>:

> On Friday, 3 February 2012 at 03:40:07 UTC, Jonathan M Davis wrote:
>> Dynamic linking is evil.
>
> Amen!
>
>> dynamic linking for a variety of reasons (saving memory being one of them).
>
> Smart operating systems don't even need it for this;
> they can do de-duplication of identical pages in both
> memory and on the drive.

Are you serious? I have seen a flag in Linux for hosting virtual machines, but it must be an enormous overhead to check every executable page or every executable file on the file system for duplicates. If this were for free, it would be great, but the most spread OSes today and in the foreseeable future wont have filesystems that do something like automatic hardlinks of duplicate pages in executables. Both the file system and more importantly executable formats aren't ready for this. Let's imagine the library to link against against was Gtk. Would you want every binary download on the internet to include libraries of that size?
The best we have for the job is dynamic linking. Personally I think it is quite good, but you have to be careful as a library author to properly version API differences.
February 03, 2012
P.S.: Another good example for the benefit of size reduction are filter plugins for multimedia application. Those are usually a dozen small libraries with a few lines of code, where the ratio of own code vs runtime can quickly run out of proportion. I think someone here mentioned writing plugins for some VST program.
February 03, 2012
On Friday, 3 February 2012 at 04:23:34 UTC, Marco Leise wrote:
> I have seen a flag in Linux for hosting virtual machines, but it must be an enormous overhead to check every executable page or every executable file on the file system for duplicates.

tbh I haven't benchmarked it, but one of the cloud
providers I worked on used it on Solaris, and we had
a good experience.

> great, but the most spread OSes today and in the foreseeable future wont have filesystems that do something like automatic hardlinks of duplicate pages in executables.

Regardless, most computers today aren't exactly
counting their kilobytes either. (And those that are
don't multitask much anyway.)

Phobos, in a typical D program, consumes a few hundred
kb. Maybe if you're running 1,000 D processes at once will
it become a problem... but that's not a realistic scenario.

Distributing standalone D programs, or having D programs that
use different versions of Phobos *is* something we face. With
static linking, this isn't a problem. It's mindlessly easy.

> Let's imagine the library to link against against was Gtk. Would you want every binary download on the internet to include libraries of that size?

Yes, actually. I'd rather download a 10 MB executable that
*actually works* than have to download a separate library,
and hope I get the right version.

(And 10 MB is probably an overestimate! I've used statically
linked Qt apps that come in at only about 6 MB.)


There's times when dynamic linking is a good call. Core
operating system libraries, plugins, stuff like that.

But, Phobos isn't one of them, and I doubt it ever will be.
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home