On Tuesday, 1 January 2019 at 09:44:40 UTC, Jacob Carlborg wrote:
> On 2018-12-31 16:51, H. S. Teoh wrote:
>> Note the order of magnitude difference in size, and that ldc2 achieves
>> this by default, with no additional options needed.
>> 
>> How do you make dmd produce the same (or comparable) output?
>
> For me I get comparable output with DMD and LDC

You guys are most likely comparing apples to oranges - H. S. using some distro-LDC preconfigured to link against shared druntime/Phobos, while LDC usually defaults to the static libs.

On Tuesday, 1 January 2019 at 11:57:26 UTC, kinke wrote:
> On Tuesday, 1 January 2019 at 09:44:40 UTC, Jacob Carlborg wrote:
>> On 2018-12-31 16:51, H. S. Teoh wrote:
>>> Note the order of magnitude difference in size, and that ldc2 achieves
>>> this by default, with no additional options needed.
>>> 
>>> How do you make dmd produce the same (or comparable) output?
>>
>> For me I get comparable output with DMD and LDC
>
> You guys are most likely comparing apples to oranges - H. S. using some distro-LDC preconfigured to link against shared druntime/Phobos, while LDC usually defaults to the static libs.

Druntime and Phobos can be used as shared libraries? What is this magical feature?

-Windows User

On Tuesday, January 1, 2019 7:04:42 AM MST Rubn via Digitalmars-d wrote:
> On Tuesday, 1 January 2019 at 11:57:26 UTC, kinke wrote:
> > On Tuesday, 1 January 2019 at 09:44:40 UTC, Jacob Carlborg
> >
> > wrote:
> >> On 2018-12-31 16:51, H. S. Teoh wrote:
> >>> Note the order of magnitude difference in size, and that ldc2
> >>> achieves
> >>> this by default, with no additional options needed.
> >>>
> >>> How do you make dmd produce the same (or comparable) output?
> >>
> >> For me I get comparable output with DMD and LDC
> >
> > You guys are most likely comparing apples to oranges - H. S. using some distro-LDC preconfigured to link against shared druntime/Phobos, while LDC usually defaults to the static libs.
>
> Druntime and Phobos can be used as shared libraries? What is this magical feature?
>
> -Windows User

LOL. Personally, it's a feature that I avoid like the plague, because I hate it when my programs stop working just because I updated dmd (which would be bad enough for a normal user, but it's particularly nasty when you're frequently following master and making local changes, because you're working on Phobos, druntime, and/or dmd). But yeah, when I saw the difference in size that H.S. Teoh was seeing, my first thought was that one was probably using Phobos as a shared library while the other was using it as a static one.

- Jonathan M Davis

On Tuesday, 1 January 2019 at 19:28:36 UTC, Jonathan M Davis wrote:
> On Tuesday, January 1, 2019 7:04:42 AM MST Rubn via Digitalmars-d wrote:
>> On Tuesday, 1 January 2019 at 11:57:26 UTC, kinke wrote:
>> > On Tuesday, 1 January 2019 at 09:44:40 UTC, Jacob Carlborg
>> >
>> > wrote:
>> >> On 2018-12-31 16:51, H. S. Teoh wrote:
>> >>> Note the order of magnitude difference in size, and that ldc2
>> >>> achieves
>> >>> this by default, with no additional options needed.
>> >>>
>> >>> How do you make dmd produce the same (or comparable) output?
>> >>
>> >> For me I get comparable output with DMD and LDC
>> >
>> > You guys are most likely comparing apples to oranges - H. S. using some distro-LDC preconfigured to link against shared druntime/Phobos, while LDC usually defaults to the static libs.
>>
>> Druntime and Phobos can be used as shared libraries? What is this magical feature?
>>
>> -Windows User
>
> LOL. Personally, it's a feature that I avoid like the plague, because I hate it when my programs stop working just because I updated dmd (which would be bad enough for a normal user, but it's particularly nasty when you're frequently following master and making local changes, because you're working on Phobos, druntime, and/or dmd). But yeah, when I saw the difference in size that H.S. Teoh was seeing, my first thought was that one was probably using Phobos as a shared library while the other was using it as a static one.
>
> - Jonathan M Davis

On linux the version should be part of the filename, and iirc it'll only try the shared library without a version if it can't find the one it needs. Not too familiar with how it is done on Windows as it isn't really standardized. Could just look at how it is done for the C++ runtime, as I usually have 20 different versions of it installed even for the same MSVC year revision. I just stick the dll in the same folder as the executable.

I'd rather have a single shared library than having 20 instances of it loaded in my 20 dll files.

I can dream though, that one day this fundamental feature will be added to D.

Wouldn't be the least bit surprised if DMD didn't follow Linux conventions though.

On Tuesday, January 1, 2019 1:59:25 PM MST Rubn via Digitalmars-d wrote:
> On Tuesday, 1 January 2019 at 19:28:36 UTC, Jonathan M Davis wrote:
> > On Tuesday, January 1, 2019 7:04:42 AM MST Rubn via
> >> Druntime and Phobos can be used as shared libraries? What is this magical feature?
> >>
> >> -Windows User
> >
> > LOL. Personally, it's a feature that I avoid like the plague, because I hate it when my programs stop working just because I updated dmd (which would be bad enough for a normal user, but it's particularly nasty when you're frequently following master and making local changes, because you're working on Phobos, druntime, and/or dmd). But yeah, when I saw the difference in size that H.S. Teoh was seeing, my first thought was that one was probably using Phobos as a shared library while the other was using it as a static one.
> >
> > - Jonathan M Davis
>
> On linux the version should be part of the filename, and iirc it'll only try the shared library without a version if it can't find the one it needs. Not too familiar with how it is done on Windows as it isn't really standardized. Could just look at how it is done for the C++ runtime, as I usually have 20 different versions of it installed even for the same MSVC year revision. I just stick the dll in the same folder as the executable.
>
> I'd rather have a single shared library than having 20 instances of it loaded in my 20 dll files.
>
> I can dream though, that one day this fundamental feature will be added to D.

The version number is in the file name for Phobos' shared library, but if you're building the development version, that really doesn't help you, because every time you rebuild it, you get a file with the same name but potentially different contents (at least until the next release gets tagged, and the version number gets bumped). If you're only using releases, you don't have that problem. However, unless you keep older versions of the shared library around, as soon as you update, all of your D programs break, because the version they need isn't there anymore. All in all, it's just way simpler to use Phobos as a static library. Sure, when you have a bunch of D programs statically linked, it takes up a few extra megabytes that way, but when systems have terabytes, that doesn't really matter. Anyone who wants to use the shared library version is free to do so, but I really don't think that it makes sense in most cases. I do agree that shared library support in general is something that we should have, but that doesn't mean that it really makes sense as the default for the standard library - especially when so much of it is templated anyway.

I really have no clue what the situation is with dll support, because I pretty much only program on Windows when I'm forced to, but given how dlls work on Windows, I've never been a fan of using them except in cases where you have to. The whole nonsense where you have to rebuild your program, because _anything_ changed in the dll is just ridiculous IMHO. At least on *nix systems, shared libraries can be changed so long as the ABIs of the existing symbols aren't changed, making it so that you can actually update shared libraries without rebuilding everything. We just get screwed with D, because the way it's designed makes it very hard to maintain ABI compatibility if templates are involved, and no attempt is made to maintain ABI compatibility across versions of Phobos. So, for Phobos, using a shared library can be very problematic (whereas a shared library with minimal templates that was designed to maintain a fixed API and ABI would work just fine, just like it does with C).

- Jonathan M Davis

On Tuesday, 1 January 2019 at 22:34:24 UTC, Jonathan M Davis wrote:
> The version number is in the file name for Phobos' shared library, but if you're building the development version, that really doesn't help you, because every time you rebuild it, you get a file with the same name but potentially different contents (at least until the next release gets tagged, and the version number gets bumped). If you're only using releases, you don't have that problem. However, unless you keep older versions of the shared library around, as soon as you update, all of your D programs break, because the version they need isn't there anymore. All in all, it's just way simpler to use Phobos as a static library. Sure, when you have a bunch of D programs statically linked, it takes up a few extra megabytes that way, but when systems have terabytes, that doesn't really matter. Anyone who wants to use the shared library version is free to do so, but I really don't think that it makes sense in most cases. I do agree that shared library support in general is something that we should have, but that doesn't mean that it really makes sense as the default for the standard library - especially when so much of it is templated anyway.

Why wouldn't you keep older versions of phobos? Why are you deleting the old ones when you install a new version of DMD. They are designed in such a way to keep them there for this very reason. I don't really understand your argument here, to me it just seems DMD is just doing something backwards in comparison to the rest of the platform (as usual).

But we aren't, there's no shared library of phobos for Windows and the looks of it Mac OS. It's not about size, it's about easy of use. The way the garbage collector works it might not be as bad in comparison with the C runtime. Where allocating memory from a shared library then unloading it would effectively free all the memory that was allocated by that shared library. But at the same time since allocating memory from the GC

> The whole nonsense where you have to rebuild your program, because _anything_ changed in the dll is just ridiculous IMHO.

What?! Where did you hear this non-sense from? I'm not surprised at the state of shared library support on windows anymore.

Optlink and DMC are the greatest compilers and linkers on Windows, they don't cause ANY problems AT ALL, they are very well supported and shouldn't be removed. Sigh.

On Wed, 02 Jan 2019 00:40:21 +0000, Rubn wrote:
> Why wouldn't you keep older versions of phobos? Why are you deleting the old ones when you install a new version of DMD.

Distributing binaries requires either static linking, shipping multiple files that must be kept together (which is strictly inferior to static linking), or a compatible library already installed on the client machine. Phobos doesn't have a stable ABI, so that means the client would need your specific version of Phobos. That's not super kind.

You can't tell which versions of libphobos.so you need to keep around without running ldd on every executable on your computer. This would be moderately annoying with only official releases; for instance, I've got one tiny project that's old enough that I built it using dsss, and either that would be broken, or I'd have a few dozen versions of phobos running around. More than one per binary that I care about.

Specific to Jonathan M Davis, if you're building dmd/druntime/phobos from git head, the previous problems are hugely compounded, plus you need to assign unique numbers to each built version and update the relevant dmd.conf to match.

On Tuesday, January 1, 2019 5:40:21 PM MST Rubn via Digitalmars-d wrote:
> On Tuesday, 1 January 2019 at 22:34:24 UTC, Jonathan M Davis
>
> wrote:
> > The version number is in the file name for Phobos' shared library, but if you're building the development version, that really doesn't help you, because every time you rebuild it, you get a file with the same name but potentially different contents (at least until the next release gets tagged, and the version number gets bumped). If you're only using releases, you don't have that problem. However, unless you keep older versions of the shared library around, as soon as you update, all of your D programs break, because the version they need isn't there anymore. All in all, it's just way simpler to use Phobos as a static library. Sure, when you have a bunch of D programs statically linked, it takes up a few extra megabytes that way, but when systems have terabytes, that doesn't really matter. Anyone who wants to use the shared library version is free to do so, but I really don't think that it makes sense in most cases. I do agree that shared library support in general is something that we should have, but that doesn't mean that it really makes sense as the default for the standard library - especially when so much of it is templated anyway.
>
> Why wouldn't you keep older versions of phobos? Why are you deleting the old ones when you install a new version of DMD. They are designed in such a way to keep them there for this very reason. I don't really understand your argument here, to me it just seems DMD is just doing something backwards in comparison to the rest of the platform (as usual).

Anyone using a package manager to install dmd is only ever going to end up with the version of Phobos that goes with that dmd. It would be highly abnormal for multiple versions of Phobos to be installed on a system. The only folks who would ever end up with multiple versions of Phobos on the same system are folks actively going to the effort of making it happen instead of using any of the normal install mechanisms. And ultimately, it's far simpler to just always use Phobos as a static library rather than trying to do anything with trying to keep older versions of the shared library around - especially if you're actually building development versions of Phobos.

In general, the only downsides to statically linking are that the executables are slightly larger, and you don't get the benefit of security updates to libraries without rebuilding the binary. And with how easy it is to break ABI compatibility with D libraries (something as simple as a function attribute changing breaks it, and that's trivial if the code is inferring attributes), updating D shared libraries in a way that doesn't require rebuilding the program is quite problematic in general. In some situations, it's worth it, but in general, it really isn't. The place where shared libraries offer far more benefit is with plugins and the like which are loaded while the program is running rather than in trying to actually share code. The ABI problems still exist there, but plugin APIs are generally far more limited than full-on libraries (which reduces the problem), and you simply can't use static libraries in those situations, whereas if you're just linking your program against a library, static libraries work just fine.

I'm by no means against having shared libraries in general work, and I think that full dll support should exist for D on Windows, but aside from plugins, in the vast majority of cases, I think that using shared libraries is far more trouble than it's worth (especially with how difficult it is to maintain ABI compatibility with D libraries).

> > The whole nonsense where you have to rebuild your program, because _anything_ changed in the dll is just ridiculous IMHO.
>
> What?! Where did you hear this non-sense from? I'm not surprised at the state of shared library support on windows anymore.

>From working with dlls with C++. With dlls on Windows, your program links
against a static library associated with the dynamic library, and if any of the symbols are changed, the addresses change, and your program will be unable to load the newer version of the library without being rebuilt against the new version of the static library. This is in stark contrast to *nix where the linking works in such a way that as long as the symbols still exist with the same ABI in the newly built library, they're found when the program loads, and it's not a problem. The addresses aren't hard-coded in the way that happens with dlls on Windows. dlls on Windows allow you to share code so long as the programs are all built against exactly the same version of the dll (and if they're not, then you need separate copies of the dll, and you get into dll hell), whereas with *nix, you can keep updating the shared library as much as you like without changing the executable as long as the API and ABI of the existing symbols don't change.

- Jonathan M Davis

On Tue, Jan 01, 2019 at 07:04:24PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
> Anyone using a package manager to install dmd is only ever going to end up with the version of Phobos that goes with that dmd. It would be highly abnormal for multiple versions of Phobos to be installed on a system.

Wait, what?  Where did you get that idea from?  Since about a half decade ago, Linux distros like Debian have had the possibility of multiple versions of the same shared library installed at the same time. It's pretty much impossible to manage a distro otherwise, since being unable to do so would mean that you cannot upgrade a shared library until ALL upstream code has been updated to use the new version.  Now granted, most of the time new library versions are ABI compatible with older versions, so you don't actually need to keep *every* version of the library around just because some executable somewhere needs them. And granted, splitting a library package into multiple simultaneous versions is only done when necessary.

But the mechanisms for doing so have been in place since a long time ago, and any decent distro management would include setting up the proper mechanisms for multiple versions of Phobos to be installable simultaneously.  Otherwise you have the untenable situation that Phobos cannot be upgraded without breaking every D program currently installed on the system.  Of course, for this to work, the soname needs to be set properly and a sane versioning system (encoded in the soname) needs to be in place. Basically, *every* ABI incompatibility (and I do mean *every*, even those with no equivalent change in the source code) needs to be reflected by a soname change. Which is likely not being done with the current makefiles in git.  Which would explain your observations. But it is certainly *possible*, and often *necessary*, to install multiple versions of the same shared library simultaneously.


> The only folks who would ever end up with multiple versions of Phobos on the same system are folks actively going to the effort of making it happen instead of using any of the normal install mechanisms.

I have no idea what you mean by "normal install mechanisms", because it makes no sense to me to use any system-wide installation mechanism that *doesn't* support multiple versions per shared library.  I mean, how do you even get a sane operating environment at all?  It would be DLL hell compounded with .so hell, episode II.

OTOH, if you're installing stuff by hand or by 3rd party installers (which is generally a bad idea on Linux distros with a distro-specific packaging system -- since colliding assumptions made by either side means endless headaches on the user end), then you're on your own. Maybe that's where you're coming from.


> And ultimately, it's far simpler to just always use Phobos as a static library rather than trying to do anything with trying to keep older versions of the shared library around - especially if you're actually building development versions of Phobos.

Yes, for development builds, I'd tend to agree.


[...]
> From working with dlls with C++. With dlls on Windows, your program links against a static library associated with the dynamic library, and if any of the symbols are changed, the addresses change, and your program will be unable to load the newer version of the library without being rebuilt against the new version of the static library.

Wow. That makes me glad I'm not programming on Windows...


> This is in stark contrast to *nix where the linking works in such a way that as long as the symbols still exist with the same ABI in the newly built library, they're found when the program loads, and it's not a problem. The addresses aren't hard-coded in the way that happens with dlls on Windows. dlls on Windows allow you to share code so long as the programs are all built against exactly the same version of the dll (and if they're not, then you need separate copies of the dll, and you get into dll hell), whereas with *nix, you can keep updating the shared library as much as you like without changing the executable as long as the API and ABI of the existing symbols don't change.
[...]

Yes, though the library authors / distributors will have to be extremely careful to bump the soname for every ABI incompatibility.  All too often people forget to do so, esp. when the ABI incompatibility is not something directly caused by a source code change (the fallacious assumption being "I didn't touch the source code, the ABI can't have changed, right?"). And more often than people would like to admit, sonames fail to get bumped even in the face of source code change, and that's where things start getting ugly.


T

-- 
Государство делает вид, что платит нам зарплату, а мы делаем вид, что работаем.