On Tuesday, 26 November 2013 at 06:54:51 UTC, H. S. Teoh wrote:
> On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
> [...]
>> I don't think I understand having to build dmd just because you have
>> a different distribution, Phobos maybe but dmd... Even for Phobos
>> I'm skeptical but am not going to install a different distro to
>> experiment myself.
>
> If you have used enough Linux distros (or even different versions of the
> same distro!), you will realize that all it takes is for a single shared
> library on the system to be a different version, missing, or just the
> same version built with different compiler flags, and the executable
> will not work.

Yes, the most common being libc. Usually coming from compiling against a newer version and trying to use it with an older version (but not all projects are as disciplined as libc).

> Not to mention that filesystem layout can be different
> across distros (and versions of the same distro), which will break
> things.

It can, but that isn't relevant to the discussion of linking we have here.

> Basically, to guarantee a program runs on distro X, the only way is it
> has to be built from source. If you're lucky, somebody else has already
> done that, and you can just download the binary. (Or better yet,
> somebody packaged it for your distro, then you can just install it via
> your system's package system.) But if not, you'll just have to do it
> yourself.

All you've said is that we are distributing DMD wrong and should be building it on every platform, I still don't know what building DMD has to do with libcurl though (DMD doesn't use it).

> This I agree with. However, it misses the point. The point is that the
> core D toolchain (dmd/phobos/druntime) would have one less dependency,
> which is generally a good thing, because you want to make it as easy as
> possible for people to get a working compiler up and running.

I didn't miss this point, I started with explaining that I've never thought about curl before (other than wondering if I had a problem which I needed to solve with it).

Though, it seems libcurl is installed due to other dependencies. So I'm less likely to see this issue.

I guess I should mention I don't use the Windows installer, just extract the zip. So doubt libcurl is on any of the Windows machines I use.

On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
> As Jonathan M Davis said:
> ---
> Several of the main devs (including Walter) have stated that having std.net.curl on Phobos was a mistake given all of the problems that we've had with it on Windows and Linux as well, and at least some of them have expressed a desire for it to be removed. I expect that there's a good chance that it can and will be removed from Phobos if brought up for discussion.
>
> Certainly, I think that it's clear that we will not add any further external dependencies like curl, because curl has proven to be a big problem. It's very desirable to have bindings and wrappers for C libraries - but putting them in the standard library when it's not guaranteed that the appropriate library is on the system by default has proven to be too problematic, so they should be left to external projects.
> ---
>
> "std.net.curl" can also be moved from Phobos to Deimos.
> Deimos can be rethink, i.e. new build master can add a make building script on Deimos in order to compile all projects included on it, generating "libdeimos.a" and "libdeimos.so", documentation, etc.

+1
It should have been removed from Phobos long ago.

On 11/26/13 7:48 AM, Dejan Lekic wrote:
> On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
>> As Jonathan M Davis said:
>> ---
>> Several of the main devs (including Walter) have stated that having
>> std.net.curl on Phobos was a mistake given all of the problems that
>> we've had with it on Windows and Linux as well, and at least some of
>> them have expressed a desire for it to be removed. I expect that
>> there's a good chance that it can and will be removed from Phobos if
>> brought up for discussion.
>>
>> Certainly, I think that it's clear that we will not add any further
>> external dependencies like curl, because curl has proven to be a big
>> problem. It's very desirable to have bindings and wrappers for C
>> libraries - but putting them in the standard library when it's not
>> guaranteed that the appropriate library is on the system by default
>> has proven to be too problematic, so they should be left to external
>> projects.
>> ---
>>
>> "std.net.curl" can also be moved from Phobos to Deimos.
>> Deimos can be rethink, i.e. new build master can add a make building
>> script on Deimos in order to compile all projects included on it,
>> generating "libdeimos.a" and "libdeimos.so", documentation, etc.
>
> +1
> It should have been removed from Phobos long ago.

The problem is that "long ago" adding libcurl seemed like a perfectly reasonable option, particularly considering that we didn't (and we don't) have anything to replace it.

Andrei

On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
> As Jonathan M Davis said:
> ---
> Several of the main devs (including Walter) have stated that having std.net.curl on Phobos was a mistake given all of the problems that we've had with it on Windows and Linux as well, and at least some of them have expressed a desire for it to be removed. I expect that there's a good chance that it can and will be removed from Phobos if brought up for discussion.
>
> Certainly, I think that it's clear that we will not add any further external dependencies like curl, because curl has proven to be a big problem. It's very desirable to have bindings and wrappers for C libraries - but putting them in the standard library when it's not guaranteed that the appropriate library is on the system by default has proven to be too problematic, so they should be left to external projects.
> ---
>
> "std.net.curl" can also be moved from Phobos to Deimos.
> Deimos can be rethink, i.e. new build master can add a make building script on Deimos in order to compile all projects included on it, generating "libdeimos.a" and "libdeimos.so", documentation, etc.

As others said, Deimos isn't really the correct place for it since it is a wrapper.

I'm fine with it being removed but only after dub has become the official package manager and there is a replacement for it on dub that is easy for users to switch to.

On 26 November 2013 16:39, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> On 11/26/13 7:48 AM, Dejan Lekic wrote:
>>
>> On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
>>>
>>> As Jonathan M Davis said:
>>> ---
>>> Several of the main devs (including Walter) have stated that having std.net.curl on Phobos was a mistake given all of the problems that we've had with it on Windows and Linux as well, and at least some of them have expressed a desire for it to be removed. I expect that there's a good chance that it can and will be removed from Phobos if brought up for discussion.
>>>
>>> Certainly, I think that it's clear that we will not add any further external dependencies like curl, because curl has proven to be a big problem. It's very desirable to have bindings and wrappers for C libraries - but putting them in the standard library when it's not guaranteed that the appropriate library is on the system by default has proven to be too problematic, so they should be left to external projects.
>>> ---
>>>
>>> "std.net.curl" can also be moved from Phobos to Deimos.
>>> Deimos can be rethink, i.e. new build master can add a make building
>>> script on Deimos in order to compile all projects included on it,
>>> generating "libdeimos.a" and "libdeimos.so", documentation, etc.
>>
>>
>> +1
>> It should have been removed from Phobos long ago.
>
>
> The problem is that "long ago" adding libcurl seemed like a perfectly reasonable option, particularly considering that we didn't (and we don't) have anything to replace it.
>
> Andrei
>

Especially when you look at the "competitors"

http://curl.haxx.se/libcurl/competitors.html


The checkboxes that would need to be ticked for me:

1. Does it have any third party dependencies?  Preferably no.
2. Can it be built into libphobos easily - see: zlib in phobos, or
sqllite3.amalgamation.c which I thought *was* in phobos at some
point... maybe not?

On 11/25/13 4:36 PM, Adam D. Ruppe wrote:
> On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:
>> First I'd like to gather an understanding on why we seem to have this
>> problem (far as I understand, the likes of php and python are doing
>> fine with curl, but maybe I'm wrong).
>
> A major difference is there's only one php/python binary, usually build
> on the same system that uses it. Phobos, on the other hand, is generally
> built on the packager's computer, which isn't always binary compatible
> with the deployment box.
>
> On Windows, the problem is simply that curl isn't packaged with dmd, for
> some weird reason, meaning people have to get curl.lib separately.
> That's idiotic. But then again, so are a lot of the dmd Windows
> deficiencies.

Please let me know whether my understanding of the situation is correct:

1. If people have a working installation of libcurl on their machine, we work with it.

2. Otherwise, phobos works fine but attempting to use std.net.curl will fail.

Is this correct? If not, please explain exactly why. If yes, this setup seems entirely appropriate to me.

>> If we do decide to do away with libcurl, one possible solution would
>> be to embed its source code within our build. That way we wouldn't
>> break code that already uses it.
>
> Yes, that would be ideal, we should just statically link curl right into
> the phobos build so it just works everywhere.

There are several questions associated with this.

1. Does the author of libcurl agree with such? My understanding is he would.

2. Would we need to actually build libcurl from source, or just distribute it? In the former case, it would be quite odd that we'd need to do that (and presumably we'd need to take some maintenance burden, too).

3. What if people already have some working libcurl version and would want to use that?


Andrei

On 11/25/13 4:36 PM, Jonathan M Davis wrote:
> The number one problem on Windows is the fact that libcurl does not come with
> Windows and that you have to get a version of it build which is compatible
> with dmd has proven to be a huge hurdle for Windows developers.

Why is "compatible with dmd" an issue? Do we make it any harder for libcurl than other clients? Could you please explain?

> There may be
> other problems, but that's the biggest. Distributing libcurl with dmd could fix
> that, but IIRC, there was some reason why we couldn't do that (or maybe it was
> just that Walter didn't want to - I can't remember).

Walter is opposed only to building libcurl from source for distributing it, which is an understandable stance.

> On Linux, I believe that the biggest problem is that dmd then has to be built
> on your specific distro for it to work with your distro, so the libraries in
> the zip file were only working on debian-based systems. Regardless of why
> libcurl is not distributed with dmd on Windows, we couldn't distribute for
> Linux to fix this problem, because that would conflict with the system's
> libcurl.

So what is the classic recommended solution for such cases? We can't be the first people who are having this issue on Unix.

> There may be other problems, but I think that those are the two bigs ones, and
> they've come up several times (particularly the issues with libcurl and
> Windows).

I've asked the question "what exactly is the problem" several times and you came closest to answering it, but a bunch of stuff is still unclear to me.

> There's also Don's recent rant on why curl was ever supported in the first
> place, but I guess that he just wasn't paying enough attention to it given
> that he only complained about it recently rather than during the review:
>
> http://forum.dlang.org/thread/CADpwU1cu=0r+NJ+4GN9P9Fd_XisK1dsduYrrkjpGAvM0YJ_tUA@mail.gmail.com

First, it's not like we slipped curl in while nobody was looking. Second, I'm sure curl has its shortcomings, but I'd be hard pressed to find better alternatives.

As to just dumping curl and doing our own: I just looked and libcurl has 160K lines. Those include examples and probably a fair amount of header chaff etc. that's unneeded in D, but I'd assess there are on the order of 100K lines of "pulp" code that does real work. Simply dumping std.net.curl would not only break existing code that depends on it in unthinkable ways (no replacement and no recourse), but would put us some 100KLOC behind in terms of a working replacement.

> At this point, I'm inclined to argue that Phobos should not depend on any
> libraries other than the system libraries for each OS so that we can avoid
> further dependencies that may not be available - especially when it comes to
> Windows, since Windows comes with no 3rd party development libraries at all.
> So, I would very much be against std.net.curl's inclusion at this point and
> would argue that it should be in a library separate from Phobos and that if we
> want similar functionality in Phobos, we need to implement it without relying
> on anything other than system libraries. And from previous discussions on the
> topic, I believe that Walter and several other dmd/Phobos devs agree with
> that.

I don't agree with that. Or I don't understand what you're asking. If what you're asking is we break all existing code using std.net.curl with no suggested replacement, that would be pretty bad, no?

If I understand correctly the complaint is: "people who don't have curl can't use std.net.curl". That's to be expected. It's in the name!

A more reasonable complaint is: "people who don't have curl can't use high-level HTTP/FTP client networking, which is expected of a modern language's standard library". For those we should NOT throw away libcurl. We should instead focus on providing e.g. std.net.durl which is an in-house library for doing great stuff. Then people can choose. In the long term we may even recommend moving away from curl because durl is mature and better etc.

> So, I'm all for removing std.net.curl, but if we do that, we'd obviously need
> to deprecate it first rather than simply removing it, and we'd need to have a
> separate library with std.net.curl in it (preferably one which could be pulled
> in via dub) which people could switch to using instead. Whether that library
> would be maintained by us, by Jonas (since he created it), or someone else, I
> don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I
> think that we should avoid putting any more 3rd party dependencies in Phobos
> in the future.

This is probably going to come again. We can't build everything in house in due time. Do we have an openssl layer written in D?


Andrei

On 11/25/13 5:51 PM, Martin Krejcirik wrote:
> On 25.11.2013 8:38, Jordi Sayol wrote:
>> As Jonathan M Davis said: --- Several of the main devs (including
>> Walter) have stated that having std.net.curl on Phobos was a mistake
>> given all of the problems that we've had with it on Windows and Linux
>
> I'm for removing as I always compile Phobos without curl anyway (I don't
> want to install all dependencies).

So then why are you bothered? Are you willing to break a lot of people's code because you don't need this feature?

Andrei

On 11/25/13 6:22 PM, Jordi Sayol wrote:
> El 26/11/13 03:11, Andrei Alexandrescu ha escrit:
>> On 11/25/13 5:51 PM, Martin Krejcirik wrote:
>>> On 25.11.2013 8:38, Jordi Sayol wrote:
>>>> As Jonathan M Davis said: --- Several of the main devs (including
>>>> Walter) have stated that having std.net.curl on Phobos was a mistake
>>>> given all of the problems that we've had with it on Windows and Linux
>>>
>>> I'm for removing as I always compile Phobos without curl anyway (I don't
>>> want to install all dependencies).
>>
>> Does curl in turn depend on many other libraries?
>>
>
> On Debian testing (64-bit):
>
> $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
> 	linux-vdso.so.1 (0x00007fff36519000)
> 	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
> 	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
> 	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
> 	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f28cc724000)
> 	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f28cc332000)
> 	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f28cc0f2000)
> 	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f28cbee3000)
> 	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f28cbc92000)
> 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
> 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
> 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
> 	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f28cb1f1000)
> 	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
> 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
> 	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
> 	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f28ca870000)
> 	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
> 	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f28ca462000)
> 	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f28ca25e000)
> 	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
> 	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
> 	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
> 	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f28c99fa000)
> 	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f28c97f6000)
>

But those are already installed libraries, right?

Andrei

On 11/25/13 10:53 PM, H. S. Teoh wrote:
> On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
> [...]
>> I don't think I understand having to build dmd just because you have
>> a different distribution, Phobos maybe but dmd... Even for Phobos
>> I'm skeptical but am not going to install a different distro to
>> experiment myself.
>
> If you have used enough Linux distros (or even different versions of the
> same distro!), you will realize that all it takes is for a single shared
> library on the system to be a different version, missing, or just the
> same version built with different compiler flags, and the executable
> will not work. Not to mention that filesystem layout can be different
> across distros (and versions of the same distro), which will break
> things.
>
> Basically, to guarantee a program runs on distro X, the only way is it
> has to be built from source. If you're lucky, somebody else has already
> done that, and you can just download the binary. (Or better yet,
> somebody packaged it for your distro, then you can just install it via
> your system's package system.) But if not, you'll just have to do it
> yourself.

Do we with our std.net.curl make this matter any worse?

>> If it is moved to a third party library, the problems will still
>> remain. If the Phobos team is expected to support it, then the
>> burden hasn't shifted either.
>
> This I agree with. However, it misses the point. The point is that the
> core D toolchain (dmd/phobos/druntime) would have one less dependency,
> which is generally a good thing, because you want to make it as easy as
> possible for people to get a working compiler up and running.

But my understanding is that the dependency is conditional, i.e. you have no dependency on libcurl unless you attempt to use std.net.curl. Which strikes me as entirely reasonable. No?

> Requiring
> this library and that library and that other tool, just to get a bare
> bones D toolchain working, is a good way to turn people off D.

Bare bones == no std.net.curl. No?

> If a bare
> bones D compiler is actually working, then they'd have more incentive to
> actually put in more effort to make the frills work too. Having a hello
> world program be uncompilable just because libcurl isn't available, is a
> very bad first impression ("why does this stupid language need libcurl
> just to compile a hello world program?!").

My understanding is that a missing or broken libcurl does not break hello world. Is that correct?

As a note regarding my many messages in this thread: I decided to get to the bottom of this given that I've asked "what is exactly the problem with libcurl" and all I got was unclear answers.


Thanks,

Andrei