Am Tue, 26 Nov 2013 10:44:07 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 11/25/13 4:36 PM, Adam D. Ruppe wrote:
> > On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:
> >> First I'd like to gather an understanding on why we seem to have this problem (far as I understand, the likes of php and python are doing fine with curl, but maybe I'm wrong).
> >
> > A major difference is there's only one php/python binary, usually build on the same system that uses it. Phobos, on the other hand, is generally built on the packager's computer, which isn't always binary compatible with the deployment box.
> >
> > On Windows, the problem is simply that curl isn't packaged with dmd, for some weird reason, meaning people have to get curl.lib separately. That's idiotic. But then again, so are a lot of the dmd Windows deficiencies.
> 
> Please let me know whether my understanding of the situation is correct:
> 
> 1. If people have a working installation of libcurl on their machine, we work with it.
> 
> 2. Otherwise, phobos works fine but attempting to use std.net.curl will fail.
> 
> Is this correct? If not, please explain exactly why. If yes, this setup seems entirely appropriate to me.
> 
> >> If we do decide to do away with libcurl, one possible solution would be to embed its source code within our build. That way we wouldn't break code that already uses it.
> >
> > Yes, that would be ideal, we should just statically link curl right into the phobos build so it just works everywhere.
> 
> There are several questions associated with this.
> 
> 1. Does the author of libcurl agree with such? My understanding is he would.
> 
> 2. Would we need to actually build libcurl from source, or just distribute it? In the former case, it would be quite odd that we'd need to do that (and presumably we'd need to take some maintenance burden, too).
> 
> 3. What if people already have some working libcurl version and would want to use that?
> 
> 
> Andrei

Point 3 is a bigger issue than it may seem at first. A reason to use shared libraries is that the modularity makes it trivial to get security updates, whereas a monolithic Phobos is not necessarily updated when a security fix for cURL becomes available.

This page shows that sometimes security holes are found on a daily basis: http://curl.haxx.se/docs/security.html

-- 
Marco

On 11/26/13 5:05 PM, Adam D. Ruppe wrote:
> On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei Alexandrescu wrote:
>> Care to make it interesting? I may have some funds for this. But we're
>> looking for no less than a glorious 100% replacement.
>
> Meh, my first choice is still to just bundle curl with phobos, like we
> do with zlib. It seems silly to me to want a glorious 100% replacement
> of an *open source* library.
>
> Regardless, I probably wouldn't be the ideal choice for changing the
> implementation because I barely use the interface: I wrote my own http
> modules, including one that uses curl and one that doesn't, before
> std.net.curl came around. The phobos thing had no compelling benefit to
> me, so I never switched most my code.
>
> Thus, I'm not very familiar with std.net.curl's strengths and weaknesses
> and would likely break it somehow while changing the implementation.
>
>
> ...unless doing a new interface is on the table too. Then, we can leave
> std.net.curl exactly how it is, so people who use it don't have broken
> code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are
> phased in for people who want them. I could get behind that.

A new interface is also on the table, but that brings the additional burden of defining your own design. std.net.curl has already been approved.

Andrei

On Wednesday, November 27, 2013 02:05:39 Adam D. Ruppe wrote:
> ...unless doing a new interface is on the table too. Then, we can leave std.net.curl exactly how it is, so people who use it don't have broken code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are phased in for people who want them. I could get behind that.

If we were going to replace std.net.curl, I would fully expect us to end up with a new API, because the current API is what is beacuse of what curl can and can't do rather than because it's the best API for what you're conceptually trying to do. The new API might end up being fairly close to the current one, but I would not consider it desirable to try and maintain a 100% compatible API if we were replacing the current implementation except in cases where the current API is the best choice even without curl underneath it. Maybe that would be most of the API, but I'd hate to see us put ourselves in a straightjacket based on how curl works when writing an API which didn't depend on curl - particularly as writing our own would give us the opportunity to make sure that we had something that made the most conceptual sense rather than necessarily having what curl has.

- Jonathan M Davis

On 11/27/2013 6:12 AM, David Nadlinger wrote:
> On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:
>> On 11/26/13 12:21 PM, Dicebot wrote:
>>> I really think providing just source + single additional .deb package as
>>> an example is the best way to go.
>>
>> Well we kind of do that already. No?
>
> The issue is (mainly) with people expecting the binaries from the
> official .zip file to work on all Linux platforms.
>

This problem would be solved by going with a dynamic binding for libcurl rather than a static binding. Then std.net.curl could load libcurl on demand and it doesn't matter how it was compiled (unless, of course, the libcurl API changes with compiler flags).

On Wednesday, 27 November 2013 at 01:40:15 UTC, Mike Parker wrote:
> This problem would be solved by going with a dynamic binding for libcurl rather than a static binding. Then std.net.curl could load libcurl on demand and it doesn't matter how it was compiled (unless, of course, the libcurl API changes with compiler flags).

In addition, even with Phobos as a shared library libcurl wouldn't necessarily be loaded if it's not actually used then.

David

Am Tue, 26 Nov 2013 11:02:06 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 11/25/13 6:22 PM, Jordi Sayol wrote:
> > On Debian testing (64-bit):
> >
> > $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
> > 	linux-vdso.so.1 (0x00007fff36519000)
> > 	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
> > 	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
> > 	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
> > 	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f28cc724000)
> > 	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f28cc332000)
> > 	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f28cc0f2000)
> > 	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f28cbee3000)
> > 	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f28cbc92000)
> > 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
> > 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
> > 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
> > 	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f28cb1f1000)
> > 	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
> > 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
> > 	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
> > 	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f28ca870000)
> > 	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
> > 	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f28ca462000)
> > 	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f28ca25e000)
> > 	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
> > 	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
> > 	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
> > 	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
> > 	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f28c99fa000)
> > 	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f28c97f6000)
> >
> 
> But those are already installed libraries, right?
> 
> Andrei
> 

Yes, the curl package on this Linux distribution depends on
all the listed libraries and when curl is installed, they are
installed with it. Often these dependencies are added on a per
case basis. E.g. when the distribution adds a program using
Kerberos network authentication through curl, it is enabled in
the curl package by the maintainer.
A notable exception is with source only distributions like
Gentoo, where packages can depend on curl WITH certain options
enabled. In this case curl is recompiled whenever an
additional requirement occurs and ideally stays as big as
necessary and as small as possible ;)

-- 
Marco

Am Tue, 26 Nov 2013 12:04:59 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> What flags are we requiring for use with std.net.curl? Are those depending on what specific calls are being used (e.g. using https etc)? We could document that stuff. All in all it doesn't seem unreasonable to me to require certain specifics for using std.net.curl.
> 
> Andrei

Yes, listing requirements is always good to do. But let's take a look at some hard facts first. `ldd libphobos2.so' on the linux amd64 version from the zip (for 2.064.2) reveals:

	linux-vdso.so.1 (0x00007fff2fd96000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f1294e66000)
	libcurl-gnutls.so.4 => not found
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1294c48000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f1294951000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f1294748000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f12943a0000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003989a00000)

libcurl-gnutls is not found on *my* installation of Linux.
You correctly made out https (SSL) as a culprit. It is not a
single flag though, but up to 6(!) different SSL "backends"
you can chose from. Typically it comes down to OpenSSL and
GnuTLS. The Debian system that this phobos.so was compiled on
obviously used a GnuTLS version that was suffixed with
"-gnutls" to allow a parallel installation of a curl
version with e.g. OpenSSL. On my Linux on the other hand, I
can only install one version of curl at a time and no suffix
is used. So:

1. curl library names differ on distributions

Next, when you issue:
`readelf --version-info libphobos2.so | less'
you will get a dump of the .gnu.version section that contains
entries like "CURL_GNUTLS_3". These correlate to the
respective "versioned symbols" from libcurl.so. Versioned
symbols are the regular C export names with an @-suffix. The
following command should reveal them:
`readelf -s /usr/lib64/libcurl-gnutls.so.4 | grep GNUTLS_3'
What this means, is that not only the library name phobos.so
uses is linked to the gnutls version, but also each of the SSL
specific functions! This makes for:

2. phobos.so uses GNUTLS version of curl functions

You can rely on the fact that curl is installed on Linux and that it exposes SSL functionality, but the way phobos.so is created it is Debian and GnuTLS specific at the moment.

-- 
Marco

On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
Alexandrescu wrote:
> A new interface is also on the table, but that brings the additional burden of defining your own design. std.net.curl has already been approved.

Aye, but that'd be worth it. There's a few pieces we should get
together:

1) a url module. My cgi.d has one that may be usable with a
little testing:
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022

about 150 lines long, including some unittest. Also has some
functions that build on std.uri to make encoding multiple values
easier - working with associative arrays instead of just strings.

This is useful since we can then pass type-safe urls to various
functions and get consistent parsing and creation. The basedOn()
method in my code there is meant to do relative links in a
browser fetch situation. Regardless, all this is fairly useful
independent of network functions too.

2) Some kind of SSL sockets, including adding ssl to an already
open socket (needed for STARTTLS on smtp at least). I did a
partial implementation as a subclass of Phobos' Socket with
openssl. Should also do a Windows impl and probably a gnutls as
well.

I think this would be ideally picked at runtime, with dlopen().
If gnutls is available, make that work. Otherwise, use openssl,
and if none are there, throw upon ssl requests. On Windows, it
can just use the built in ssl facilities the operating system
provides.

I'd probably make this a new module that extends std.socket
rather than editing std/socket.d.

3) async (overlapped) I/O on those sockets. This might need
std.socket to be edited to add the functionality.



Then, we go into the curl replacement:


4) The http module. Offers simple functions (get(), post() that
just work), a reusable client object, and advanced functions, all
in sync and async versions.

My http.d can be a starting point, though I have a new api in
mind. I'll perhaps slap together a first draft over the next
week, if my other work leaves enough time. I have to write a book
chapter on Phobos soon too! lol maybe time to scratch the
std.net.curl portion.

5) The smtp module. Offers the low level send functions, and a
higher level MIME message builder class. I've written the latter
already.
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/email.d#L27

though it uses my html to text module, which uses my dom module,
which uses my characterencodings module. That could be easily
pulled out though, it would just require the user to set the
plain text version of an html email themselves.

As you can see there too, it currently uses std.net.curl to
actually send the email, since my smtp function (written but it
isn't great) doesn't support the starttls command that Amazon SES
uses and i needed that for a work project...

My module can also read email, but I don't think that's needed in
the std.net.email.

6) The ftp module. I haven't written ftp code for a long time, it
would perhaps be better if someone else wrote this, though if
nobody else does, I'm sure I can refresh myself well enough with
a lil time. But the beauty of modules is we can divide and
conquer!

7) Any other protocols we care about. Personally, I'd kinda love
some ssh and scp code... but I'm afraid actually writing that is
something I don't want to do, perhaps we can do a wrapper for
libssh. Perhaps not appropriate for phobos due to dependency and
licensing but I'd still love someone to write it and post a link
here!

8) If we want to, there could be a combined module that reads a
runtime url and dispatches to the appropriate protocol, like the
std.net.curl.download and upload can do. But tbh I don't even
care much about this. It is trivial once the other modules are
written anyway.

9) Maybe my OAuth module's client code can be cleaned up and added too, as an add-on to the http functionality.


Each piece here is pretty independent, except the first three,
and I have enough written already for them to get started at
least.

On 2013-11-27 05:09, Adam D. Ruppe wrote:

> 2) Some kind of SSL sockets, including adding ssl to an already
> open socket (needed for STARTTLS on smtp at least). I did a
> partial implementation as a subclass of Phobos' Socket with
> openssl. Should also do a Windows impl and probably a gnutls as
> well.

Doesn't GnuTLS provide an OpenSSL API?

-- 
/Jacob Carlborg

On Wednesday, 27 November 2013 at 04:09:53 UTC, Adam D. Ruppe wrote:
> On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
> Alexandrescu wrote:
>> A new interface is also on the table, but that brings the additional burden of defining your own design. std.net.curl has already been approved.
>
> Aye, but that'd be worth it. There's a few pieces we should get
> together:
>
> 1) a url module. My cgi.d has one that may be usable with a
> little testing:
> https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022

Should we consider interoperability with std.path too?

> 4) The http module. Offers simple functions (get(), post() that
> just work), a reusable client object, and advanced functions, all
> in sync and async versions.

I worked on http protocol, and I don't think that's will be a big issue. Http client side is not so complex if you need a working subset. I think we don't need gzip compression in first release, for example.

> 9) Maybe my OAuth module's client code can be cleaned up and added too, as an add-on to the http functionality.

+1 for an oauth module. It would be very useful to work with third party api (facebook, google, etc ...) and to write api too! However I think it depends on http(s) module.