On Monday, May 09, 2016 09:35:18 sigod via Digitalmars-d wrote:
> On Monday, 9 May 2016 at 08:55:36 UTC, Jonathan M Davis wrote:
> > But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually replace all of its functionality - at least not without adding a dependency on a different C library, since there's no way that it's sane to do the crypto stuff ourselves without a crypto expert, and even then, we should think twice about it. I could see implementing the SSL/TLS protocols themselves but not the crypto they use. If we replace std.net.curl, we likely should just provide the basic HTTP functionality, and leave the rest to a dub package that we move std.net.curl to.
>
> Any chances that we can produce good crypto code over time? And verify it with experts, of course.

I'm sure that it's possible, but it's also very dangerous territory to deal with. Even if you get the crypto itself right, there are all kinds of other nasty problems you have to worry about that most people won't - like ensuring that certain operations take the same amount of time regardless of whether they succeed or not so that folks snooping can't figure out what is and isn't succeeding. There's also the issue of keeping up-to-date with which protocols should be used and which shouldn't (e.g. if I understand correctly, pretty much nothing should be using SSL now; it should all be TLS). So, while there is no technical barrier to us implementing all of this in D (the crypto itself included), it's a very tall order to get it right. Having the code vetted by experts would _definitely_ help, but it's going to go a lot better if we have a security expert writing and maintaining the code, and I'm not sure that we have anyone like that among our active contributors right now (though someone like that is definitely welcome to start contributing code).

I actually considered implementing SSL in D at one point (with the idea that I'd use a C or C++ library for the crypto itself), but the more that I learn about this stuff, the more leery I am of implementing anything that's in the security domain. And the only way that I would ever consider doing the crypto itself would be by porting C code that did it, and even that seems pretty hairy - especially since I wouldn't have the knowledge necessary to maintain it.

So, we may very well end up with full-on crypto stuff written in D at some point - maybe even in the standard library - but it's something that we need to be very careful with.

- Jonathan M Davis

On Monday, 9 May 2016 at 09:35:18 UTC, sigod wrote:
> On Monday, 9 May 2016 at 08:55:36 UTC, Jonathan M Davis wrote:
>> But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually replace all of its functionality - at least not without adding a dependency on a different C library, since there's no way that it's sane to do the crypto stuff ourselves without a crypto expert, and even then, we should think twice about it. I could see implementing the SSL/TLS protocols themselves but not the crypto they use. If we replace std.net.curl, we likely should just provide the basic HTTP functionality, and leave the rest to a dub package that we move std.net.curl to.
>
> Any chances that we can produce good crypto code over time? And verify it with experts, of course.

https://github.com/etcimon/botan AFAIK, it is already used in production by its author, in combination with libasync + vibe.d + http2 for a full stack D solution.

> ...

On Sunday, 8 May 2016 at 13:31:46 UTC, Vladimir Panteleev wrote:
> On Sunday, 8 May 2016 at 09:43:14 UTC, Steven Schveighoffer wrote:
>> My understanding is that libcurl isn't default installed on some platforms (e.g. windows). So we have a dependency on an external library that may not be present. If in "first five minutes" user wants to execute downloads from http server and is told "oh, to use Phobos, you must download another library, sorry", I don't think that helps.
>
> I understand that this is no longer a problem. We link dynamically to the DLL, which is distributed with DMD, and it's lazy-loaded (so we don't do anything at all until the first std.net.curl function is called).

That could lead to some nasty surprises when distributing a binary. I'd rather hear my customer say "I can't start your program, because some .dll is missing" than "it randomly crashes / doesn't work".

On Tuesday, 10 May 2016 at 14:19:15 UTC, krzaq wrote:
> On Sunday, 8 May 2016 at 13:31:46 UTC, Vladimir Panteleev wrote:
>> On Sunday, 8 May 2016 at 09:43:14 UTC, Steven Schveighoffer wrote:
>>> [...]
>>
>> I understand that this is no longer a problem. We link dynamically to the DLL, which is distributed with DMD, and it's lazy-loaded (so we don't do anything at all until the first std.net.curl function is called).
>
> That could lead to some nasty surprises when distributing a binary. I'd rather hear my customer say "I can't start your program, because some .dll is missing" than "it randomly crashes / doesn't work".

You imply that the error message that Druntime generates when it cannot find the DLL is significantly worse than the error message that the OS generates when it cannot find the DLL.

On Tuesday, 10 May 2016 at 14:26:27 UTC, Vladimir Panteleev wrote:
> On Tuesday, 10 May 2016 at 14:19:15 UTC, krzaq wrote:
>> On Sunday, 8 May 2016 at 13:31:46 UTC, Vladimir Panteleev wrote:
>>> On Sunday, 8 May 2016 at 09:43:14 UTC, Steven Schveighoffer wrote:
>>>> [...]
>>>
>>> I understand that this is no longer a problem. We link dynamically to the DLL, which is distributed with DMD, and it's lazy-loaded (so we don't do anything at all until the first std.net.curl function is called).
>>
>> That could lead to some nasty surprises when distributing a binary. I'd rather hear my customer say "I can't start your program, because some .dll is missing" than "it randomly crashes / doesn't work".
>
> You imply that the error message that Druntime generates when it cannot find the DLL is significantly worse than the error message that the OS generates when it cannot find the DLL.

Even if it's better, it's significantly *later*. Once my code runs in a production environment I'd really prefer it to not to stop over things that could've easily been prevented.

On Tuesday, 10 May 2016 at 14:35:17 UTC, krzaq wrote:
> On Tuesday, 10 May 2016 at 14:26:27 UTC, Vladimir Panteleev wrote:
>> On Tuesday, 10 May 2016 at 14:19:15 UTC, krzaq wrote:
>>> On Sunday, 8 May 2016 at 13:31:46 UTC, Vladimir Panteleev wrote:
>>>> On Sunday, 8 May 2016 at 09:43:14 UTC, Steven Schveighoffer wrote:
>>>>> [...]
>>>>
>>>> I understand that this is no longer a problem. We link dynamically to the DLL, which is distributed with DMD, and it's lazy-loaded (so we don't do anything at all until the first std.net.curl function is called).
>>>
>>> That could lead to some nasty surprises when distributing a binary. I'd rather hear my customer say "I can't start your program, because some .dll is missing" than "it randomly crashes / doesn't work".
>>
>> You imply that the error message that Druntime generates when it cannot find the DLL is significantly worse than the error message that the OS generates when it cannot find the DLL.
>
> Even if it's better, it's significantly *later*. Once my code runs in a production environment I'd really prefer it to not to stop over things that could've easily been prevented.

You can actually still statically link libcurl to the executable if you so desire.

Regardless, I'm not sure that we should worry much about the particular case where you 1) link to libcurl dynamically 2) distribute an incomplete application with some DLLs missing and 3) use libcurl late during your application's runtime.

On Tuesday, May 10, 2016 14:51:02 Vladimir Panteleev via Digitalmars-d wrote:
> On Tuesday, 10 May 2016 at 14:35:17 UTC, krzaq wrote:
> > On Tuesday, 10 May 2016 at 14:26:27 UTC, Vladimir Panteleev wrote:
> >> You imply that the error message that Druntime generates when it cannot find the DLL is significantly worse than the error message that the OS generates when it cannot find the DLL.
> >
> > Even if it's better, it's significantly *later*. Once my code runs in a production environment I'd really prefer it to not to stop over things that could've easily been prevented.
>
> You can actually still statically link libcurl to the executable if you so desire.
>
> Regardless, I'm not sure that we should worry much about the particular case where you 1) link to libcurl dynamically 2) distribute an incomplete application with some DLLs missing and 3) use libcurl late during your application's runtime.

Testing is supposed to fix that sort of thing. And since you _can_ statically link against libcurl if you want to, then you can still get the compile time error, and I really don't see such issues as a viable argument either.

I do wish that std.net.curl were a dub package rather than in Phobos, but having issues where you forgot to include libcurl's dll with your binary is a separate issue.

- Jonathan M Davis

Am Sun, 8 May 2016 11:33:07 +0300
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> On 5/8/16 11:05 AM, Jonathan M Davis via Digitalmars-d wrote:
> > On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
> >> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
> >>> But std.net.curl supports not just HTTP but also FTP etc. so i guess that won't suffice.
> >>
> >> We can always implement ftp too, it isn't that complicated of a protocol.
> >>
> >> Though, I suspect its users are a tiny minority and they might not mind depending on a separate curl package.
> >
> > An alternative would be to move std.net.curl into a dub package.
> 
> That would still be a breaking change, is that correct?
> 
> I'm unclear on what the reasons are for removing libcurl so I'd love to see them stated clearly. Walter's argumentation was vague - code that we don't control etc. There have been past reports of issues with libcurl on windows, have those not been permanently solved?
> 
> I even see a plus: dealing with libcurl is a good exercise in eating our dogfood regarding "interfacing with C libraries is trivial" stance. Having to deal with it is a reflection of what other projects have to do on an ongoing basis.
> 
> 
> Andrei
> 

The curl problems are more or less solved now, but it has caused quite some trouble:

As long as we were statically linking curl:
 * We can't use curl when producing cross compilers for GDC as the
   minimal builds used by crosstool do not include curl. They do not
   even include zlib, we're just lucky that zlib is in GCC and we
   compile it statically into druntime. OTOH I'm not sure if we can get
   conflicts between our statically compiled zlib and libraries which
   link against zlib.
 * For static libraries, we don't need curl at link time, but for
   dynamic libraries we do need it.
 * There was the library versioning issue which made DMD builds
   unusable on some distributions.
 * http://bugzilla.gdcproject.org/show_bug.cgi?id=202 Even programs not
   using libcurl will sometimes require linking curl (This is because
   common templates such as std.conv.to might be instatiated in curl,
   so curl.o is pulled in, which depends on libcurl)
 * Library order when linking is important nowadays, so you need a way
   to specify -lcurl in the correct location relative to -lphobos

Still open issues:
 * Even when dynamically loading curl, it introduces a new dependency:
   libdl for dynamic loading. This is not an issue for shared
   libraries, but the list of libraries which need to be hard coded when
   linking a static libphobos is already quite long:
   -lc -lrt -lm -ldl -lz -lstdc++ -luuid -lws2_32
   In fact GDC doesn't link some of these yet and Iain doesn't want to
   add more special cases to our linking code
(https://github.com/D-Programming-GDC/GDC/pull/182
https://github.com/D-Programming-GDC/GDC/pull/181).


Additionally the complete API, integration with D features and performance is not really up to phobos standards. This is because of libcurl API limitations though, so there's nothing we can do about it. As long as we don't have a D replacement though, it's still the best HTTP client available...

On Monday, 9 May 2016 at 10:44:13 UTC, ZombineDev wrote:
> On Monday, 9 May 2016 at 09:35:18 UTC, sigod wrote:
>> On Monday, 9 May 2016 at 08:55:36 UTC, Jonathan M Davis wrote:
>>> But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually replace all of its functionality - at least not without adding a dependency on a different C library, since there's no way that it's sane to do the crypto stuff ourselves without a crypto expert, and even then, we should think twice about it. I could see implementing the SSL/TLS protocols themselves but not the crypto they use. If we replace std.net.curl, we likely should just provide the basic HTTP functionality, and leave the rest to a dub package that we move std.net.curl to.
>>
>> Any chances that we can produce good crypto code over time? And verify it with experts, of course.
>
> https://github.com/etcimon/botan AFAIK, it is already used in production by its author, in combination with libasync + vibe.d + http2 for a full stack D solution.


So what's the consensus on this issue?
Can't we simply move std.net.curl and etc.c.curl to undead and just _not_ include it in Phobos?
Anyone reasonable will use requests [1], vibe.d's async requestHTTP [2] or their home-grown library anyways. As mentioned in another thread [3], Phobos is bloated with "old" modules that wouldn't have made it through today's review process.

With DUB's "new" single file feature and DUB being bundled in the DMD release process, any DUB library is just a header comment away.
Moreover we can configure the DUB package to automatically compile the curl sources, so that it won't create any troubles as e.g. the people on Windows or the gdc maintainers experience.

[1] https://github.com/ikod/dlang-requests
[2] vibed.org/api/vibe.http.client/requestHTTP
[3] http://forum.dlang.org/post/odbddahgxadkffbwcuje@forum.dlang.org

Seb wrote:
> So what's the consensus on this issue?

leave curl where it is now.

i am teh user. i have alot of code depending on std.net.curl. i hope that "we are not breaking user's code" motto is not just a PR slogan.