On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
> > But std.net.curl supports not just HTTP but also FTP etc. so i guess that won't suffice.
>
> We can always implement ftp too, it isn't that complicated of a protocol.
>
> Though, I suspect its users are a tiny minority and they might not mind depending on a separate curl package.

An alternative would be to move std.net.curl into a dub package. And even if we want to have a Phobos replacement for std.net.curl before ripping it out rather than just redirecting folks to a dub package, we could still take the approach of replacing the primary stuff that std.net.curl does without necessarily replacing all of it, since it would still be easy to use what we have now by grabbing it from dub. And it _is_ the sort of thing that makes perfect sense as a dub package.

- Jonathan M Davis

On 5/8/16 11:05 AM, Jonathan M Davis via Digitalmars-d wrote:
> On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
>> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
>>> But std.net.curl supports not just HTTP but also FTP etc. so i
>>> guess that won't suffice.
>>
>> We can always implement ftp too, it isn't that complicated of a
>> protocol.
>>
>> Though, I suspect its users are a tiny minority and they might
>> not mind depending on a separate curl package.
>
> An alternative would be to move std.net.curl into a dub package.

That would still be a breaking change, is that correct?

I'm unclear on what the reasons are for removing libcurl so I'd love to see them stated clearly. Walter's argumentation was vague - code that we don't control etc. There have been past reports of issues with libcurl on windows, have those not been permanently solved?

I even see a plus: dealing with libcurl is a good exercise in eating our dogfood regarding "interfacing with C libraries is trivial" stance. Having to deal with it is a reflection of what other projects have to do on an ongoing basis.


Andrei

On 5/8/16 10:33 AM, Andrei Alexandrescu wrote:
> On 5/8/16 11:05 AM, Jonathan M Davis via Digitalmars-d wrote:
>> On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
>>> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
>>>> But std.net.curl supports not just HTTP but also FTP etc. so i
>>>> guess that won't suffice.
>>>
>>> We can always implement ftp too, it isn't that complicated of a
>>> protocol.
>>>
>>> Though, I suspect its users are a tiny minority and they might
>>> not mind depending on a separate curl package.
>>
>> An alternative would be to move std.net.curl into a dub package.
>
> That would still be a breaking change, is that correct?
>
> I'm unclear on what the reasons are for removing libcurl so I'd love to
> see them stated clearly. Walter's argumentation was vague - code that we
> don't control etc. There have been past reports of issues with libcurl
> on windows, have those not been permanently solved?

My understanding is that libcurl isn't default installed on some platforms (e.g. windows). So we have a dependency on an external library that may not be present. If in "first five minutes" user wants to execute downloads from http server and is told "oh, to use Phobos, you must download another library, sorry", I don't think that helps.

Ironically, zlib is statically compiled by the build, always included, it's not changing anytime soon (Latest release is April 2013), and so requires zero maintenance. Since it's statically in phobos, it is zero pain to the end user. Yet that is the one you want to remove over libcurl? I agree the D wrapper needs re-implementing, but the C library part of it is rock-solid.

-Steve

On Sunday, 8 May 2016 at 09:43:14 UTC, Steven Schveighoffer wrote:
> My understanding is that libcurl isn't default installed on some platforms (e.g. windows). So we have a dependency on an external library that may not be present. If in "first five minutes" user wants to execute downloads from http server and is told "oh, to use Phobos, you must download another library, sorry", I don't think that helps.

I understand that this is no longer a problem. We link dynamically to the DLL, which is distributed with DMD, and it's lazy-loaded (so we don't do anything at all until the first std.net.curl function is called).

Andrei, is there any plans to drop etc.c.odbc ?

On 5/8/16 5:37 PM, Suliman wrote:
> Andrei, is there any plans to drop etc.c.odbc ?

No. -- Andrei

On 6/5/2016 20:17, qznc wrote:
> On Friday, 6 May 2016 at 09:21:43 UTC, Andrei Alexandrescu wrote:
>> On 5/6/16 10:32 AM, Robert burner Schadek wrote:
>>> As discussed yesterday at DConf, curl in phobos must go.
>>>
>>> The plan is as follows.
>>>
>>> 1. undocument everything curl related in may 2016
>>> 2. deprecate everything curl related in may 2017
>>> 3. delete everything curl related in may 2018
>>> 3.1 move curl stuff to undead
>>>
>>> PR: https://github.com/dlang/phobos/pull/4283
>>
>> 0. Write curl replacement
>
> Could we simply move it into a dub package? Or does that clash symbols
> or anything somehow?

Just tried this, works fine! Try it yourself:

$ sed -i '' 's/curl //' phobos/posix.mak
$ make -C phobos -f posix.mak clean
$ make -C phobos -f posix.mak

$ dub init undead-curl
$ echo 'targetType "sourceLibrary"' >> undead-curl/dub.sdl
$ mkdir -p undead-curl/source/etc/c undead-curl/source/std/net/
$ mv phobos/etc/c/curl.d undead-curl/source/etc/c/
$ mv phobos/std/net/curl.d undead-curl/source/std/net/
$ dub add-local undead-curl

Tested with an app I had made not long ago (guess what it does!)

$ cd sciamspider
$ echo 'dependency "undead-curl" version="~master"' >> dub.sdl
$ dub

Ran without any changes!

On Sunday, May 08, 2016 11:33:07 Andrei Alexandrescu via Digitalmars-d wrote:
> On 5/8/16 11:05 AM, Jonathan M Davis via Digitalmars-d wrote:
> > On Sunday, May 08, 2016 02:44:48 Adam D. Ruppe via Digitalmars-d wrote:
> >> On Saturday, 7 May 2016 at 20:50:53 UTC, Jonas Drewsen wrote:
> >>> But std.net.curl supports not just HTTP but also FTP etc. so i guess that won't suffice.
> >>
> >> We can always implement ftp too, it isn't that complicated of a protocol.
> >>
> >> Though, I suspect its users are a tiny minority and they might not mind depending on a separate curl package.
> >
> > An alternative would be to move std.net.curl into a dub package.
>
> That would still be a breaking change, is that correct?

Any replacement would be a breaking change, unless the intention is to keep to std.net.curl's API while ripping out the actual curl usage, and that seems unnecessarily restrictive to me and likely to end up with something inferior to what we would otherwise get. IIRC, we've already had issues with std.net.curl not quite doing what we want (though at least some of those could likely be fixed if we have full control over what the underlying code is doing).

But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually replace all of its functionality - at least not without adding a dependency on a different C library, since there's no way that it's sane to do the crypto stuff ourselves without a crypto expert, and even then, we should think twice about it. I could see implementing the SSL/TLS protocols themselves but not the crypto they use. If we replace std.net.curl, we likely should just provide the basic HTTP functionality, and leave the rest to a dub package that we move std.net.curl to.

But even if we did manage to replace all of std.net.curl's functionality in Phobos, I definitely think that we should put in dub. Whatever problems we've had with it, they're definitely not bad enough for us to want to eradicate it as D code, just remove it from Phobos. And if it were a dub package, updating your code to use it should be pretty trivial. Once std.net.curl was deprecated in Phobos, and it was put in dub as a separate package, anyone who wanted to update would just change their import statements to use the module name from the dub package and add it as a dependency to their dub.json/dub.sdl file. So, as far as breaking changes go, moving std.net.curl from Phobos to code.dlang.org would be very easy to deal with.

> I'm unclear on what the reasons are for removing libcurl so I'd love to see them stated clearly. Walter's argumentation was vague - code that we don't control etc. There have been past reports of issues with libcurl on windows, have those not been permanently solved?
>
> I even see a plus: dealing with libcurl is a good exercise in eating our dogfood regarding "interfacing with C libraries is trivial" stance. Having to deal with it is a reflection of what other projects have to do on an ongoing basis.

Walter's main complaint seemed to be that he didn't like the idea of depending on C libraries other than the C runtime in our standard library, since it meant that there was code in our standard library that we did not have control of, and it arguably looks bad to have to use C libraries in our own standard library. But the big problem that I'm aware of has had to do with the fact that we then have an external dependency that not everyone has on their system. Even on Linux, on distros that separate out "dev" packages so that you have to install one package to use a library and another to build code using it, libcurl is not necessarily going to be there to be built against (and it's definitely not on Windows normally). And that's definitely caused problems - though including libcurl with the dmd installer (which we didn't do initially) has reduced those problems.

Personally, I've run into issues with the tools repo due to its dependence on std.net.curl, and while that may be related to issues on my system rather than anything that's been done with Phobos itself, that doesn't exactly give me warm, fuzzy feelings for std.net.curl.

In addition, Phobos is not tied to curl's release cycle, and if curl gets a version bump on someone's system, and Phobos hasn't been updated to match, they're going to have a problem. And if we updated to match the version bump, and their distro hadn't, then we'd also have a problem. AFAIK, curl's API has been fairly stable such that this hasn't been a problem, but it's definitely a concern with C libraries in general, and it was pointed out at dconf that it _is_ a problem with the sqlite bindings. So, even if we decide to keep std.net.curl, I think that versioning issues alone should make us very leery of including C libraries in Phobos - be they via bindings or wrappers. I don't think that we want to be in a situation where someone has to use a specific version of dmd and Phobos, because it's the one that matches a C library on their system that Phobos depends on. And while that might be manageable with one C library dependency, it likely wouldn't be with multiple. Having such dependencies in dub packages should make it easier to manage them.

Another thing to consider is portability. Just because a library is written in C doesn't mean that it's going to work on all of the targets that we want to hit with dmd, ldc, and gdc. We have control over that sort of thing if the code is in D. We don't if it's in C. From the looks of it, curl is available for the vast majority of platforms, so this likely won't be a problem with curl, but it is a reason to think twice before considering adding another dependency on a C library to Phobos.

If std.net.curl were not already in Phobos, I think that it's quite clear that it would be a great fit for a dub package, and I would definitely vote against its inclusion in Phobos. And at this point, I will likely do the same for any C library. I think that those all belong on code.dlang.org and not in Phobos.

- Jonathan M Davis

On Monday, 9 May 2016 at 08:55:36 UTC, Jonathan M Davis wrote:
> But given that std.net.curl handles stuff like SSL/TLS, we _can't_ actually replace all of its functionality - at least not without adding a dependency on a different C library, since there's no way that it's sane to do the crypto stuff ourselves without a crypto expert, and even then, we should think twice about it. I could see implementing the SSL/TLS protocols themselves but not the crypto they use. If we replace std.net.curl, we likely should just provide the basic HTTP functionality, and leave the rest to a dub package that we move std.net.curl to.

Any chances that we can produce good crypto code over time? And verify it with experts, of course.

> In addition, Phobos is not tied to curl's release cycle, and if curl gets a version bump on someone's system, and Phobos hasn't been updated to match, they're going to have a problem. And if we updated to match the version bump, and their distro hadn't, then we'd also have a problem.

Oh, that explains why I constantly ran into [this issue][0]. While it was fixed in curl itself.

[0]: https://github.com/curl/curl/issues/447

On Monday, 9 May 2016 at 08:55:36 UTC, Jonathan M Davis wrote:
> Walter's main complaint seemed to be that he didn't like the idea of depending on C libraries other than the C runtime in our standard library, since it meant that there was code in our standard library that we did not have control of, and it arguably looks bad to have to use C libraries in our own standard library. But the big problem that I'm aware of has had to do with the fact that we then have an external dependency that not everyone has on their system. Even on Linux, on distros that separate out "dev" packages so that you have to install one package to use a library and another to build code using it, libcurl is not necessarily going to be there to be built against (and it's definitely not on Windows normally). And that's definitely caused problems - though including libcurl with the dmd installer (which we didn't do initially) has reduced those problems.

Martin has also made curl lazy-loaded, so you should not have any problems unless you both try to use curl and don't have the curl DLL in your PATH.