On 11/26/13 12:24 PM, Dmitry Olshansky wrote:
> 26-Nov-2013 23:40, Andrei Alexandrescu пишет:
>> On 11/26/13 11:23 AM, Dmitry Olshansky wrote:
>>> See some hilarious things I had to do to link it properly:
>>> https://github.com/blackwhale/tools/blob/master/posix.mak#L35
>>
>> Interesting. Never knew of the pattern syntax in there.
>
> Me neither, must be some google-foo of mine.

Found it! http://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage

Thanks,

Andrei

On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu wrote:
> On 11/25/13 10:53 PM, H. S. Teoh wrote:
>> On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
>> [...]
>>> I don't think I understand having to build dmd just because you have
>>> a different distribution, Phobos maybe but dmd... Even for Phobos
>>> I'm skeptical but am not going to install a different distro to
>>> experiment myself.
>>
>> If you have used enough Linux distros (or even different versions of the
>> same distro!), you will realize that all it takes is for a single shared
>> library on the system to be a different version, missing, or just the
>> same version built with different compiler flags, and the executable
>> will not work. Not to mention that filesystem layout can be different
>> across distros (and versions of the same distro), which will break
>> things.
>>
>> Basically, to guarantee a program runs on distro X, the only way is it
>> has to be built from source. If you're lucky, somebody else has already
>> done that, and you can just download the binary. (Or better yet,
>> somebody packaged it for your distro, then you can just install it via
>> your system's package system.) But if not, you'll just have to do it
>> yourself.
>
> Do we with our std.net.curl make this matter any worse?
>
>>> If it is moved to a third party library, the problems will still
>>> remain. If the Phobos team is expected to support it, then the
>>> burden hasn't shifted either.
>>
>> This I agree with. However, it misses the point. The point is that the
>> core D toolchain (dmd/phobos/druntime) would have one less dependency,
>> which is generally a good thing, because you want to make it as easy as
>> possible for people to get a working compiler up and running.
>
> But my understanding is that the dependency is conditional, i.e. you have no dependency on libcurl unless you attempt to use std.net.curl. Which strikes me as entirely reasonable. No?
>
>> Requiring
>> this library and that library and that other tool, just to get a bare
>> bones D toolchain working, is a good way to turn people off D.
>
> Bare bones == no std.net.curl. No?
>
>> If a bare
>> bones D compiler is actually working, then they'd have more incentive to
>> actually put in more effort to make the frills work too. Having a hello
>> world program be uncompilable just because libcurl isn't available, is a
>> very bad first impression ("why does this stupid language need libcurl
>> just to compile a hello world program?!").
>
> My understanding is that a missing or broken libcurl does not break hello world. Is that correct?
>
> As a note regarding my many messages in this thread: I decided to get to the bottom of this given that I've asked "what is exactly the problem with libcurl" and all I got was unclear answers.
>
>
> Thanks,
>
> Andrei

In some cases, it can prevent the whole stuff to link. Otherwise, hello world should simply work.

On 11/26/13 2:04 PM, Adam D. Ruppe wrote:
> On Tuesday, 26 November 2013 at 21:33:50 UTC, Jonas Drewsen wrote:
>> But have a look at the libcurl API docs and think about how many lines
>> it would take to implement that in D.
>
> Meh, I don't think I've ever needed any of it. Set header and request
> data is enough to do anything on http.

That's quite a claim to make. I know very little on the topic, but from what I see at http://curl.haxx.se/changes.html it seems to me like a hard working professional is responding to real bug reports and feature requests from the field (as opposed to just toying with some sophisticated useless artifact).


Andrei

On 11/26/13 3:03 PM, deadalnix wrote:
> In some cases, it can prevent the whole stuff to link.

This is very vague. Are there cases in which something NOT using std.net.curl fails to link?

> Otherwise, hello world should simply work.

I guess that's where we want it.

My conclusion after this whole discussion: in the short term we need to have both Fedora and Debian builds. Then we'll move from there.


Andrei

On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
> As Jonathan M Davis said:
> ---
> Several of the main devs (including Walter) have stated that having std.net.curl on Phobos was a mistake given all of the problems that we've had with it on Windows and Linux as well, and at least some of them have expressed a desire for it to be removed. I expect that there's a good chance that it can and will be removed from Phobos if brought up for discussion.
>
> Certainly, I think that it's clear that we will not add any further external dependencies like curl, because curl has proven to be a big problem. It's very desirable to have bindings and wrappers for C libraries - but putting them in the standard library when it's not guaranteed that the appropriate library is on the system by default has proven to be too problematic, so they should be left to external projects.
> ---
>
> "std.net.curl" can also be moved from Phobos to Deimos.
> Deimos can be rethink, i.e. new build master can add a make building script on Deimos in order to compile all projects included on it, generating "libdeimos.a" and "libdeimos.so", documentation, etc.

We need something to replace std.net.curl with in the first place.

The problem is that if want to support for example HTTPS or FTPS we would end with dependencie (on openssl for example) anyways.

Another thing is that making anything more than trivial requests, requires a bit of work - compressions, encodings, authentication, RFC (*Grins) etc - it all makes it complex task.

Removing std.net.curl at this point will probably break a bit of code.
About cases where people have problems with linking when using std.net.curl for http - advise them to use Vibe.d http client or Vladimir's ae.net.* stuff or Adam's Ruppe custom framework? Dunno. Merge any of those 3 projects work into Phobos? Bounty for D implementation of libCURL - like for phobos inclusion?

On Tuesday, 26 November 2013 at 23:05:08 UTC, Andrei Alexandrescu wrote:
> That's quite a claim to make.

If you've used HTTP, you'll know how true it is. Indeed, since a request consists solely of a request and a body, being able to change them by definition gives you full access!

The most complex things in http are proxying and caching. curl does proxying, wich my 300 line http.d doesn't do. As far as I know, curl doesn't do caching at all. (My curl.d offers a cache functionality on top of curl, but it ignores the http headers, instead opting simply for a max age provided by the programmer, e.g. "refresh cache if the file is more than four hours old or doesn't exist, otherwise just return the cache file's contents".)

> but from what I see at http://curl.haxx.se/changes.html it seems to me like a hard working professional is responding to real bug reports and feature requests from the field

Yet, very little of that stuff is actually exposed through std.net.curl anyway, except perhaps through the download() and upload() functions. Phobos doesn't directly expose tftp, imap, pop, globbing, ntlm, nor most of the other things mentioned on that changelog.

SSL is legitimately a bit of a pain, but curl doesn't actually implement ssl, it just uses it, with a few different backends - openssl, gnutls, and so forth. Indeed, this is one of the difficulties in using curl with D, it is compiled with openssl but the client machine uses gnutls or whatever. We'd have that same problem.

Async requests are similar to ssl: it should implemented in a generic way for all I/O, then the http layer can just use that. It's probably just a matter of time before there's demand for an async file and socket facility in there anyway.


Again though, my position is that curl is ok and I use it myself, but let's not assume we need the entire kitchen sink that curl offers, especially given the fact that std.net.curl barely scratches it anyway. If someone does need one of curl's less common features, it is still there for them, but for the 99% we can do with the std.net.curl api, I'm sure we could make that happen in ~1000 lines of curl replacement.

On 11/26/13 3:40 PM, Adam D. Ruppe wrote:
> Again though, my position is that curl is ok and I use it myself, but
> let's not assume we need the entire kitchen sink that curl offers,
> especially given the fact that std.net.curl barely scratches it anyway.
> If someone does need one of curl's less common features, it is still
> there for them, but for the 99% we can do with the std.net.curl api, I'm
> sure we could make that happen in ~1000 lines of curl replacement.

Care to make it interesting? I may have some funds for this. But we're looking for no less than a glorious 100% replacement.

Andrei

On Tue, Nov 26, 2013 at 04:19:31PM -0800, Andrei Alexandrescu wrote:
> On 11/26/13 3:40 PM, Adam D. Ruppe wrote:
> >Again though, my position is that curl is ok and I use it myself, but let's not assume we need the entire kitchen sink that curl offers, especially given the fact that std.net.curl barely scratches it anyway.  If someone does need one of curl's less common features, it is still there for them, but for the 99% we can do with the std.net.curl api, I'm sure we could make that happen in ~1000 lines of curl replacement.
> 
> Care to make it interesting? I may have some funds for this. But we're looking for no less than a glorious 100% replacement.
[...]

This piqued my interest.

Please excuse another of my TL;DR stories / rants. Some time earlier this year, I wanted to write a simple utility to (1) login to a website with a preconfigured username/password; (2) navigate to a specific page on the site; (3) extract a specific HTML table from that page, and (4) parse and format the table into an ASCII-graphics text format. Sounds like the perfect job for a quick-n-dirty D program, right? In the old days, I'd hack up a Perl script to do this in a couple o' hours, but this time, I thought, since I have D at my disposal, let's do this the Right Way(tm).  Where in the past I'd just run wget or curl (the CLI utility) to fetch the web pages, I thought it'd be much cleaner to do the web interaction directly so that I didn't have to mess around with parsing temporary cookie files and what-not just to keep track of my login session.

So I scrolled through the Phobos docs, and lo and behold, there was std.net.curl sitting right there, looking all pretty and beckoning to me. I glanced over its API, and found that it was just the right fit for what I needed to do. Or so I thought...  About 30 minutes into the coding, I ran into a blocker: std.net.curl didn't let me specify Content-Type of POST requests!!! The server simply refused to accept anything that wasn't explicitly stated to be x/www-urlencoded, but std.net.curl only allows text/plain or application/octet-stream!  I simply couldn't login.  Thinking that I must've missed something obvious, I spent the rest of the day (and probably several more hours the next day) combing through the docs to find out how to set the Content-Type to something else. Finally I resorted to reading the std.net.curl source code, only to discover, to my dismay, that there was NO way to do this unless I bypassed all the nice high-level syntactic sugar, and got my hands down and dirty with low-level libcurl API calls (or their thin wrappers thereof).

At that point, I threw up my hands. The original task was far too simple for something this heavy-handed, so I backed off and said, it was nice knowing you, std.net.curl, but I think we're better off going our separate ways.

TL;DR: I would *love* to see a replacement for std.net.curl that isn't so straitjacketed that I can't even do something so basic as logging on to a website. I'm sure there are very good reasons why the current API is the way it is, but the fact of the matter is that, from an end-user's POV, its limitations were a deal-breaker.

So, I'm all for a D-only replacement of std.net.curl with a superior API. Getting rid of the external dependence on libcurl would be an added bonus.


T

-- 
"I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you already said that." -- User-Friendly

On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei Alexandrescu wrote:
> Care to make it interesting? I may have some funds for this. But we're looking for no less than a glorious 100% replacement.

Meh, my first choice is still to just bundle curl with phobos, like we do with zlib. It seems silly to me to want a glorious 100% replacement of an *open source* library.

Regardless, I probably wouldn't be the ideal choice for changing the implementation because I barely use the interface: I wrote my own http modules, including one that uses curl and one that doesn't, before std.net.curl came around. The phobos thing had no compelling benefit to me, so I never switched most my code.

Thus, I'm not very familiar with std.net.curl's strengths and weaknesses and would likely break it somehow while changing the implementation.


...unless doing a new interface is on the table too. Then, we can leave std.net.curl exactly how it is, so people who use it don't have broken code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are phased in for people who want them. I could get behind that.

On Wednesday, 27 November 2013 at 00:49:57 UTC, H. S. Teoh wrote:
> Sounds like the perfect job for a quick-n-dirty D program, right?

Oh yeah, my http.d and dom.d can make short work of that. I had a similar problem last year and ended up getting it to work very easily (well, that one used my curl.d, but http.d does cookies too now).

> coding, I ran into a blocker: std.net.curl didn't let me specify
> Content-Type of POST requests!!!

.....holy crap, you're not kidding. I thought addRequestHeader can do it, but that duplicates the header. Unbelievable.

My curl.d is an ugly piece of garbage. Seriously, I hate it, just look at this signature:

string curlAuth(string url, string data = null, string username = null, string password = null, string contentType = "application/x-www-form-urlencoded", string methodOverride = null, string[] customHeaders = null, string cookieJar = null) {

but at least it gets the job done - I use it on my work projects to access all kinds of things from arbitrary website downloads (one of the work sites offers a link posting functionality similar to Facebook, it downloads the site and reads description, images, etc. to make a richer link preview) to a variety of APIs like bing search, Facebook, twitter, linkedin, youtube, Authorize.net, I've done a lot with it, and especially my helper oauth.d file for many of them, all built on top of this hideous do-it-all function.

If I was doing a whole new interface, I think I'd prefer to do an object that keeps this state internally. So it'd be like:

auto client = new HttpClient("host", port);
client.userAgent = "whatever";
// authorization would ideally be a UFCS function or at least do some template method check, so it can be extended by things like oauth
client.authorization = BasicAuth("username", "password");

client.host = "host override if you want.com";

auto response = client.post("/login", ["username":"lol", "password":"rofl"]);

// or
client.post("/login", "text/plain", "text");

// or
client.post("/login", "text/xml", someXmlInputRange);

or something like that. Url and method could be set via properties too; get, post, head, put, delete, and so on could perhaps be implemented in terms of opDispatch that sets the properties then calls .execute();


Anyway, the important thing would be if you do future changes to this object, it maintains a bit of state. Cookies and such. That could work with async too, since the response returned would be more like a handle, so you then do

response.wait();

or do callbacks or whatever.


But the point is it'd be nice and simple for most series of requests.