Hi,

   So I've spent some time trying to wrap libcurl for D. There is a lot of things that you can do with libcurl which I did not know so I'm starting out small.

For now I've created all the declarations for the latest public curl C api. I have put that in the etc.c.curl module.

On top of that I've created a more D like api as seen below. This is located in the 'etc.curl' module. What you can see below currently works but before proceeding further down this road I would like to get your comments on it.

//
// Simple HTTP GET with sane defaults
// provides the .content, .headers and .status
//
writeln( Http.get("http://www.google.com").content );

//
// GET with custom data receiver delegates
//
Http http = new Http("http://www.google.dk");
http.setReceiveHeaderCallback( (string key, string value) {
	writeln(key ~ ":" ~ value);
} );
http.setReceiveCallback( (string data) { /* drop */ } );
http.perform;

//
// POST with some timouts
//
http.setUrl("http://www.testing.com/test.cgi");
http.setReceiveCallback( (string data) { writeln(data); } );
http.setConnectTimeout(1000);
http.setDataTimeout(1000);
http.setDnsTimeout(1000);
http.setPostData("The quick....");
http.perform;

//
// PUT with data sender delegate
//
string msg = "Hello world";
size_t len = msg.length; /* using chuncked transfer if omitted */

http.setSendCallback( delegate size_t(char[] data) {
    if (msg.empty) return 0;
    auto l = msg.length;
    data[0..l] = msg[0..$];
    msg.length = 0;
    return l;
    },
    HttpMethod.put, len );
http.perform;

//
// HTTPS
//
writeln(Http.get("https://mail.google.com").content);

//
// FTP
//
writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
                "./downloaded-file"));


// ... authenication, cookies, interface select, progress callback
// etc. is also implemented this way.


/Jonas

I don't know much about this kind of stuff except that I use it for very simple use cases occasionally.  One thing I'll definitely give your design credit for, based on your examples, is making simple things simple.  I don't know how it scales to more complex use cases (not saying it doesn't, just that I'm not qualified to evaluate that), but I definitely would use this.  Nice work.

BTW, what is the license status of libcurl?  According to Wikipedia it's MIT licensed.  Where does that leave us with regard to the binary attribution issue?

== Quote from Jonas Drewsen (jdrewsen@nospam.com)'s article
> Hi,
>     So I've spent some time trying to wrap libcurl for D. There is a lot
> of things that you can do with libcurl which I did not know so I'm
> starting out small.
> For now I've created all the declarations for the latest public curl C
> api. I have put that in the etc.c.curl module.
> On top of that I've created a more D like api as seen below. This is
> located in the 'etc.curl' module. What you can see below currently works
> but before proceeding further down this road I would like to get your
> comments on it.
> //
> // Simple HTTP GET with sane defaults
> // provides the .content, .headers and .status
> //
> writeln( Http.get("http://www.google.com").content );
> //
> // GET with custom data receiver delegates
> //
> Http http = new Http("http://www.google.dk");
> http.setReceiveHeaderCallback( (string key, string value) {
> 	writeln(key ~ ":" ~ value);
> } );
> http.setReceiveCallback( (string data) { /* drop */ } );
> http.perform;
> //
> // POST with some timouts
> //
> http.setUrl("http://www.testing.com/test.cgi");
> http.setReceiveCallback( (string data) { writeln(data); } );
> http.setConnectTimeout(1000);
> http.setDataTimeout(1000);
> http.setDnsTimeout(1000);
> http.setPostData("The quick....");
> http.perform;
> //
> // PUT with data sender delegate
> //
> string msg = "Hello world";
> size_t len = msg.length; /* using chuncked transfer if omitted */
> http.setSendCallback( delegate size_t(char[] data) {
>      if (msg.empty) return 0;
>      auto l = msg.length;
>      data[0..l] = msg[0..$];
>      msg.length = 0;
>      return l;
>      },
>      HttpMethod.put, len );
> http.perform;
> //
> // HTTPS
> //
> writeln(Http.get("https://mail.google.com").content);
> //
> // FTP
> //
> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
>                  "./downloaded-file"));
> // ... authenication, cookies, interface select, progress callback
> // etc. is also implemented this way.
> /Jonas

On Fri, 11 Mar 2011 17:20:38 +0200, Jonas Drewsen <jdrewsen@nospam.com> wrote:

> writeln( Http.get("http://www.google.com").content );

Does this return a string? What if the page's encoding isn't UTF-8?

Data should probably be returned as void[], similar to std.file.read.

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

dsimcha wrote:

> I don't know much about this kind of stuff except that I use it for very
> simple
> use cases occasionally.  One thing I'll definitely give your design credit
> for,
> based on your examples, is making simple things simple.  I don't know how
> it scales to more complex use cases (not saying it doesn't, just that I'm
> not
> qualified to evaluate that), but I definitely would use this.  Nice work.
> 
> BTW, what is the license status of libcurl?  According to Wikipedia it's
> MIT
> licensed.  Where does that leave us with regard to the binary attribution
> issue?
> 

Walter contacted the author, it's not a problem:

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=112832

On 2011-03-11 16:20, Jonas Drewsen wrote:
> Hi,
>
> So I've spent some time trying to wrap libcurl for D. There is a lot of
> things that you can do with libcurl which I did not know so I'm starting
> out small.
>
> For now I've created all the declarations for the latest public curl C
> api. I have put that in the etc.c.curl module.
>
> On top of that I've created a more D like api as seen below. This is
> located in the 'etc.curl' module. What you can see below currently works
> but before proceeding further down this road I would like to get your
> comments on it.
>
> //
> // Simple HTTP GET with sane defaults
> // provides the .content, .headers and .status
> //
> writeln( Http.get("http://www.google.com").content );
>
> //
> // GET with custom data receiver delegates
> //
> Http http = new Http("http://www.google.dk");
> http.setReceiveHeaderCallback( (string key, string value) {
> writeln(key ~ ":" ~ value);
> } );
> http.setReceiveCallback( (string data) { /* drop */ } );
> http.perform;
>
> //
> // POST with some timouts
> //
> http.setUrl("http://www.testing.com/test.cgi");
> http.setReceiveCallback( (string data) { writeln(data); } );
> http.setConnectTimeout(1000);
> http.setDataTimeout(1000);
> http.setDnsTimeout(1000);
> http.setPostData("The quick....");
> http.perform;
>
> //
> // PUT with data sender delegate
> //
> string msg = "Hello world";
> size_t len = msg.length; /* using chuncked transfer if omitted */
>
> http.setSendCallback( delegate size_t(char[] data) {
> if (msg.empty) return 0;
> auto l = msg.length;
> data[0..l] = msg[0..$];
> msg.length = 0;
> return l;
> },
> HttpMethod.put, len );
> http.perform;
>
> //
> // HTTPS
> //
> writeln(Http.get("https://mail.google.com").content);
>
> //
> // FTP
> //
> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
> "./downloaded-file"));
>
>
> // ... authenication, cookies, interface select, progress callback
> // etc. is also implemented this way.
>
>
> /Jonas

Is there support for other HTTP methods/verbs in the D wrapper, like delete?

-- 
/Jacob Carlborg

I'll make some comments on the API. Do we have to choose Http/Ftp...? The URI already contains this, I could see being able to specifically request one or the other for performance or so www.google.com works.

And what about properties? They tend to be very nice instead of set methods. examples below.

Jonas Drewsen Wrote:

> //
> // Simple HTTP GET with sane defaults
> // provides the .content, .headers and .status
> //
> writeln( Http.get("http://www.google.com").content );
> 
> //
> // GET with custom data receiver delegates
> //
> Http http = new Http("http://www.google.dk");
> http.setReceiveHeaderCallback( (string key, string value) {
> 	writeln(key ~ ":" ~ value);
> } );
> http.setReceiveCallback( (string data) { /* drop */ } );
> http.perform;

http.onHeader = (string key, string value) {...};
http.onContent = (string data) { ... };
http.perform();

On 3/11/11 12:20 PM, Jonas Drewsen wrote:
> Hi,
>
> So I've spent some time trying to wrap libcurl for D. There is a lot of
> things that you can do with libcurl which I did not know so I'm starting
> out small.
>
> For now I've created all the declarations for the latest public curl C
> api. I have put that in the etc.c.curl module.
>
> On top of that I've created a more D like api as seen below. This is
> located in the 'etc.curl' module. What you can see below currently works
> but before proceeding further down this road I would like to get your
> comments on it.

I *love* it.

All APIs should be like yours. One-liners for what you want right now. If it's a little more complex, some more lines. This is perfect.

Congratulations!

Thank you.

Regarding scalability: In my experience the fastest network handling for multiple concurrent request is done asyncronously using select or epoll. The current wrapper would probably use threading and messages to handle multiple concurrent requests which is not as efficient.

Usually you only need this kind of scalability for server side networking and not client side like libcurl is providing so I do not see this as a major issue for an initial version.

I do know how to support epoll/select based curl and by that better scalability and that would fortunately just be an extension to the API I've shown. Currently I will focus on getting the common things finished and rock solid.

/Jonas


On 11/03/11 16.30, dsimcha wrote:
> I don't know much about this kind of stuff except that I use it for very simple
> use cases occasionally.  One thing I'll definitely give your design credit for,
> based on your examples, is making simple things simple.  I don't know how it
> scales to more complex use cases (not saying it doesn't, just that I'm not
> qualified to evaluate that), but I definitely would use this.  Nice work.
>
> BTW, what is the license status of libcurl?  According to Wikipedia it's MIT
> licensed.  Where does that leave us with regard to the binary attribution issue?
>
> == Quote from Jonas Drewsen (jdrewsen@nospam.com)'s article
>> Hi,
>>      So I've spent some time trying to wrap libcurl for D. There is a lot
>> of things that you can do with libcurl which I did not know so I'm
>> starting out small.
>> For now I've created all the declarations for the latest public curl C
>> api. I have put that in the etc.c.curl module.
>> On top of that I've created a more D like api as seen below. This is
>> located in the 'etc.curl' module. What you can see below currently works
>> but before proceeding further down this road I would like to get your
>> comments on it.
>> //
>> // Simple HTTP GET with sane defaults
>> // provides the .content, .headers and .status
>> //
>> writeln( Http.get("http://www.google.com").content );
>> //
>> // GET with custom data receiver delegates
>> //
>> Http http = new Http("http://www.google.dk");
>> http.setReceiveHeaderCallback( (string key, string value) {
>> 	writeln(key ~ ":" ~ value);
>> } );
>> http.setReceiveCallback( (string data) { /* drop */ } );
>> http.perform;
>> //
>> // POST with some timouts
>> //
>> http.setUrl("http://www.testing.com/test.cgi");
>> http.setReceiveCallback( (string data) { writeln(data); } );
>> http.setConnectTimeout(1000);
>> http.setDataTimeout(1000);
>> http.setDnsTimeout(1000);
>> http.setPostData("The quick....");
>> http.perform;
>> //
>> // PUT with data sender delegate
>> //
>> string msg = "Hello world";
>> size_t len = msg.length; /* using chuncked transfer if omitted */
>> http.setSendCallback( delegate size_t(char[] data) {
>>       if (msg.empty) return 0;
>>       auto l = msg.length;
>>       data[0..l] = msg[0..$];
>>       msg.length = 0;
>>       return l;
>>       },
>>       HttpMethod.put, len );
>> http.perform;
>> //
>> // HTTPS
>> //
>> writeln(Http.get("https://mail.google.com").content);
>> //
>> // FTP
>> //
>> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
>>                   "./downloaded-file"));
>> // ... authenication, cookies, interface select, progress callback
>> // etc. is also implemented this way.
>> /Jonas
>

On 11/03/11 19.31, Jacob Carlborg wrote:
> On 2011-03-11 16:20, Jonas Drewsen wrote:
>> Hi,
>>
>> So I've spent some time trying to wrap libcurl for D. There is a lot of
>> things that you can do with libcurl which I did not know so I'm starting
>> out small.
>>
>> For now I've created all the declarations for the latest public curl C
>> api. I have put that in the etc.c.curl module.
>>
>> On top of that I've created a more D like api as seen below. This is
>> located in the 'etc.curl' module. What you can see below currently works
>> but before proceeding further down this road I would like to get your
>> comments on it.
>>
>> //
>> // Simple HTTP GET with sane defaults
>> // provides the .content, .headers and .status
>> //
>> writeln( Http.get("http://www.google.com").content );
>>
>> //
>> // GET with custom data receiver delegates
>> //
>> Http http = new Http("http://www.google.dk");
>> http.setReceiveHeaderCallback( (string key, string value) {
>> writeln(key ~ ":" ~ value);
>> } );
>> http.setReceiveCallback( (string data) { /* drop */ } );
>> http.perform;
>>
>> //
>> // POST with some timouts
>> //
>> http.setUrl("http://www.testing.com/test.cgi");
>> http.setReceiveCallback( (string data) { writeln(data); } );
>> http.setConnectTimeout(1000);
>> http.setDataTimeout(1000);
>> http.setDnsTimeout(1000);
>> http.setPostData("The quick....");
>> http.perform;
>>
>> //
>> // PUT with data sender delegate
>> //
>> string msg = "Hello world";
>> size_t len = msg.length; /* using chuncked transfer if omitted */
>>
>> http.setSendCallback( delegate size_t(char[] data) {
>> if (msg.empty) return 0;
>> auto l = msg.length;
>> data[0..l] = msg[0..$];
>> msg.length = 0;
>> return l;
>> },
>> HttpMethod.put, len );
>> http.perform;
>>
>> //
>> // HTTPS
>> //
>> writeln(Http.get("https://mail.google.com").content);
>>
>> //
>> // FTP
>> //
>> writeln(Ftp.get("ftp://ftp.digitalmars.com/sieve.ds",
>> "./downloaded-file"));
>>
>>
>> // ... authenication, cookies, interface select, progress callback
>> // etc. is also implemented this way.
>>
>>
>> /Jonas
>
> Is there support for other HTTP methods/verbs in the D wrapper, like
> delete?
>

Yes.. all methods in libcurl are supported.

/Jonas

On 11/03/11 22.21, Jesse Phillips wrote:
> I'll make some comments on the API. Do we have to choose Http/Ftp...? The URI already contains this, I could see being able to specifically request one or the other for performance or so www.google.com works.

That is a good question.

The problem with creating a grand unified Curl class that does it all is that each protocol supports different things ie. http supports cookie handling and http redirection, ftp supports passive/active mode and dir listings and so on.

I think it would confuse the user of the API if e.g. he were allowed to set cookies on his ftp request.

The protocols supported (Http, Ftp,... classes) do have a base class Protocol that implements common things like timouts etc.


> And what about properties? They tend to be very nice instead of set methods. examples below.

Actually I thought off this and went the usual C++ way of _not_ using public properties but use accessor methods. Is public properties accepted as "the D way" and if so what about the usual reasons about why you should use accessor methods (like encapsulation and tolerance to future changes to the API)?

I do like the shorter onHeader/onContent much better though :)

/Jonas

> Jonas Drewsen Wrote:
>
>> //
>> // Simple HTTP GET with sane defaults
>> // provides the .content, .headers and .status
>> //
>> writeln( Http.get("http://www.google.com").content );
>>
>> //
>> // GET with custom data receiver delegates
>> //
>> Http http = new Http("http://www.google.dk");
>> http.setReceiveHeaderCallback( (string key, string value) {
>> 	writeln(key ~ ":" ~ value);
>> } );
>> http.setReceiveCallback( (string data) { /* drop */ } );
>> http.perform;
>
> http.onHeader = (string key, string value) {...};
> http.onContent = (string data) { ... };
> http.perform();