Menu
Search

Forums

Forum Index

Thread overview
uploading with curl
Apr 06, 2012
Gleb
Apr 07, 2012
Jonas Drewsen
Apr 09, 2012
Gleb
Apr 17, 2012
Gleb
Apr 17, 2012
Jonas Drewsen
Apr 19, 2012
Gleb
Apr 30, 2012
Jonas Drewsen
May 04, 2012
Gleb
April 06, 2012
Gravatar of GlebPosted by GlebReply
Gleb
Gravatar of Gleb


Reply
Hello guys,

I'm trying to use curl library to satisfy my file transfer needs
under Windows 7. I've spent all the day and the most of
functionality I have already tried works like a charm. But I have
a few issues with "upload" function.

First of all, if I try to use something like:
   auto client = FTP("192.168.110.58");
or:
   upload!FTP("file.zip", "192.168.110.58");
curl wrapper does not understand we are trying to use
ftp-protocol and uses http instead, returning something like:
   <?xml version="1.0" encoding="iso-8859-1"?>
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
lang="en">
    <head>
     <title>400 - Bad Request</title>
    </head>
    <body>
     <h1>400 - Bad Request</h1>
    </body>
   </html>
Not a big deal, I'll use "ftp://xx.xx.xx.xx" format everywhere
below.

Here is the code I'm trying to use to upload a local file to the
ftp-host with an authentication:
   auto client = FTP();
   client.setAuthentication("login", "pass");
   upload!FTP("file.zip", "ftp://192.168.110.58/file.zip",
client);
This will pass the authentication but won't upload the file.

Then I decided to take a look to the code of std.net.curl.upload
function and use a low-level API the same way to find the
solution. Here is what I got:
   auto f = new std.stream.BufferedFile("file.zip", FileMode.In);
   scope (exit) f.close();
   auto client = FTP("ftp://192.168.110.58");
   client.verbose(true);
   client.setAuthentication("login", "pass");
   client.onSend = (void[] data)
   {
      return f.read(cast(ubyte[])data);
   };
   client.contentLength = cast(size_t)f.size;
   client.perform();
It's basically the same as "upload" function. This authenticates
correctly, gets directory listing and then nothing happens:
   * Connection #0 to host 192.168.110.58 left intact
   > QUIT
   < 221 Goodbye.
   * Closing connection #0
And it looks correct for me, why should it upload any file!?

So I decided to replace the last line of the code with the
following:
   client.addCommand("epsv");
   client.addCommand("stor file.zip");
   client.perform();
Here are the results:
   > epsv
   < 229 Entering Extended Passive Mode (|||12761|)
   > stor file.zip
   * FTP response timeout
   * Connection #0 to host 192.168.110.58 left intact
   * Timeout was reached
   > QUIT
   * server response timeout
   * Closing connection #0
   std.net.curl.CurlTimeoutException@std\net\curl.d(3333):
6B2BD8C6 on handle 166CB60
This way the file was created on the server, but it's empty. It
looks like client.onSend statements are never executed.
Unfortunately, I didn't managed to find out why, it's somewhere
in object "private RefCounted!Impl p" (curl.d:1956), but I didn't
find where it is.

So, what am I doing wrong? Does std.net.curl.upload work for you
correctly? How do I upload a file to the ftp-host with
authentication? What is "RefCounted!Impl p" and where do I find
it's "p.curl.perform()" method?

P.S.: Firewall is not the case, other ftp clients and
std.net.curl.download function work fine, rules for the program
are created (just in case).

Thank you in advance, any advice is really appreciated.
April 07, 2012
Gravatar of Jonas DrewsenPosted by Jonas Drewsen
in reply to Gleb		Reply
Jonas Drewsen
Gravatar of Jonas Drewsen
Posted in reply to Gleb


Reply
Answers below:

On Friday, 6 April 2012 at 13:24:03 UTC, Gleb wrote:
> Hello guys,
>
> I'm trying to use curl library to satisfy my file transfer needs
> under Windows 7. I've spent all the day and the most of
> functionality I have already tried works like a charm. But I have
> a few issues with "upload" function.
>
> First of all, if I try to use something like:
>    auto client = FTP("192.168.110.58");
> or:
>    upload!FTP("file.zip", "192.168.110.58");
> curl wrapper does not understand we are trying to use
> ftp-protocol and uses http instead, returning something like:
>    <?xml version="1.0" encoding="iso-8859-1"?>
>    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
>
> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
>    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
> lang="en">
>     <head>
>      <title>400 - Bad Request</title>
>     </head>
>     <body>
>      <h1>400 - Bad Request</h1>
>     </body>
>    </html>
> Not a big deal, I'll use "ftp://xx.xx.xx.xx" format everywhere
> below.

This is not how it should be. Github url with fixes below...

> Here is the code I'm trying to use to upload a local file to the
> ftp-host with an authentication:
>    auto client = FTP();
>    client.setAuthentication("login", "pass");
>    upload!FTP("file.zip", "ftp://192.168.110.58/file.zip",
> client);
> This will pass the authentication but won't upload the file.
>
> Then I decided to take a look to the code of std.net.curl.upload
> function and use a low-level API the same way to find the
> solution. Here is what I got:
>    auto f = new std.stream.BufferedFile("file.zip", FileMode.In);
>    scope (exit) f.close();
>    auto client = FTP("ftp://192.168.110.58");
>    client.verbose(true);
>    client.setAuthentication("login", "pass");
>    client.onSend = (void[] data)
>    {
>       return f.read(cast(ubyte[])data);
>    };
>    client.contentLength = cast(size_t)f.size;
>    client.perform();
> It's basically the same as "upload" function. This authenticates
> correctly, gets directory listing and then nothing happens:
>    * Connection #0 to host 192.168.110.58 left intact
>    > QUIT
>    < 221 Goodbye.
>    * Closing connection #0
> And it looks correct for me, why should it upload any file!?

I can reproduce this and included the fix for it.

> So I decided to replace the last line of the code with the
> following:
>    client.addCommand("epsv");
>    client.addCommand("stor file.zip");
>    client.perform();
> Here are the results:
>    > epsv
>    < 229 Entering Extended Passive Mode (|||12761|)
>    > stor file.zip
>    * FTP response timeout
>    * Connection #0 to host 192.168.110.58 left intact
>    * Timeout was reached
>    > QUIT
>    * server response timeout
>    * Closing connection #0
>    std.net.curl.CurlTimeoutException@std\net\curl.d(3333):
> 6B2BD8C6 on handle 166CB60
> This way the file was created on the server, but it's empty. It
> looks like client.onSend statements are never executed.
> Unfortunately, I didn't managed to find out why, it's somewhere
> in object "private RefCounted!Impl p" (curl.d:1956), but I didn't
> find where it is.
>
> So, what am I doing wrong? Does std.net.curl.upload work for you
> correctly? How do I upload a file to the ftp-host with
> authentication? What is "RefCounted!Impl p" and where do I find
> it's "p.curl.perform()" method?

RefCount!Impl p is a reference counted struct Impl defined above the declaration in the source file.

The p.curl is an instance of the Curl struct defined later in the source file. That struct has a perform() method.

> P.S.: Firewall is not the case, other ftp clients and
> std.net.curl.download function work fine, rules for the program
> are created (just in case).
>
> Thank you in advance, any advice is really appreciated.

You have identified a couple of bugs. A corrected version of curl.d is located at https://github.com/jcd/phobos/blob/curlfixes/std/net/curl.d

I've created a pull requests to get it upstream.

/Jonas
April 07, 2012
Gravatar of Andrei AlexandrescuPosted by Andrei Alexandrescu
in reply to Jonas Drewsen		Reply
Andrei Alexandrescu
Gravatar of Andrei Alexandrescu
Posted in reply to Jonas Drewsen


Reply
On 4/7/12 3:14 PM, Jonas Drewsen wrote:
> I've created a pull requests to get it upstream.

Merged: https://github.com/D-Programming-Language/phobos/pull/528

Andrei
April 09, 2012
Gravatar of GlebPosted by Gleb
in reply to Jonas Drewsen		Reply
Gleb
Gravatar of Gleb
Posted in reply to Jonas Drewsen


Reply
Woks perfectly! Thanks a lot!
April 17, 2012
Gravatar of GlebPosted by Gleb
in reply to Gleb		Reply
Gleb
Gravatar of Gleb
Posted in reply to Gleb


Reply
Gentlemen,

While working with curl library, I would like to discuss a few more issues I faced with, if you don't mind.

Here is the code sample I use:

    auto f = new std.stream.BufferedFile("file.zip", FileMode.In);
    scope (exit) f.close();
    auto client = FTP();
    client.verbose(true);
    client.url = "ftp://192.168.110.58/file.zip";
    client.setAuthentication("user", "pass");
    client.contentLength = cast(size_t)f.size;
    client.handle.set(CurlOption.upload, 1L);
    client.onSend = (void[] data)
    {
        return f.read(cast(ubyte[])data);
    };
    client.perform();

1. TIMEOUTS

The code sample will work great if file.zip is small.
If the file is big enough, we will get the following result:

   * Connecting to 192.168.110.58 (192.168.110.58) port 1468
   > TYPE I
   < 200 Type set to I
   > STOR file.zip
   < 150 Opening BINARY mode data connection for file.zip
   * Operation timed out after 120105 milliseconds with 687751168 bytes received
   * Closing connection #0
   * Timeout was reached std.net.curl.CurlTimeoutException@std\net\curl.d(3348): Timeout was reached on handle 16BCB60

So we were disconnected from server because of the timeout. To avoid this behavior I use the following:

   client.dataTimeout(dur!"weeks"(10));

I'm not sure if we really need this kind of timeout so I use the really big value. Anyway, I believe the default value of 120 seconds is not enough for practical usage of file transfer protocol. Maybe it's suitable for HTTP only?

By the way, for some reason I can't use the value more then 10 weeks. In this case we can't even connect to server:

   * About to connect() to 192.168.110.58 port 21 (#0)
   *   Trying 192.168.110.58...
   * connected
   * Connected to 192.168.110.58 (192.168.110.58) port 21 (#0)
   * server response timeout
   * Closing connection #0
   * Timeout was reached
   std.net.curl.CurlTimeoutException@std\net\curl.d(3348): Timeout was reached on handle 164CB60

Of course, even bigger value is not necessary, but I'm not sure this behavior is correct.


2. PROGRESSBAR

I've added the following to the above example:
    client.onProgress = delegate int(size_t dlTotal, size_t dlNow, size_t ulTotal, size_t ulNow)
    {
        return 0;
    };

When compiled and run we will see the following:

    > STOR file.zip
    < 150 Opening BINARY mode data connection for file.zip
    * We are completely uploaded and fine
    * Remembering we are in dir ""
    < 226 Transfer complete
    * Connection #0 to host 192.168.110.58 left intact
    > QUIT
    std.net.curl.CurlException@std\net\curl.d(3365): Progress callback called on cleaned up Curl instance

It looks like onProgress was called when the file was uploaded and the data connection was closed. What am I doing wrong? Should I return any other value when ulNow == ulTotal? In code examples in the documentation onProgress does not return any value, but it must.

Maybe it would be better to use some simple progress bar by default when curl.verbose was true? At least something like this:

    static const bStr = replicate("\b", 20);
    static float perc;
    client.onProgress = delegate int(size_t dlTotal, size_t dlNow, size_t ulTotal, size_t ulNow)
    {
        perc = ulTotal ? cast(real)ulNow/cast(real)ulTotal*100 : 0;
        writef("%s* %.2f%c uploaded", bStr, perc, '%');
        return 0;
    };


Thank you in advance for your opinion on this issues.
April 17, 2012
Gravatar of Jonas DrewsenPosted by Jonas Drewsen
in reply to Gleb		Reply
Jonas Drewsen
Gravatar of Jonas Drewsen
Posted in reply to Gleb


Reply
On Tuesday, 17 April 2012 at 07:07:06 UTC, Gleb wrote:
> Gentlemen,
>
> While working with curl library, I would like to discuss a few more issues I faced with, if you don't mind.
>
> Here is the code sample I use:
>
>     auto f = new std.stream.BufferedFile("file.zip", FileMode.In);
>     scope (exit) f.close();
>     auto client = FTP();
>     client.verbose(true);
>     client.url = "ftp://192.168.110.58/file.zip";
>     client.setAuthentication("user", "pass");
>     client.contentLength = cast(size_t)f.size;
>     client.handle.set(CurlOption.upload, 1L);
>     client.onSend = (void[] data)
>     {
>         return f.read(cast(ubyte[])data);
>     };
>     client.perform();
>
> 1. TIMEOUTS
>
> The code sample will work great if file.zip is small.
> If the file is big enough, we will get the following result:
>
>    * Connecting to 192.168.110.58 (192.168.110.58) port 1468
>    > TYPE I
>    < 200 Type set to I
>    > STOR file.zip
>    < 150 Opening BINARY mode data connection for file.zip
>    * Operation timed out after 120105 milliseconds with 687751168 bytes received
>    * Closing connection #0
>    * Timeout was reached std.net.curl.CurlTimeoutException@std\net\curl.d(3348): Timeout was reached on handle 16BCB60
>
> So we were disconnected from server because of the timeout. To avoid this behavior I use the following:
>
>    client.dataTimeout(dur!"weeks"(10));
>
> I'm not sure if we really need this kind of timeout so I use the really big value. Anyway, I believe the default value of 120 seconds is not enough for practical usage of file transfer protocol. Maybe it's suitable for HTTP only?
>
> By the way, for some reason I can't use the value more then 10 weeks. In this case we can't even connect to server:
>
>    * About to connect() to 192.168.110.58 port 21 (#0)
>    *   Trying 192.168.110.58...
>    * connected
>    * Connected to 192.168.110.58 (192.168.110.58) port 21 (#0)
>    * server response timeout
>    * Closing connection #0
>    * Timeout was reached
>    std.net.curl.CurlTimeoutException@std\net\curl.d(3348): Timeout was reached on handle 164CB60
>
> Of course, even bigger value is not necessary, but I'm not sure this behavior is correct.

This is one of the many reasons I believe we should do our own
network library. Curl only support timeouts for connecting and
for the entire transfer. What you really want is better control
with sane defaults: DNS lookup timeout, connect timeout,
read/write activity timeout and timout for the entire operation.

Default "entire operation" timeout should be infinity and the
rest just some qualified guesses.

Anyway... I think the current default timeouts are how they
should be.

Regarding the 10 week limit you're mentioning please see the docs
for the Duration type:
http://dlang.org/phobos/core_time.html#Duration


>
> 2. PROGRESSBAR
>
> I've added the following to the above example:
>     client.onProgress = delegate int(size_t dlTotal, size_t dlNow, size_t ulTotal, size_t ulNow)
>     {
>         return 0;
>     };
>
> When compiled and run we will see the following:
>
>     > STOR file.zip
>     < 150 Opening BINARY mode data connection for file.zip
>     * We are completely uploaded and fine
>     * Remembering we are in dir ""
>     < 226 Transfer complete
>     * Connection #0 to host 192.168.110.58 left intact
>     > QUIT
>     std.net.curl.CurlException@std\net\curl.d(3365): Progress callback called on cleaned up Curl instance
>
> It looks like onProgress was called when the file was uploaded and the data connection was closed. What am I doing wrong? Should I return any other value when ulNow == ulTotal? In code examples in the documentation onProgress does not return any value, but it must.
>
> Maybe it would be better to use some simple progress bar by default when curl.verbose was true? At least something like this:
>
>     static const bStr = replicate("\b", 20);
>     static float perc;
>     client.onProgress = delegate int(size_t dlTotal, size_t dlNow, size_t ulTotal, size_t ulNow)
>     {
>         perc = ulTotal ? cast(real)ulNow/cast(real)ulTotal*100 : 0;
>         writef("%s* %.2f%c uploaded", bStr, perc, '%');
>         return 0;
>     };
>
>
> Thank you in advance for your opinion on this issues.

You should always return 0 if you do not want to abort the job.
What you're describing sounds like a bug and I'll have a look at
it.

I think that including the progress in the verbose mode per
default will generate too much noise.

/Jonas
April 19, 2012
Gravatar of GlebPosted by Gleb
in reply to Jonas Drewsen		Reply
Gleb
Gravatar of Gleb
Posted in reply to Jonas Drewsen


Reply
Jonas, thanks for your answer.

On Tuesday, 17 April 2012 at 20:03:57 UTC, Jonas Drewsen wrote:
> This is one of the many reasons I believe we should do our own
> network library. Curl only support timeouts for connecting and
> for the entire transfer. What you really want is better control
> with sane defaults: DNS lookup timeout, connect timeout,
> read/write activity timeout and timout for the entire operation.
>
> Default "entire operation" timeout should be infinity and the
> rest just some qualified guesses.
>
> Anyway... I think the current default timeouts are how they
> should be.
There are DNS timeout and connect timeout in current std.net.curl. The entire operation timeout is infinity means there is no such timeout - this is what curl library offers.
There is also some kind of read/write activity "timeout" in the library because the connection will be dropped if ftp-server shutdowns suddenly during uploading. That's why I don't see why we have to have the data connection timeout which does not allow the big files to be downloaded or uploaded.

> Regarding the 10 week limit you're mentioning please see the docs
> for the Duration type:
> http://dlang.org/phobos/core_time.html#Duration
Thank's for the link! I've read it twice but unfortunately the reason of 10 weeks limit is still not clear for me. Moreover I can't understand why would dataTimeout influences connectTimeout in such way if the value I use is more then 10 weeks.

> What you're describing sounds like a bug and I'll have a look at
> it.
Thank you very much! Your contribution to D's curl library support is outstanding.
April 30, 2012
Gravatar of Jonas DrewsenPosted by Jonas Drewsen
in reply to Gleb		Reply
Jonas Drewsen
Gravatar of Jonas Drewsen
Posted in reply to Gleb


Reply
On Thursday, 19 April 2012 at 07:26:44 UTC, Gleb wrote:
> Jonas, thanks for your answer.
>
> On Tuesday, 17 April 2012 at 20:03:57 UTC, Jonas Drewsen wrote:
>> This is one of the many reasons I believe we should do our own
>> network library. Curl only support timeouts for connecting and
>> for the entire transfer. What you really want is better control
>> with sane defaults: DNS lookup timeout, connect timeout,
>> read/write activity timeout and timout for the entire operation.
>>
>> Default "entire operation" timeout should be infinity and the
>> rest just some qualified guesses.
>>
>> Anyway... I think the current default timeouts are how they
>> should be.
> There are DNS timeout and connect timeout in current std.net.curl. The entire operation timeout is infinity means there is no such timeout - this is what curl library offers.
> There is also some kind of read/write activity "timeout" in the library because the connection will be dropped if ftp-server shutdowns suddenly during uploading. That's why I don't see why we have to have the data connection timeout which does not allow the big files to be downloaded or uploaded.

Sorry for the delayed answer.

If the ftp server shuts down the tcp connection is broken and you will get notified immediately. This has nothing to do with timeouts (except maybe tcp timeouts in some cases which are normally handled by the OS).

Most other libraries have a default timeout afaik and that makes sense to me. As stated in my last reply the real solution would to have an activity timeout which would make it reasonable to set an infinite timeout for the entire transfer.

>> Regarding the 10 week limit you're mentioning please see the docs
>> for the Duration type:
>> http://dlang.org/phobos/core_time.html#Duration
> Thank's for the link! I've read it twice but unfortunately the reason of 10 weeks limit is still not clear for me. Moreover I can't understand why would dataTimeout influences connectTimeout in such way if the value I use is more then 10 weeks.

I've not read the code for Duration but my guess is that it wraps around and becomes a negative duration if you exceed the limit. This would of course make it timeout immediately.


>> What you're describing sounds like a bug and I'll have a look at
>> it.
> Thank you very much! Your contribution to D's curl library support is outstanding.

Just trying to make D a better place to be as so many others do :)

/Jonas
May 04, 2012
Gravatar of GlebPosted by Gleb
in reply to Jonas Drewsen		Reply
Gleb
Gravatar of Gleb
Posted in reply to Jonas Drewsen


Reply
Jonas, thank you for your answer.