Thread overview
Payload Details with std.net.curl:post
Nov 27, 2021
Kyle Ingraham
Nov 27, 2021
frame
Nov 27, 2021
Kyle Ingraham
Nov 27, 2021
ikod
Nov 28, 2021
Kyle Ingraham
Nov 28, 2021
ikod
Nov 29, 2021
Kyle Ingraham
Nov 29, 2021
ikod
November 27, 2021

Happy Saturday everyone. I am using std.net.curl:post to request an OAuth access token from a Google API. Initially I was getting a 400 status code back but could not tell why. I used the verbose option on HTTP() to get more details:

char[] tokenResponse;
auto connection = HTTP();
connection.verbose = true;

try
{
    tokenResponse = post
    (
    tokenRequestEndpoint,
    ["code": incomingCode, "redirect_uri": tokenRequestRedirectUrl,
     "grant_type": "authorization_code", "code_verifier": to!string(codeVerifier),
     "client_id": clientId, "client_secret": clientSecret],
    connection
    );
}
catch(HTTPStatusException e)
{
    writefln("Token request failed - Status code: %d - Message: %s\n", e.status, e.msg);
    return 1;
}

That verbose option gets me a detailed log but I cannot see the payload of my request nor that of the response:

*   Trying 172.217.165.10:443...
* TCP_NODELAY set
* Connected to www.googleapis.com (172.217.165.10) port 443 (#0)
> POST /oauth2/v4/token HTTP/1.1
Host: www.googleapis.com
User-Agent: Phobos-std.net.curl/2.098 (libcurl/7.68.0)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Content-Length: 16
Content-Type: application/x-www-form-urlencoded

* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
< Vary: X-Origin
< Vary: Referer
< Date: Sat, 27 Nov 2021 14:24:45 GMT
< Server: scaffolding on HTTPServer2
< Cache-Control: private
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
* Connection #0 to host www.googleapis.com left intact

Is there a way to see that information? Google's API does not seem to provide much more than a status code as to the reason for the failure.

November 27, 2021

On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham wrote:

>

Is there a way to see that information? Google's API does not seem to provide much more than a status code as to the reason for the failure.

No, and it's a shame that we cannot use CurlOption.debugfunction because the Curl struct on HTTP is private and the only way to set this option is to use a raw Curl instance.

You are on your own working with onReceive(), setPostData() and std.uri.urlEncode() to encode your post data.

November 27, 2021

On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

>

On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham wrote:

>

Is there a way to see that information? Google's API does not seem to provide much more than a status code as to the reason for the failure.

No, and it's a shame that we cannot use CurlOption.debugfunction because the Curl struct on HTTP is private and the only way to set this option is to use a raw Curl instance.

You are on your own working with onReceive(), setPostData() and std.uri.urlEncode() to encode your post data.

I was afraid of that. Thank you for clarifying. I’ve tried other clients but they all seem to have trouble verifying HTTPS connections.

I’ll see about debugging with curl outside of D or go the manual route like you suggested.

November 27, 2021

On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham wrote:

>

On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

>

On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham wrote:

I was afraid of that. Thank you for clarifying. I’ve tried other clients but they all seem to have trouble verifying HTTPS connections.

Which clients did you try, and what the troubles they have?

November 28, 2021

On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

>

On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham wrote:

>

On Saturday, 27 November 2021 at 19:21:28 UTC, frame wrote:

>

On Saturday, 27 November 2021 at 17:41:55 UTC, Kyle Ingraham wrote:

I was afraid of that. Thank you for clarifying. I’ve tried other clients but they all seem to have trouble verifying HTTPS connections.

Which clients did you try, and what the troubles they have?

Hi ikod. I tried requests on Windows 10 (Build 19042) and got the following exception:

object.Exception@C:\Users\Kyle Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requests\streams.d(890): ssl connect failed: certificate verify failed

I checked the issues page for hunt-http and saw that there were similar reports there from Windows users (I have not verified them myself). I did not try vibe-http because the callback interface in the docs wasn't one that grabbed me.

I would have dug further into the solution for the requests exception as I like the interface it provides but I was eager to power through the OAuth bit of my prototyping so that I could experiment with the Google Photos API.

Do you have tips on why I might be getting that exception? I'm guessing I need to point requests at a trust store.

November 28, 2021

On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham wrote:

>

On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

>

On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham

Hi Kyle,

>
object.Exception@C:\Users\Kyle Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requests\streams.d(890): ssl connect failed: certificate verify failed

I checked the issues page for hunt-http and saw that there were similar reports there from Windows users (I have not verified them myself). I did not try vibe-http because the callback interface in the docs wasn't one that grabbed me.

Do you have tips on why I might be getting that exception? I'm guessing I need to point requests at a trust store.

Yes, trust store location can be the problem. I hope you checked this readme sections: windows-ssl-notes and ssl-settings.

It is difficult to answer without having more details, but I recently fixed a bug that could lead to a similar problem on 32-bit systems (regardless of OS). If this is your case, then you can try to use the latest commit from the github. Anyway, it would be nice if you create an issue on github and post more details there, so that I can try to reproduce the problem.

November 29, 2021

On Sunday, 28 November 2021 at 07:27:35 UTC, ikod wrote:

>

On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham wrote:

>

On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

>

On Saturday, 27 November 2021 at 20:31:16 UTC, Kyle Ingraham

Hi Kyle,

>
object.Exception@C:\Users\Kyle Ingraham\AppData\Local\dub\packages\requests-2.0.2\requests\source\requests\streams.d(890): ssl connect failed: certificate verify failed

I checked the issues page for hunt-http and saw that there were similar reports there from Windows users (I have not verified them myself). I did not try vibe-http because the callback interface in the docs wasn't one that grabbed me.

Do you have tips on why I might be getting that exception? I'm guessing I need to point requests at a trust store.

Yes, trust store location can be the problem. I hope you checked this readme sections: windows-ssl-notes and ssl-settings.

It is difficult to answer without having more details, but I recently fixed a bug that could lead to a similar problem on 32-bit systems (regardless of OS). If this is your case, then you can try to use the latest commit from the github. Anyway, it would be nice if you create an issue on github and post more details there, so that I can try to reproduce the problem.

I had read those sections but couldn't figure out which combination of certificates and function calls would make a difference for my URL.

I found https://pki.goog/repository/ which has links to all certificates relevant to Google URLs. I worked my way down the certificate chain and the call that worked was:

auto rq = Request();
rq.sslSetCaCert(r"gtsr1.pem");

That certificate is for one of Google's root CAs but not the root of the certificate chain. With that combination my request worked and the payload was delivered successfully. I was also able to see the payload that had been sent in my request which was my motivation for starting this thread in the first place.

I think my trouble here was mostly due to my limited familiarity with SSL verification. I ended up using a CA certificate and the method made available was sslSetCaCert. A bit obvious when I look back. I had not used it at first because it was referenced as being needed for vibe.d and when perusing requests I thought that it wasn't being called for my setup.

Glad to have a working setup now. Thank you ikod for stopping by and asking about my experiences outside of std.net.curl.

P.S. openssl s_client -connect <URL>:443 was useful for viewing the certificate chain.

November 29, 2021

On Monday, 29 November 2021 at 01:49:37 UTC, Kyle Ingraham wrote:

>

On Sunday, 28 November 2021 at 07:27:35 UTC, ikod wrote:

>

On Sunday, 28 November 2021 at 01:06:45 UTC, Kyle Ingraham wrote:

>

On Saturday, 27 November 2021 at 22:18:48 UTC, ikod wrote:

Hi Kyle!

>

I found https://pki.goog/repository/ which has links to all certificates relevant to Google URLs. I worked my way down the certificate chain and the call that worked was:

auto rq = Request();
rq.sslSetCaCert(r"gtsr1.pem");

Nice to hear that you solved this problem! Openssl under windows sometimes require
extra efforts. I tried to add support for native windows SecureChannel API in requests, but got stuck somewhere in the middle of the process.