Downloading files over TLS. This seems that it's something that should be quite simple to do. My high level goals are cross-platform and easy distribution. I don't need anything fancy just a simple API like this:

download("https://url.com", "/local/file");

Because of these goals, I have a few requirements, which I don't think are unreasonable:

* Cross-platform (macOS, Linux, Windows, possible FreeBSD)

* Minimal runtime requirements (I don't mind a few compile time requirements). I also don't mind runtime requirements that are guaranteed to exist

* Ideally using the platform provided TLS implementation, or at least platform provided certificates (I've been bitten several times due to old certificates making request fail)

* On Linux I would really like to have a fully statically linked binary, this will make distribution easier

* Due to the previous requirement, I think using musl will be required. For most parts glibc works fine for static linking, but I think in the case of networking it does not work. The DNS lookup (or something like that) doesn't work with static linking

I think the main problem of all this is picking the right TLS implementation.

The obvious candidate is std.net.curl, but I'm going to explain why that is not ideal, at least not how it's currently implemented. std.net.curl relies on OpenSSL, which is deprecated on several platforms. On several platforms it's not the main, platform provided TLS implementation. As far as I know, it also provides its own certificates and doesn't rely on the ones provided by the platform.

I going to break down some alternatives per platform.

macOS:
* std.net.curl. Depends on libcurl and OpenSSL. OpenSSL has been deprecated on macOS for many years. In one wants to use OpenSSL anyway, one need to ship it with the binary (statically link it). For other problems, see above

* requests [1]. This is a potential better alternative than std.net.curl since it doesn't rely on libcurl. But, it still relies on OpenSSL, so it has the same problems as std.net.curl

* vibe.d [2]. Does not depend on libcurl, but does depend on OpenSSL. But, it also supports Botan, which is promising. I'll discuss Botan separately

* Hunt [3]. Same problem as the other ones, relies on OpenSSL

* Arsd [4]. Relies on OpenSSL

* SecureTransport [5]. This is an Apple specific library provided by the platform. It seems to more or less follow the same idea as OpenSSL, it's independent on the transport layer and works with BSD sockets. Initially this looks perfect, it's provided by the platform and uses certificates provided by the platform. The main problem is that it has been deprecated. It also only supports TLS up to version 1.2 and since it's deprecated, it's likely that it won't support any newer versions

* Network [6]. This is an Apple specific library provided by the platform. This is the recommend alternative to SecureTransport. The problem is that this is not just an alternative TLS implementation, it's a completely different alternative to BSD sockets. The API is completely different and will require some extra layers to to provide a cross-platform API. This means that I cannot use any of the existing library to just add TLS, it will be a completely different implementation, which might be ok. Another big problem is that it only available on macOS 10.14 and later. I have not decided yet if this is acceptable or not

* NSURLSession [7]. This is an  Apple specific Objective-C class, provided by the platform. It's higher level than the Network framework and is available on all versions of macOS. This will again require a some extra layers to hide the platform specific API. There's also a problem with LDC which has very limited support for Objective-C interoperability.

Linux:
As far as I know, OpenSSL (or some version of it, like LibreSSL) is what's provided by the platform. So I guess that has to be acceptable on Linux.

* std.net.curl. The main problem with std.net.curl on Linux is that it uses `dlopen` to load the TLS library. `dlopen` doesn't work for a statically linked executable. Side note, `dlopen` doesn't work on iOS (for other reasons) so it would be nice if this could be fixed

* requests. Seems to do that same as std.net.curl, uses `dlopen`

* For the other alternatives (mentioned in the macOS section), I haven't investigated if they use `dlopen` or not

Windows:
I don't know that much of this platform.

* std.net.curl and basically all other options already mentioned relies on OpenSSL, which is not provided by the platform

* SChannel. As far as I know, this the the platform provided implementation of TLS on Windows.

* Are there any high level APIs, like NSURLSession, on Windows that can be used to download files?

Botan:

This seems like a good candidate. It even exists a full D port of it [8].

* This is not used by any specific platform but it can use the platform provided certificates, which is a plus.

* The downside is that the code to access the platform provided certificates has not been ported to D (or it was added to a newer version of the C++ implementation).

* I think the the D port of Botan does not verify certificates, which is not great. It might be due to the previous point

* I also don't know if the D port is updated to match the latest version of the C++ implementation. By experience I know this can be a huge task, so I don't expect it

* Botan only supports up to TLS 1.2 (even the C++ implementation). Due to the previous point it's unclear if the D port of Botan will get support for TLS 1.3

I know there are other alternative TLS implementation, but I have not looked into detail on these.

A question regarding libcurl. I know that curl (the CLI tool) can take advantage of SecureTransport and SChannel as alternatives to OpenSSL. Due to std.net.curl containing some OpenSSL related code, is TLS handled completely outside of libcurl? So std.net.curl  can't easily take advantage of SecureTransport and SChannel the same way as curl does?

Is there anything obvious I'm missing or why does this seem so difficult? Do I have too high expectations and requirements?

[1] https://code.dlang.org/packages/requests
[2] https://code.dlang.org/packages/vibe-d
[3] https://code.dlang.org/packages/hunt
[4] https://code.dlang.org/packages/arsd-official%3Ahttp
[5] https://developer.apple.com/documentation/security/secure_transport
[6] https://developer.apple.com/documentation/network
[7] https://developer.apple.com/documentation/foundation/nsurlsession
[8] https://code.dlang.org/packages/botan

--
/Jacob Carlborg

On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:
> Downloading files over TLS. This seems that it's something that should be quite simple to do. My high level goals are cross-platform and easy distribution. I don't need anything fancy just a simple API like this:
>
> download("https://url.com", "/local/file");
>
....

> Is there anything obvious I'm missing or why does this seem so difficult? Do I have too high expectations and requirements?
>
> [1] https://code.dlang.org/packages/requests
> [2] https://code.dlang.org/packages/vibe-d
> [3] https://code.dlang.org/packages/hunt
> [4] https://code.dlang.org/packages/arsd-official%3Ahttp
> [5] https://developer.apple.com/documentation/security/secure_transport
> [6] https://developer.apple.com/documentation/network
> [7] https://developer.apple.com/documentation/foundation/nsurlsession
> [8] https://code.dlang.org/packages/botan
>
> --
> /Jacob Carlborg

Hello,

re `requests` - it uses dlopen (and variants for OSX and Windows, see https://github.com/ikod/dlang-requests/blob/master/source/requests/ssl_adapter.d#L50). The reason for dlopen is simple - compatibility with both openssl ver 1.0 and 1.1 (which have incompatible interfaces). To solve this problem I expose common interface for Requests internal needs, and detect and use different underlying openssl interfaces depending on library version.

I'm sure it is possible to detect library version at build time, and then use static linking. It just require some time investment to find most acceptable solution on how to do this.

Re Windows - I'd be happy to use native SecureChannel but I have zero windows programming experience. I'm completely open for any help in this field.

Bottom line - problem with SSL/TLS libraries lies in incompatibility between platforms and even inside the single library.

On Friday, 26 June 2020 at 11:10:27 UTC, ikod wrote:

> Hello,
>
> re `requests` - it uses dlopen (and variants for OSX and Windows, see https://github.com/ikod/dlang-requests/blob/master/source/requests/ssl_adapter.d#L50). The reason for dlopen is simple - compatibility with both openssl ver 1.0 and 1.1 (which have incompatible interfaces). To solve this problem I expose common interface for Requests internal needs, and detect and use different underlying openssl interfaces depending on library version.

Oh, it's that bad. That's disappointing.

> I'm sure it is possible to detect library version at build time, and then use static linking. It just require some time investment to find most acceptable solution on how to do this.

I'm using a script (written in D) in my tool, DStep [1][2], to identify which versions of libclang is available and to select between dynamic link (not using `dlopen`) and static linking of libclang. It works pretty well.

> Re Windows - I'd be happy to use native SecureChannel but I have zero windows programming experience. I'm completely open for any help in this field.

I don't have much experience with programming on Windows either. macOS is my main platform.

> Bottom line - problem with SSL/TLS libraries lies in incompatibility between platforms and even inside the single library.

Yes. I'm trying to identify the best solution, which ideally requires the least amount of work.

[1] https://github.com/jacob-carlborg/dstep/blob/master/configure.d
[2] https://github.com/jacob-carlborg/dstep/blob/master/dub.json#L34-L41

--
/Jacob Carlborg

On Friday, 26 June 2020 at 11:41:27 UTC, Jacob Carlborg wrote:
> On Friday, 26 June 2020 at 11:10:27 UTC, ikod wrote:
>
>> [...]
>
> Oh, it's that bad. That's disappointing.
>
>> [...]
>
> I'm using a script (written in D) in my tool, DStep [1][2], to identify which versions of libclang is available and to select between dynamic link (not using `dlopen`) and static linking of libclang. It works pretty well.
>

>
> [1] https://github.com/jacob-carlborg/dstep/blob/master/configure.d
> [2] https://github.com/jacob-carlborg/dstep/blob/master/dub.json#L34-L41

Thanks, will try to improve this (at least to add static openssl linking, as this is not first time somebody asked about it)

>
> --
> /Jacob Carlborg

On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:
> Downloading files over TLS. This seems that it's something that should be quite simple to do. My high level goals are cross-platform and easy distribution. I don't need anything fancy just a simple API like this:
>
> download("https://url.com", "/local/file");

Maybe just start wget or something like that?

> * Network [6]. This is an Apple specific library provided by the platform. This is the recommend alternative to SecureTransport. The problem is that this is not just an alternative TLS implementation, it's a completely different alternative to BSD sockets. The API is completely different and will require some extra layers to to provide a cross-platform API. This means that I cannot use any of the existing library to just add TLS, it will be a completely different implementation, which might be ok. Another big problem is that it only available on macOS 10.14 and later. I have not decided yet if this is acceptable or not

Since you want the latest certificate storage, you intend to support only the latest system. Many root certificates will timeout now.

> * Are there any high level APIs, like NSURLSession, on Windows that can be used to download files?

https://docs.microsoft.com/en-us/windows/win32/winhttp/about-winhttp

On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:
>
> Windows:
> I don't know that much of this platform.
>
> * std.net.curl and basically all other options already mentioned relies on OpenSSL, which is not provided by the platform
>
> * SChannel. As far as I know, this the the platform provided implementation of TLS on Windows.
>
> * Are there any high level APIs, like NSURLSession, on Windows that can be used to download files?
>

It is possible to statically link libcurl into your application. No need to use OpenSSL as libcurl can be built with SChannel.

On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:
> * Arsd [4]. Relies on OpenSSL

Yeah, I've been wanting to change that and use the native apis for years but like I just haven't been able to figure out the documentation of them.

Though for plain download, on Windows there's a high level function you can just call that's relatively easy... just the rest of the module needs low level access so it isn't super useful.

And of course Mac does it differently, I don't know much about there.

> The main problem is that it has been deprecated.

That's the problem of everything Apple makes. Kinda drives me nuts trying to keep up with their endless churn and constantly Think Different campaigns.

> * Are there any high level APIs, like NSURLSession, on Windows that can be used to download files?

https://docs.microsoft.com/en-us/windows/win32/winhttp/winhttp-sessions-overview

I guess that is more middle level but it isn't too hard to use. I think there's a flat out url -> bytes function too but I don't remember what it is.

Regardless, the Windows functions may look familiar if you have done AJAX - that was based on an IE object which was based on the Windows API.

On 2020-06-26 14:41, Kagamin wrote:

> Maybe just start wget or something like that?

The point was to avoid runtime dependencies.

> Since you want the latest certificate storage, you intend to support only the latest system. Many root certificates will timeout now.

I didn't say the latest certificate storage. I said the platform provided. If the system provided certificate expires I guess a lot of things won't work. An alternative would be to both ship with a certificate and be able to use the system provided one and try both

> https://docs.microsoft.com/en-us/windows/win32/winhttp/about-winhttp

I'll take a look, thanks.

-- 
/Jacob Carlborg

On 2020-06-26 14:43, User wrote:

> It is possible to statically link libcurl into your application. No need to use OpenSSL as libcurl can be built with SChannel.

That sounds good.

It looks like I remembered wrong. std.net.curl uses `dlopen` on libcurl, not OpenSSL. This might be less of an issue assuming libcurl is built with the platform provided TLS implementation. Just make sure it's possible to statically link libcurl, without using `dlopen`.

-- 
/Jacob Carlborg

On 2020-06-26 15:16, Adam D. Ruppe wrote:

> Yeah, I've been wanting to change that and use the native apis for years but like I just haven't been able to figure out the documentation of them.

That would be nice.

> That's the problem of everything Apple makes. Kinda drives me nuts trying to keep up with their endless churn and constantly Think Different campaigns.

Hehe, yeah.

> https://docs.microsoft.com/en-us/windows/win32/winhttp/winhttp-sessions-overview 
> 
> 
> I guess that is more middle level but it isn't too hard to use. I think there's a flat out url -> bytes function too but I don't remember what it is.
> 
> Regardless, the Windows functions may look familiar if you have done AJAX - that was based on an IE object which was based on the Windows API.

Thanks, I'll take a look.

-- 
/Jacob Carlborg