Hello!

Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.

range algorithms can be applied to server responses, so that simple chain

getContentByLine("https://httpbin.org/anything")
    .map!"cast(string)a"
    .filter!(a => a.canFind("data"))

should work.

These calls work lazily so you can apply them to large documents.

dlang-requests - HTTP client library, inspired by python-requests with goals:

small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Always waiting for your bugreports and proposals on project page.

Best regards!

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
> Hello!
>
> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
>
> range algorithms can be applied to server responses, so that simple chain
>
> getContentByLine("https://httpbin.org/anything")
>     .map!"cast(string)a"
>     .filter!(a => a.canFind("data"))
>
> should work.
>
> These calls work lazily so you can apply them to large documents.
>
> dlang-requests - HTTP client library, inspired by python-requests with goals:
>
> small memory footprint
> performance
> simple, high level API
> native D implementation
>
> https://github.com/ikod/dlang-requests
> https://code.dlang.org/packages/requests
>
> Always waiting for your bugreports and proposals on project page.
>
> Best regards!

Nice work!

One quick suggestion: avoid direct casting from `ubyte[]` to `string`:

/+dub.sdl:
dependency "requests" version="~>1.1"
+/

void main()
{
    import requests : getContentByLine;
    import std : assumeUTF, canFind, each, filter, map, write;

    getContentByLine("https://httpbin.org/anything")
        .map!assumeUTF // instead of map!"cast(string)a"
        .filter!(a => a.canFind("data"))
        .each!write;
}

1. From a code-style point of view, assumeUTF is better as it shows the intention to the reader - assume that the content is valid UTF8 encoded text, without performing validation. And if there are UTF8 errors, it is easy to go back and add validation there.
2. Avoid casting mutable data to immutable. The data path in your library is rather complex (getContentByLine -> _LineReader -> LineSplitter -> Buffer -> ...) and so it was hard to understand from a quick glance whether or not the buffer array is reused (but I would guess that it is). If the buffer array is reused, it means that the result of calling _LineReader.front() may be modified at a later point in time, which I think is obvious that it can lead to some rather nasty bugs in users' code.

I suggest you look into Steven's iopipe[1] library, as I believe it can help you clean up and refactor this part of the codebase (and can probably yield some performance improvements along the way).

[1]: https://github.com/schveiguy/iopipe

On Sunday, 5 April 2020 at 11:53:29 UTC, Petar Kirov [ZombineDev] wrote:
> On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
>> Hello!
>>
>> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
>>
>> range algorithms can be applied to server responses, so that simple chain
>>
>> getContentByLine("https://httpbin.org/anything")
>>     .map!"cast(string)a"
>>     .filter!(a => a.canFind("data"))
>>
>> should work.
>>
>> These calls work lazily so you can apply them to large documents.
>>
>> dlang-requests - HTTP client library, inspired by python-requests with goals:
>>
>> small memory footprint
>> performance
>> simple, high level API
>> native D implementation
>>
>> https://github.com/ikod/dlang-requests
>> https://code.dlang.org/packages/requests
>>
>> Always waiting for your bugreports and proposals on project page.
>>
>> Best regards!
>
> Nice work!
>

I noticed that by default trace logging from requests is turned on, which is quite verbose. I checked the docs in the readme file and at first, I only saw suggestions on how to customize the logging by writing a LoggerInterceptor, which I thought was a bit too much for casual use. Only after a bit more digging I found that you were using std.experimental.logging (e.g. here [1]). Later I found that the SSL example in the readme used std.experimental.logging [2], but I think it would be better to explicitly mention it earlier on, e.g. like this:

/+dub.sdl:
dependency "requests" version="~>1.1"
+/

void main()
{
    import requests : getContentByLine;
    import std : assumeUTF, canFind, each, filter, map, write, writeln;

    /*
     * dlang_requests uses `std.experimental.logger` for interal logging.
     *
     * The globalLogLevel is set to `LogLevel.all` by default in Phobos, which
     * may be too verbose for most applications.
     *
     * This can be changed like this:
     */
    import std.experimental.logger : globalLogLevel, LogLevel;
    globalLogLevel = LogLevel.info;

    getContentByLine("https://httpbin.org/anything")
        .map!assumeUTF
        .filter!(a => a.canFind("data"))
        .each!write;
}


[1]: https://github.com/ikod/dlang-requests/blob/v1.1.0/source/requests/streams.d#L383

[2]: https://github.com/ikod/dlang-requests/blob/v1.1.0/README.md#ssl-settings

On Sunday, 5 April 2020 at 11:53:29 UTC, Petar Kirov [ZombineDev] wrote:
> On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
>> Hello!
>>
>> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
>>
>> range algorithms can be applied to server responses, so that simple chain
>>
>> getContentByLine("https://httpbin.org/anything")
>>     .map!"cast(string)a"
>>     .filter!(a => a.canFind("data"))
>>
>> should work.
>>
>> These calls work lazily so you can apply them to large documents.
>>
>> dlang-requests - HTTP client library, inspired by python-requests with goals:
>>
>> small memory footprint
>> performance
>> simple, high level API
>> native D implementation
>>
>> https://github.com/ikod/dlang-requests
>> https://code.dlang.org/packages/requests
>>
>> Always waiting for your bugreports and proposals on project page.
>>
>> Best regards!
>
> Nice work!
>
> One quick suggestion: avoid direct casting from `ubyte[]` to `string`:
>
> /+dub.sdl:
> dependency "requests" version="~>1.1"
> +/
>
> void main()
> {
>     import requests : getContentByLine;
>     import std : assumeUTF, canFind, each, filter, map, write;
>
>     getContentByLine("https://httpbin.org/anything")
>         .map!assumeUTF // instead of map!"cast(string)a"
>         .filter!(a => a.canFind("data"))
>         .each!write;
> }
>
> 1. From a code-style point of view, assumeUTF is better as it shows the intention to the reader - assume that the content is valid UTF8 encoded text, without performing validation. And if there are UTF8 errors, it is easy to go back and add validation there.
> 2. Avoid casting mutable data to immutable. The data path in your library is rather complex (getContentByLine -> _LineReader -> LineSplitter -> Buffer -> ...) and so it was hard to understand from a quick glance whether or not the buffer array is reused (but I would guess that it is). If the buffer array is reused, it means that the result of calling _LineReader.front() may be modified at a later point in time, which I think is obvious that it can lead to some rather nasty bugs in users' code.

Internally my code do not modify these data, but I understand your concern.

>
> I suggest you look into Steven's iopipe[1] library, as I believe it can help you clean up and refactor this part of the codebase (and can probably yield some performance improvements along the way).
>
> [1]: https://github.com/schveiguy/iopipe

Thanks! You are totally right regarding casting mutable to immutable. Base code was written when I did not understand this problem well enough. I know how to fix this but this probably require large enough rewrite. I hope I'll do this in some near future.

PS. I also have package which convert nogc unique mutable byte array to immutable byte array and attach this immutable data "as is" to buffer (which is essentally array of immutable(ubyte)[]). Then every operation on buffer (trim, split, search, etc) performed on immutable with zero copy and with warranties on data immutability.

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
> Hello!
>
> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
[...]

As always, thanks for your hard work! I don't use much more than the normal `get` and `receiveAsRange`, but it's been working well.

On Sunday, 5 April 2020 at 16:25:52 UTC, Anonymouse wrote:
> On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
>> Hello!
>>
>> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
> [...]
>
> As always, thanks for your hard work! I don't use much more than the normal `get` and `receiveAsRange`, but it's been working well.

Thanks for your support :) This work is pleasure for me, glad that it can be useful.

On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
> Hello!
>
> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
>
> range algorithms can be applied to server responses, so that simple chain
>
> getContentByLine("https://httpbin.org/anything")
>     .map!"cast(string)a"
>     .filter!(a => a.canFind("data"))
>
> should work.
>
> These calls work lazily so you can apply them to large documents.
>
> dlang-requests - HTTP client library, inspired by python-requests with goals:
>
> small memory footprint
> performance
> simple, high level API
> native D implementation
>
> https://github.com/ikod/dlang-requests
> https://code.dlang.org/packages/requests
>
> Always waiting for your bugreports and proposals on project page.
>
> Best regards!

Very nice to see requests for D.

It is written in README that it has a small memory footprint and performance but how does it compare against Python version?

I am asking because we have some Python packages using requests and I wanted to try to rewrite them in D. To avoid ppl raising eyebrows it would be nice to know a bit more details if possible.

On Tuesday, 12 May 2020 at 09:56:44 UTC, Pavel Shkadzko wrote:
> On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:
>> Hello!
>>
>> Just a note that dlang-requests ver 1.1.0 released with new 'ByLine' interfaces added for get/post/put requests.
>>
>> range algorithms can be applied to server responses, so that simple chain
>>
>> getContentByLine("https://httpbin.org/anything")
>>     .map!"cast(string)a"
>>     .filter!(a => a.canFind("data"))
>>
>> should work.
>>
>> These calls work lazily so you can apply them to large documents.
>>
>> dlang-requests - HTTP client library, inspired by python-requests with goals:
>>
>> small memory footprint
>> performance
>> simple, high level API
>> native D implementation
>>
>> https://github.com/ikod/dlang-requests
>> https://code.dlang.org/packages/requests
>>
>> Always waiting for your bugreports and proposals on project page.
>>
>> Best regards!
>
> Very nice to see requests for D.
>
> It is written in README that it has a small memory footprint and performance but how does it compare against Python version?

I never compared performance with Python requests, but I made some comparisons with
curl and wget, but results were never published. I just checked that there are no any odd delays.

There were some performance issues with connection pooling and caching redirects (like https://github.com/ikod/dlang-requests/issues/80), but they were fixed quite long ago.

>
> I am asking because we have some Python packages using requests and I wanted to try to rewrite them in D. To avoid ppl raising eyebrows it would be nice to know a bit more details if possible.

There are some functions of Python requests that were not implemented for dlang - for example, conversion to json and maybe a few others, but I can easily add them if someone really needs it.

At the same time I tried to adapt API to Dlang way - like using lazy ranges when sending or receiving data, and like lazy byLine interator over response body, so your code may look and perform better with dlang-requests than with python.


PS. I mentioned small memory footprint in README because, for some reason, I failed to keep memory usage in reasonable bounds with std.net.curl at the time I tried to use it.