On Wednesday, 19 April 2023 at 16:15:53 UTC, Adam D Ruppe wrote:
> On Tuesday, 18 April 2023 at 08:13:05 UTC, Dmitry Olshansky wrote:
>> This maps to D beautifully:
>
> I recently wrote a new stream class that works kinda like this and thought about doing put() and front etc, but I wanted heterogeneous types.
>
> Still, it kinda did work.
>
> https://github.com/adamdruppe/arsd/blob/master/core.d#L5040
>
> there's the test rn to show how it works


Cool stuff, as usual!

--
Dmitry Olshansky

On Tuesday, 18 April 2023 at 08:13:05 UTC, Dmitry Olshansky wrote:
> A streaming parser is doing 2 things:
> - waiting for input
> - processing the current input and spitting out Events
>
> This maps to D beautifully:
>
> 1. Waiting for input means it's an OutputRange with explicit put method!
>
> 2. Processing events means it's an InputRange of events. It may even be ForwardRange, mine first of this kind is Forward range.

This is how `requests` handle input stream from the remote end and convert it to "input stream" of http response body chunks.

Input is a stream of bytes, arbitrary splitted by remote end or by network, etc. Moreover,response content can be gzipped, chunk-encoded and whatever else. So you have to process each network input with several "processors", like "inflate", "chunk-encoded decoder" and so on.

      +------------+   +--------------+   +---------+   +-----------------+
      |input range |   | chunk-encoded|   |   gzip  |   | input range     |
Net->-| of byte[]  |->-|   decoder    |->-| decoder |->-| of response body|->- App
      |            |   |              |   |         |   | chunks: byte[]  |
      +------------+   +--------------+   +---------+   +-----------------+

There is some problems like push all unprocessed data to the right end of the picture at the end of the network stream and maybe some other... But essentially it works.

Immutability of underlying data on every step of this processing stream helps very much.