On Tue, Apr 18, 2023 at 08:13:05AM +0000, Dmitry Olshansky via Digitalmars-d wrote:
> A streaming parser is doing 2 things:
> - waiting for input
> - processing the current input and spitting out Events
> 
> This maps to D beautifully:
> 
> 1. Waiting for input means it's an OutputRange with explicit put method!
> 
> 2. Processing events means it's an InputRange of events. It may even be ForwardRange, mine first of this kind is Forward range.
[...]

In practice, the input would be an input range consumed by the parser which returns an input range of tokens.  IOW the parser is a filter that transforms an input range of characters into a range of tokens.

You'd only need an output range interface if you needed to explicitly feed characters to the parser, such as if you were using an async I/O API and you just received a block of fresh input, and now you need to stuff the input into the parser.  But even then, I wouldn't do it this way, because on the other end, the input range wouldn't be able to return a result if, say, the next block of input hasn't arrived yet. So .empty would have to block, which is an ugly design for something with an input range API.

At the end of the day, I think treating a parser as a transformation (range of char -> range of tokens) is a better abstraction than trying to shoehorn it into something involved output ranges.


T

-- 
Nearly all men can stand adversity, but if you want to test a man's character, give him power. -- Abraham Lincoln

On 4/18/23 7:24 AM, H. S. Teoh wrote:

> At the end of the day, I think treating a parser as a transformation
> (range of char -> range of tokens) is a better abstraction than trying
> to shoehorn it into something involved output ranges.

This is exactly how jsoniopipe works (though the input is not a "range" but an iopipe of course).

The resulting thing is essentially a "range", but not with the range functions. The implementation doesn't quite fit the range paradigm.

https://github.com/schveiguy/jsoniopipe

-Steve

On 4/18/2023 7:24 AM, H. S. Teoh wrote:
> In practice, the input would be an input range consumed by the parser
> which returns an input range of tokens.  IOW the parser is a filter that
> transforms an input range of characters into a range of tokens.

The lexer produces a stream of tokens, not the parser. The output of the parser is an AST.

On Tuesday, 18 April 2023 at 08:13:05 UTC, Dmitry Olshansky wrote:
> This maps to D beautifully:

I recently wrote a new stream class that works kinda like this and thought about doing put() and front etc, but I wanted heterogeneous types.

Still, it kinda did work.

https://github.com/adamdruppe/arsd/blob/master/core.d#L5040

there's the test rn to show how it works