I have been writing several lexers and parsers. The grammars I need to
parse are really complex, and consequently I didn't feel confident about
the code quality, especially in the lexers.  So I decided to jump on the functional progamming bandwagon to see if that would help.  It definitely
does help, there are fewer lines of code, and I feel better about the code
quality.  I started at the high level, and had the input buffer return a
range of characters, and the lexer return a range of tokens.  But when I got
down to the lower levels of building up tokens, I ran into a problem:

First I started with this which worked:

private void getNumber(MCInputStreamRange buf)
{
    while (!buf.empty())
    {
        p++;
        buf.popFront();
        if (buf.front() <= '0' || buf.front() >= '9') break;
        *p = buf.front();
    }
    curTok.kind = Token_t.NUMBER;
    curTok.image = cast(string) cbuffer[0 .. (p - cbuffer.ptr)].dup;
}

I thought I could improve this like so:

private void getNumber(MCInputStreamRange buf)
{
    auto s = buf.until("a <= '0' || a >= '9'");
    curTok.kind = Token_t.NUMBER;
    curTok.image = to!string(s);
}

The problem is that "until" seems to not stop at the end of the number,
and instead continues until the end of the buffer.  Am I doing something
wrong here?  Also, what is the fastest way to convert a range to a string?

Thanks,

Eric

Eric:

>     while (!buf.empty())
>     {
>         p++;
>         buf.popFront();

Those () can be omitted, if you mind the noise (but you can also keep them).


>         if (buf.front() <= '0' || buf.front() >= '9') break;

std.ascii.isDigit helps.


>     curTok.image = cast(string) cbuffer[0 .. (p - cbuffer.ptr)].dup;

If you want a string, then idup is better. Try to minimize the number of casts in your code.


>     auto s = buf.until("a <= '0' || a >= '9'");

Perhaps you need a ! after the until, or a !q{a <= '0' || a >= '9'}.


> Also, what is the fastest way to convert a range to a string?

The "text" function is the simplest.

Bye,
bearophile

On Tuesday, 22 July 2014 at 16:50:47 UTC, Eric wrote:
> private void getNumber(MCInputStreamRange buf)
> {
>     auto s = buf.until("a <= '0' || a >= '9'");
>     curTok.kind = Token_t.NUMBER;
>     curTok.image = to!string(s);
> }
>
> The problem is that "until" seems to not stop at the end of the number,
> and instead continues until the end of the buffer.  Am I doing something
> wrong here?

You've forgotten the exclamation mark: buf.until!(...)
Without it, the string is not the predicate, but the sentinel
value. I.e. the range stops when it sees the characters "a <= '0'
|| a >= '9'".

By the way, do you really mean to stop on '0' and '9'? Do you
perhaps mean "a < '0' || a > '9'"?

>  Also, what is the fastest way to convert a range to a string?

The fastest to type is probably text(r) (or r.text). The fastest
for me to come up with is r.to!string, which does exactly the
same. I don't know about run time, but text/to!string is
hopefully fine.

On Tuesday, 22 July 2014 at 17:09:29 UTC, bearophile wrote:
> Eric:
>
>>    while (!buf.empty())
>>    {
>>        p++;
>>        buf.popFront();
>
> Those () can be omitted, if you mind the noise (but you can also keep them).
>
>
>>        if (buf.front() <= '0' || buf.front() >= '9') break;
>
> std.ascii.isDigit helps.
>
>
>>    curTok.image = cast(string) cbuffer[0 .. (p - cbuffer.ptr)].dup;
>
> If you want a string, then idup is better. Try to minimize the number of casts in your code.
>
>
>>    auto s = buf.until("a <= '0' || a >= '9'");
>
> Perhaps you need a ! after the until, or a !q{a <= '0' || a >= '9'}.
>
>
>> Also, what is the fastest way to convert a range to a string?
>
> The "text" function is the simplest.
>
> Bye,
> bearophile

Thanks!  All very good suggestions...

-Eric

>
> By the way, do you really mean to stop on '0' and '9'? Do you
> perhaps mean "a < '0' || a > '9'"?
>

Yes, my bad...

On Tuesday, 22 July 2014 at 17:09:29 UTC, bearophile wrote:
> Eric:
>
>>    while (!buf.empty())
>>    {
>>        p++;
>>        buf.popFront();
>
> Those () can be omitted, if you mind the noise (but you can also keep them).

Actually, the ones behind `empty` and `front` are wrong, because these are defined to be properties. They just happen to work currently.