Hi,

Looking at the DMD Source Guide it says "The lexer transforms the file into an array of tokens."

Why is this step taken instead of, say, just calling a function that returns the next token (or however many required for the look-ahead)?

Regards,
  -=mike=-

On Fri, 29 Aug 2014 13:41:20 +0000
Mike James via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com>
wrote:

> Looking at the DMD Source Guide it says "The lexer transforms the file into an array of tokens."
> 
> Why is this step taken instead of, say, just calling a function that returns the next token (or however many required for the look-ahead)?
D parser is somewhat complicated and it can peek alot of tokens from input stream without consuming 'em. so it's just easier to convert source code to tokens and then work with token stream.

On 8/29/14, 10:41 AM, Mike James wrote:
> Hi,
>
> Looking at the DMD Source Guide it says "The lexer transforms the file
> into an array of tokens."
>
> Why is this step taken instead of, say, just calling a function that
> returns the next token (or however many required for the look-ahead)?
>
> Regards,
>    -=mike=-

I believe this is just an abstract description of how it works. It actually uses something like next token with a freelist of tokens if it needs to do some lookahead.

https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L260
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L261
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L237