Thread overview | |||||
---|---|---|---|---|---|
|
August 29, 2014 DMD Compiler - lexer | ||||
---|---|---|---|---|
| ||||
Hi, Looking at the DMD Source Guide it says "The lexer transforms the file into an array of tokens." Why is this step taken instead of, say, just calling a function that returns the next token (or however many required for the look-ahead)? Regards, -=mike=- |
August 29, 2014 Re: DMD Compiler - lexer | ||||
---|---|---|---|---|
| ||||
Posted in reply to Mike James Attachments: | On Fri, 29 Aug 2014 13:41:20 +0000 Mike James via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com> wrote: > Looking at the DMD Source Guide it says "The lexer transforms the file into an array of tokens." > > Why is this step taken instead of, say, just calling a function that returns the next token (or however many required for the look-ahead)? D parser is somewhat complicated and it can peek alot of tokens from input stream without consuming 'em. so it's just easier to convert source code to tokens and then work with token stream. |
August 29, 2014 Re: DMD Compiler - lexer | ||||
---|---|---|---|---|
| ||||
Posted in reply to Mike James | On 8/29/14, 10:41 AM, Mike James wrote: > Hi, > > Looking at the DMD Source Guide it says "The lexer transforms the file > into an array of tokens." > > Why is this step taken instead of, say, just calling a function that > returns the next token (or however many required for the look-ahead)? > > Regards, > -=mike=- I believe this is just an abstract description of how it works. It actually uses something like next token with a freelist of tokens if it needs to do some lookahead. https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L260 https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L261 https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L237 |
Copyright © 1999-2021 by the D Language Foundation