http://aljudy.org/dstuff/simple.d.lexer.zip

This is something I'm working on -lazily-, as a part of a bigger project.
It's not heavliy tested or anything .. read the doc.txt that comes with it ..
The code uses several Objects, but it's not entirely OO (unless you think of modules as singleton classes!)

I tried to make the code modular and cohesive. Hopefully, most bugs would be easily fixable.

I'll quote myself here (from doc.txt) to give an example of what this lexer can do:
The following code segment
<code>	
void main()
{
   //comment
   /+
      /* */
    /+
      +/
   +/
   printf("hi");
}
</code>
produces the folloing token list:
type                             text
----------------------------------------
TOK.TOKvoid                     void
TOK.TOKidentifier               main
TOK.TOKopenparn                 (
TOK.TOKcloseparen               )
TOK.TOKopencurly                {
TOK.TOKidentifier               printf
TOK.TOKopenparen                (
TOK.TOKstring                   "hi"
TOK.TOKcloseparen               )
TOK.TOKsemicolon                ;
TOK.TOKclosecurly               }


The code ships with this example, to compile it, cd to the directory where you unzipped it, and invoke build against it:
>build main.d -clean

and run it
>main


For details, see doc.txt in the zip file. As you can see, the lexer handles nesting comments pretty well.

I am also working on a parser, but do not expect anything.
I'm a kind of guy who works on alot of projects but doesn't finish most of them, which is why I'm releasing this lexer now. Basically I don't know whether or not I will continue working on this baby, and I don't want it to be wasted for nothing.

I hope people will find it useful.
Enjoy.

On Sun, 04 Sep 2005 00:36:52 -0600, Hasan Aljudy wrote:

> http://aljudy.org/dstuff/simple.d.lexer.zip
> 
[snip]
> Basically I don't know whether or not I will continue working on this baby, and I don't want it to be wasted for nothing.

It compiled and worked well.

> I hope people will find it useful.

I have a few uses for such a beastie. Thanks.

-- 
Derek Parnell
Melbourne, Australia
4/09/2005 6:43:44 PM

I just noticed a slight problem in lex/string_scanner.d, so I fixed it and uploaded a new `version`.
also, the
"string"\escape"string2"
issue is fixed too.

Hasan Aljudy wrote:
> http://aljudy.org/dstuff/simple.d.lexer.zip
> 
> This is something I'm working on -lazily-, as a part of a bigger project.
> It's not heavliy tested or anything .. read the doc.txt that comes with it ..
> The code uses several Objects, but it's not entirely OO (unless you think of modules as singleton classes!)
> 
> I tried to make the code modular and cohesive. Hopefully, most bugs would be easily fixable.
> 
> I'll quote myself here (from doc.txt) to give an example of what this lexer can do:
> The following code segment
> <code>   void main()
> {
>    //comment
>    /+
>       /* */
>     /+
>       +/
>    +/
>    printf("hi");
> }
> </code>
> produces the folloing token list:
> type                             text
> ----------------------------------------
> TOK.TOKvoid                     void
> TOK.TOKidentifier               main
> TOK.TOKopenparn                 (
> TOK.TOKcloseparen               )
> TOK.TOKopencurly                {
> TOK.TOKidentifier               printf
> TOK.TOKopenparen                (
> TOK.TOKstring                   "hi"
> TOK.TOKcloseparen               )
> TOK.TOKsemicolon                ;
> TOK.TOKclosecurly               }
> 
> 
> The code ships with this example, to compile it, cd to the directory where you unzipped it, and invoke build against it:
>  >build main.d -clean
> 
> and run it
>  >main
> 
> 
> For details, see doc.txt in the zip file. As you can see, the lexer handles nesting comments pretty well.
> 
> I am also working on a parser, but do not expect anything.
> I'm a kind of guy who works on alot of projects but doesn't finish most of them, which is why I'm releasing this lexer now. Basically I don't know whether or not I will continue working on this baby, and I don't want it to be wasted for nothing.
> 
> I hope people will find it useful.
> Enjoy.

Derek Parnell wrote:
> On Sun, 04 Sep 2005 00:36:52 -0600, Hasan Aljudy wrote:
> 
> 
>>http://aljudy.org/dstuff/simple.d.lexer.zip
>>
> 
> [snip]
> 
>>Basically I don't know whether or not I will continue working on this baby, and I don't want it to be wasted for nothing.
> 
> 
> It compiled and worked well. 
> 
> 
>>I hope people will find it useful.
> 
> 
> I have a few uses for such a beastie. Thanks.
> 
Thank you. It's good to hear that.

Hasan Aljudy schrieb:
> http://aljudy.org/dstuff/simple.d.lexer.zip

Link doesn't work.
(
Not Found
The requested URL http:// was not found on this server.
)