I've created a dub package for the D version of DMD's lexer, generated automatically from the C++ source.

github: https://github.com/yebblies/ddmd
dub: http://code.dlang.org/packages/ddmd

There are a few annoying limitations, such that it uses dmd's error printing and allocation functions, and requires configuration through 'global'.

Here is an example program that uses the lexer:

======================

import std.stdio;
import std.file;

import ddmd.tokens;
import ddmd.lexer;

/////////////////////////

void main()
{
   Lexer.initLexer();

   string data = "void blah() {} // stuff";
   auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
   l.nextToken();
   do
   {
       printf("token: %s\n", l.token.toChars());
   }
   while (l.nextToken() != TOKeof);
}

======================

Prints:

token: void
token: blah
token: (
token: )
token: {
token: } 

On 5/01/2015 2:07 a.m., Daniel Murphy wrote:
> I've created a dub package for the D version of DMD's lexer, generated
> automatically from the C++ source.
>
> github: https://github.com/yebblies/ddmd
> dub: http://code.dlang.org/packages/ddmd
>
> There are a few annoying limitations, such that it uses dmd's error
> printing and allocation functions, and requires configuration through
> 'global'.
>
> Here is an example program that uses the lexer:
>
> ======================
>
> import std.stdio;
> import std.file;
>
> import ddmd.tokens;
> import ddmd.lexer;
>
> /////////////////////////
>
> void main()
> {
>     Lexer.initLexer();
>
>     string data = "void blah() {} // stuff";
>     auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
>     l.nextToken();
>     do
>     {
>         printf("token: %s\n", l.token.toChars());
>     }
>     while (l.nextToken() != TOKeof);
> }
>
> ======================
>
> Prints:
>
> token: void
> token: blah
> token: (
> token: )
> token: {
> token: }

I saw that. I'm really looking forward to getting my teeth into it and doing some good old refactoring. Although that will be a while because of the auto generated thing.

"Rikki Cattermole"  wrote in message news:m8be2m$1dlp$1@digitalmars.com... 

> I saw that. I'm really looking forward to getting my teeth into it and doing some good old refactoring. Although that will be a while because of the auto generated thing.

There's plenty of refactoring to be done on the C++ side too.

On Sunday, 4 January 2015 at 13:07:34 UTC, Daniel Murphy wrote:
> I've created a dub package for the D version of DMD's lexer, generated automatically from the C++ source.
>
> github: https://github.com/yebblies/ddmd
> dub: http://code.dlang.org/packages/ddmd
>
> There are a few annoying limitations, such that it uses dmd's error printing and allocation functions, and requires configuration through 'global'.
>
> Here is an example program that uses the lexer:
>
> ======================
>
> import std.stdio;
> import std.file;
>
> import ddmd.tokens;
> import ddmd.lexer;
>
> /////////////////////////
>
> void main()
> {
>    Lexer.initLexer();
>
>    string data = "void blah() {} // stuff";
>    auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
>    l.nextToken();
>    do
>    {
>        printf("token: %s\n", l.token.toChars());
>    }
>    while (l.nextToken() != TOKeof);
> }
>
> ======================
>
> Prints:
>
> token: void
> token: blah
> token: (
> token: )
> token: {
> token: }

(sorry if you get this question too often)

How is DDMD as a whole going? Is it getting closer or are ongoing DMD changes slowing it down too much?

Anyway, great to have the lexer on DUB, not going to use if for now because I expect there to be changes, but I guess it may eventually be useful for writing tools.

"Kiith-Sa"  wrote in message news:nffxogzwpmayydyomaju@forum.dlang.org...

> (sorry if you get this question too often)
>
> How is DDMD as a whole going? Is it getting closer or are ongoing DMD changes slowing it down too much?

It's been sitting still for 8 nearly months because of https://github.com/braddr/d-tester/issues/39 and https://github.com/braddr/d-tester/issues/40

I don't mind getting asked, I just wish the answer would change.

> Anyway, great to have the lexer on DUB, not going to use if for now because I expect there to be changes, but I guess it may eventually be useful for writing tools.

Hopefully lots of changes. 

On 2015-01-04 14:07, Daniel Murphy wrote:
> I've created a dub package for the D version of DMD's lexer, generated
> automatically from the C++ source.

This is very cool :), keep up to good work.

-- 
/Jacob Carlborg

> I've created a dub package for the D version of DMD's lexer, generated automatically from the C++ source.

very cool.

on a related note, have you considered sharing your translation tool (c++ -> D)?  I completely understand if you would rather not of course.



Laeeth.

On Sunday, 4 January 2015 at 17:27:57 UTC, Daniel Murphy wrote:
> "Kiith-Sa"  wrote in message news:nffxogzwpmayydyomaju@forum.dlang.org...
>
>> (sorry if you get this question too often)
>>
>> How is DDMD as a whole going? Is it getting closer or are ongoing DMD changes slowing it down too much?
>
> It's been sitting still for 8 nearly months because of https://github.com/braddr/d-tester/issues/39 and https://github.com/braddr/d-tester/issues/40
>
> I don't mind getting asked, I just wish the answer would change.
>
>> Anyway, great to have the lexer on DUB, not going to use if for now because I expect there to be changes, but I guess it may eventually be useful for writing tools.
>
> Hopefully lots of changes.

I can't really help with those, but FWIW I bumped the issues.

On 1/4/2015 5:07 AM, Daniel Murphy wrote:
> I've created a dub package for the D version of DMD's lexer, generated
> automatically from the C++ source.
>
> github: https://github.com/yebblies/ddmd
> dub: http://code.dlang.org/packages/ddmd

Great! Thank you!

On 5/01/2015 2:39 a.m., Daniel Murphy wrote:
> "Rikki Cattermole"  wrote in message news:m8be2m$1dlp$1@digitalmars.com...
>> I saw that. I'm really looking forward to getting my teeth into it and
>> doing some good old refactoring. Although that will be a while because
>> of the auto generated thing.
>
> There's plenty of refactoring to be done on the C++ side too.

What I want to do is D side unfortunately. I just hate to see large lists in e.g. globals with constructors. When they can be dealt with a simple array and a bit of CTFE magic.