View mode: basic / threaded / horizontal-split · Log in · Help
November 25, 2010
custom AST library
Hi,

I'm currently making some custom AST parser for D :

http://pastebin.com/1hmnTUbY -- AST Parser

http://pastebin.com/ZRc5be1p -- Sum example

Right now both lexer and syntax parser are just awfull and they don't work 
in CTFE because of class usage. But example looks rather interesting, with 
such approach (if it works) one can create full list comprehension like 
python's one or MS LINQ for C# system as user library.

I tried to change class to struct, but failed :-( Any ideas how to make it 
work in CTFE ?
November 25, 2010
Re: custom AST library
Hm... there is some limitations in example code, concerning local variables 
in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
December 21, 2010
Re: custom AST library
I'm just having a look at this now, looks rather interesting! I'd love
to be able to use some Python-like list comprehension tricks in my D
code.

P.S. not sure if it's your fault, but you're code has spaces between
"q" and "{", which gives a compiler error. q{} has to start without
spaces between q and {.   ;)

On 11/25/10, Alex_Dovhal <alex_dovhal@yahoo.com> wrote:
> Hm... there is some limitations in example code, concerning local variables
> in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
>
>
>
December 21, 2010
Re: custom AST library
SORRY.

Disregard that, it was my fault. My reindenter f*ked it up. You're
code is fine. :)

On 12/21/10, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:
> I'm just having a look at this now, looks rather interesting! I'd love
> to be able to use some Python-like list comprehension tricks in my D
> code.
>
> P.S. not sure if it's your fault, but you're code has spaces between
> "q" and "{", which gives a compiler error. q{} has to start without
> spaces between q and {.   ;)
>
> On 11/25/10, Alex_Dovhal <alex_dovhal@yahoo.com> wrote:
>> Hm... there is some limitations in example code, concerning local
>> variables
>> in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
>>
>>
>>
>
December 21, 2010
Re: custom AST library
"Andrej Mitrovic" wrote:
> I'm just having a look at this now, looks rather interesting! I'd love
> to be able to use some Python-like list comprehension tricks in my D
> code.

Yes list comprehension is also possible this way. But it is not very eye 
handy right now even if fully implemented, because one can't use variables 
external to the scope of sum or listcomp... templates, also those 
parenthesis and braces..., so must write:

float a = 1 ;
int n = 10 ;
float x = mixin(sumImpl(q{i=0:10; a*i})) ;
float y = mixin(sumImpl(q{i=1:n; i*mixin(prodImpl(q{j=0:i ; i+j}))}));

instead of:

float a = 1 ;
int n = 10 ;
float x = sum!(q{i=0:10; a*i}) ;
float y = sum!(q{i=1:n; i*prod!(q{j=0:i ; i+j})});

I made it as my first attempt to use D, that's why maybe it's not solid or 
well designed, so any ideas are welcomed.
December 21, 2010
Re: custom AST library
That's why I think it might be a better idea to simply use "DSL
blocks" throughout your code. Instead of having to mixin each
template, you would only have one template where you have all the
rules for your DSL:

float a = 1 ;
int n = 10 ;
// more D code..

mixin(MyDSL!(q{
   // Here you use the syntax defined by the "MyDSL" language
   float x = sum!(i=0:10; a*i) ;
   float y = sum!(i=1:n; i*prod!(j=0:i ; i+j);
   // More MyDSL code..
}));

// Here you use D syntax again, and D will know about float x and float y.

Not to mention that you could potentially have some control over
syntax highlighting (if you have a capable editor :p). For example you
could write a little script in VIM which detects the use of "MyDSL!()"
template, and highlights the code inside the block differently than
the one outside the block.



On 12/21/10, Alex_Dovhal <alex_dovhal@yahoo.com> wrote:
> "Andrej Mitrovic" wrote:
>> I'm just having a look at this now, looks rather interesting! I'd love
>> to be able to use some Python-like list comprehension tricks in my D
>> code.
>
> Yes list comprehension is also possible this way. But it is not very eye
> handy right now even if fully implemented, because one can't use variables
> external to the scope of sum or listcomp... templates, also those
> parenthesis and braces..., so must write:
>
> float a = 1 ;
> int n = 10 ;
> float x = mixin(sumImpl(q{i=0:10; a*i})) ;
> float y = mixin(sumImpl(q{i=1:n; i*mixin(prodImpl(q{j=0:i ; i+j}))}));
>
> instead of:
>
> float a = 1 ;
> int n = 10 ;
> float x = sum!(q{i=0:10; a*i}) ;
> float y = sum!(q{i=1:n; i*prod!(q{j=0:i ; i+j})});
>
> I made it as my first attempt to use D, that's why maybe it's not solid or
> well designed, so any ideas are welcomed.
>
>
>
December 21, 2010
Re: custom AST library
"Andrej Mitrovic" <andrej.mitrovich@gmail.com> wrote:
> That's why I think it might be a better idea to simply use "DSL
> blocks" throughout your code. Instead of having to mixin each
> template, you would only have one template where you have all the
> rules for your DSL:
>
> float a = 1 ;
> int n = 10 ;
> // more D code..
>
> mixin(MyDSL!(q{
>    // Here you use the syntax defined by the "MyDSL" language
>    float x = sum!(i=0:10; a*i) ;
>    float y = sum!(i=1:n; i*prod!(j=0:i ; i+j);
>    // More MyDSL code..
> }));
>
> // Here you use D syntax again, and D will know about float x and float y.
>
> Not to mention that you could potentially have some control over
> syntax highlighting (if you have a capable editor :p). For example you
> could write a little script in VIM which detects the use of "MyDSL!()"
> template, and highlights the code inside the block differently than
> the one outside the block.

Very interesting idea :), one note - sum!(...) in DSL is not template any 
more, so one can change a design as he wants, say:

mixin(MyDSL_Impl(q{
   float x = sum {i=0:10 ; a * i};
   float y = sum {i=1:n; i*prod{j=0:i; i+j}} ;
}));

this MyDSL_Impl only scans for "name" "{" text "}" and replaces it with 
mixin(nameImpl(q{text})) ;

User can put entire function body or even almost full module (if don't care 
about compile time) into that DSL in order not to change from DSL to normal 
code and back. One more thing to think about - how to show correct error 
line number in there?
December 21, 2010
Re: custom AST library
On 12/21/10, Alex_Dovhal <alex_dovhal@yahoo.com> wrote:
> User can put entire function body or even almost full module (if don't care
> about compile time) into that DSL in order not to change from DSL to normal
> code and back.

Yep. You could even take it a step further and use some form of escape
mechanism, which would leave D code alone but parse the rest of the
string as a DSL. E.g.:

mixin(MyDSL!(q{

   $(D)
   float a = 1 ;    // will be mixed in as is
   int n = 10 ;
   void test(int t)
   {
       writeln(t);
   }
   $(/D)  // end of D code

  x = sum(i=0:10; a*i) ;  // will be parsed and reconstructed as D code
  y = sum(i=1:n; i*prod!(j=0:i ; i+j);
}));

So now you wouldn't have to write a lot of MyDSL blocks in-between D code.

> One more thing to think about - how to show correct error
> line number in there?

See https://docs.google.com/leaf?id=0B-f1J0HiG2ovNmQzNzM1ODYtNDM3ZC00N2I4LWI2YmUtYTIyM2Y2MzcyZWIw&sort=name&layout=list&num=50&pli=1
, there's a unittests.d module in /src in the archive. It uses a
parameter that is default initialized to the line (size_t line =
__LINE__))

I guess from there you could chomp up your string into an array for
each new line, and if your DSL parser finds a syntax error it would
report the line number as e.g. currentLineNumber + line, where
currentLineNumber is the line number you're currently parsing, e.g.
DSLStringArray[currentLineNumber] << this line has the error.

I think that's about it (but I haven't had my nighttime coffee yet so
don't trust me fully on this one :p).
Top | Discussion index | About this forum | D home