Hello everybody!

Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome.

So I built the library and everything went smooth. Thanks for the effort of all the involved people who made that possible!

I've had a look at the 2 examples, too, the avg. function lines ( https://github.com/dlang/dmd/blob/master/src/examples/avg.d ) and the import ( https://github.com/dlang/dmd/blob/master/src/examples/impvisitor.d ) ones and for a start I decided to make a program that prints the outline of a module.

Turns out I don't really understand how to access the data in the AST.
For everything there's a visitor method and overriding a few of them to print return statements and some such works as advertised.

However, I have no idea where I am in the tree when any of those methods are called.
Like for example in FunctionLengthVisitor(AST).visitFuncBody(AST.FuncDeclaration fd).
I have a function declaration object which tells me everything about what's inside the function, but how do I know what or where this function belongs to, where can I get that information ? I don't see anything about UDAs either, nor the doc comment.

I understand when visitor.getAvgLen is called with the parsed module, the accept function calls a visitor overload for each member.
But this sounds to me like I'd have to do a lot of book keeping in my visitor to keep track of things which are already present in the AST.

Any insight to this would be much appreciated :)

Hi Joe,

I suggest you watch this video which explains how the parse time visitors work: https://www.youtube.com/watch?v=tK072jcoWv4 .

On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
> Hello everybody!
>
> Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome.
>
> So I built the library and everything went smooth. Thanks for the effort of all the involved people who made that possible!
>
> I've had a look at the 2 examples, too, the avg. function lines ( https://github.com/dlang/dmd/blob/master/src/examples/avg.d ) and the import ( https://github.com/dlang/dmd/blob/master/src/examples/impvisitor.d ) ones and for a start I decided to make a program that prints the outline of a module.
>
> Turns out I don't really understand how to access the data in the AST.
> For everything there's a visitor method and overriding a few of them to print return statements and some such works as advertised.
>
> However, I have no idea where I am in the tree when any of those methods are called.
> Like for example in FunctionLengthVisitor(AST).visitFuncBody(AST.FuncDeclaration fd).
> I have a function declaration object which tells me everything about what's inside the function, but how do I know what or where this function belongs to, where can I get that information ? I don't see anything about UDAs either, nor the doc comment.
>

The FuncDeclaration node contains all the information for that.
For example, you can access fd.parent to see if the function is
declared at top-level (in which case, the parent is going to be a module
declaration ) or if it is a nested function (in a class, in a struct, in a function).
Every AST node contains information about the position in the AST, all you
have to do is find how to get that information: which field to access or which
member function to call.

> I understand when visitor.getAvgLen is called with the parsed module, the accept function calls a visitor overload for each member.
> But this sounds to me like I'd have to do a lot of book keeping in my visitor to keep track of things which are already present in the AST.

The function average length visitor inherits a transitive visitor
which means that the AST traversal logic is already implemented for you.
All you have to do is override the visiting methods of interest and do
whatever suits you : print stuff, alter the ast, stop the visitation or
continue the visitation (by calling super.visit(ASTnode)).

>
> Any insight to this would be much appreciated :)

I know that my explanations might not be very explicit, but if you have an example please post it and we can work on it.

Cheers,
RazvanN

On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote:
> Hi Joe,
>
> I suggest you watch this video which explains how the parse time visitors work: https://www.youtube.com/watch?v=tK072jcoWv4 .
>
> On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
>>[...]
>
> The FuncDeclaration node contains all the information for that.
> For example, you can access fd.parent to see if the function is
> declared at top-level (in which case, the parent is going to be a module
> declaration ) or if it is a nested function (in a class, in a struct, in a function).
> Every AST node contains information about the position in the AST, all you
> have to do is find how to get that information: which field to access or which
> member function to call.
>
>> [...]
>
> The function average length visitor inherits a transitive visitor
> which means that the AST traversal logic is already implemented for you.
> All you have to do is override the visiting methods of interest and do
> whatever suits you : print stuff, alter the ast, stop the visitation or
> continue the visitation (by calling super.visit(ASTnode)).
>
>> [...]
>
> I know that my explanations might not be very explicit, but if you have an example please post it and we can work on it.
>
> Cheers,
> RazvanN

Hello RazvanN,

thank you very much for taking the time to reply and also your effort in making this happen.

I watched the video you linked and read your reply over and over, yet I still have a hard time to wrap my head around this idea.

Like for example DHTML DOM is very easy for me to grasp. It's like riding the car down the country road and I know where I am and which town I'm going to be next, etc.

This AST thing is more like a teleporter room on the Enterprise. Scotty activates the teleporter and a canister appears an a spot labeled imports. He repeats and a canister appears on a spot labeled functions, etc.

I will try again...

On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote:
> Hi Joe,
>
> /SNIP
>
> On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
>>[...]
>
> The FuncDeclaration node contains all the information for that.
> For example, you can access fd.parent to see if the function is
> declared at top-level (in which case, the parent is going to be a module
> declaration ) or if it is a nested function (in a class, in a struct, in a function).
> Every AST node contains information about the position in the AST, all you
> have to do is find how to get that information: which field to access or which
> member function to call.
>
> /SNIP
>
> Cheers,
> RazvanN

Follow up question...

Why is *.parent always null?
e.g.:

extern(C++) class MyVisitor(AST): ParseTimeTransitiveVisitor!AST
{
  override void visit(AST.Import i)
  {
    assert(i.parent is null); // always true
  }

  override void visitFuncBody(AST.FuncDeclaration f)
  {
    assert(f.parent is null); // always true
  }
}

On Thursday, 22 February 2018 at 13:21:04 UTC, joe wrote:
> On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote:
>> [...]
>
> Follow up question...
>
> Why is *.parent always null?
> e.g.:
>
> extern(C++) class MyVisitor(AST): ParseTimeTransitiveVisitor!AST
> {
>   override void visit(AST.Import i)
>   {
>     assert(i.parent is null); // always true
>   }
>
>   override void visitFuncBody(AST.FuncDeclaration f)
>   {
>     assert(f.parent is null); // always true
>   }
> }

I think parent is only set after sema.
and you are overriding the parsetime visitor.

On Thursday, 22 February 2018 at 13:21:04 UTC, joe wrote:
> On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote:
>> Hi Joe,
>>
>> /SNIP
>>
>> On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
>>>[...]
>>
>> The FuncDeclaration node contains all the information for that.
>> For example, you can access fd.parent to see if the function is
>> declared at top-level (in which case, the parent is going to be a module
>> declaration ) or if it is a nested function (in a class, in a struct, in a function).
>> Every AST node contains information about the position in the AST, all you
>> have to do is find how to get that information: which field to access or which
>> member function to call.
>>
>> /SNIP
>>
>> Cheers,
>> RazvanN
>
> Follow up question...
>
> Why is *.parent always null?
> e.g.:
>
> extern(C++) class MyVisitor(AST): ParseTimeTransitiveVisitor!AST
> {
>   override void visit(AST.Import i)
>   {
>     assert(i.parent is null); // always true
>   }
>
>   override void visitFuncBody(AST.FuncDeclaration f)
>   {
>     assert(f.parent is null); // always true
>   }
> }

Indeed, @Stefan is right. The ParseTimeVisitor only contains information available at parse time. If you are interested in the parent you have 2 options: either (1) use the ParseTimeVisitor and implement the AST traversal logic yourself or (2) you can use the SemanticTimeTransitiveVisitor in which case the parent is not going to be null. In the case of (2) you need to also do some semantic analysis (so you need the whole dmd library, not just the parsing one). Here's an example on using the dmd library (including semantic) [1]. You can copy paste that example and add a few lines of code where you instantiate your visitor (which will inherit SemanticTimeTransitiveVisitor).

[1] https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d

RazvanN

On Thursday, 22 February 2018 at 13:44:51 UTC, RazvanN wrote:
> On Thursday, 22 February 2018 at 13:21:04 UTC, joe wrote:
>> [...]
>
> Indeed, @Stefan is right. The ParseTimeVisitor only contains information available at parse time. If you are interested in the parent you have 2 options: either (1) use the ParseTimeVisitor and implement the AST traversal logic yourself or (2) you can use the SemanticTimeTransitiveVisitor in which case the parent is not going to be null. In the case of (2) you need to also do some semantic analysis (so you need the whole dmd library, not just the parsing one). Here's an example on using the dmd library (including semantic) [1]. You can copy paste that example and add a few lines of code where you instantiate your visitor (which will inherit SemanticTimeTransitiveVisitor).
>
> [...]

awesome, that helps a lot!

Thanks both of you :)

On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
> Hello everybody!
>
> Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome.
>
> [...]


BTW I know it's not as powerful as DMD (and not the real thing), but often the AST XML dump from DScanner helps to deepen the understanding:

https://github.com/dlang-community/D-Scanner#ast-dump

You can even play with libdparse on the web:

https://run.dlang.io/is/qZsGDD

On Thursday, 22 February 2018 at 14:53:11 UTC, Seb wrote:
> On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
>> Hello everybody!
>>
>> Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome.
>>
>> [...]
>
>
> BTW I know it's not as powerful as DMD (and not the real thing), but often the AST XML dump from DScanner helps to deepen the understanding:
>
> https://github.com/dlang-community/D-Scanner#ast-dump
>
> You can even play with libdparse on the web:
>
> https://run.dlang.io/is/qZsGDD

Hello Seb,

I had a look at the resources you provided and they are quite useful. Thank you.

However, while technically a lexer would be enough to solve the problem at hand, I think I'm going to want that full front-end a bit later.

The more information, the better. I rather have the option to ignore something I don't need than to need something I don't have :)