Hi!

I am trying to implement autocomplete in DCD using DMD as a library. After parsing & the semantic analysis of a given source file, given a location (line & column number), I want to retrieve the Scope* that the statement at the given line is part of so that I can get the symbol table.

I noticed that every Scope has a pointer to its parent Scope (enclosing), but as far as I could see there's no way to access inner scopes with the information that there is currently in the Scope struct. Would it be ok to add a list of children scopes inside the struct? So, during semantic analysis, whenever the push method of the Scope structure is called I could also push the newly created scope inside this list of children?

What I tried to do without being able to access children directly was to create a visitor which goes through the entire AST until it reaches the statement at the given location. Once I've found this statement, a solution would be to find a Dsymbol inside of it and get its _scope, but for many of these symbols (for instance, local variables of functions) this field is null. I had a look at visit(VarDeclaration) in dsymbolsem.d and the _scope would be set for the variable declaration symbol if its parent was an aggregate declaration, but why isn't it set for variable declarations local to functions?

Is there any easier way to go about this?

On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:
> Hi!
>
> I am trying to implement autocomplete in DCD using DMD as a library. After parsing & the semantic analysis of a given source file, given a location (line & column number), I want to retrieve the Scope* that the statement at the given line is part of so that I can get the symbol table.
>
> I noticed that every Scope has a pointer to its parent Scope (enclosing), but as far as I could see there's no way to access inner scopes with the information that there is currently in the Scope struct. Would it be ok to add a list of children scopes inside the struct?

No I think this is a wrong trail. You could be able to do that without modifying the compiler.

> So, during semantic analysis, whenever the push method of the Scope structure is called I could also push the newly created scope inside this list of children?
>
> What I tried to do without being able to access children directly was to create a visitor which goes through the entire AST until it reaches the statement at the given location.

Yes I think you can do that.

> Once I've found this statement, a solution would be to find a Dsymbol inside of it and get its _scope, but for many of these symbols (for instance, local variables of functions) this field is null. I had a look at visit(VarDeclaration) in dsymbolsem.d and the _scope would be set for the variable declaration symbol if its parent was an aggregate declaration, but why isn't it set for variable declarations local to functions?
>
> Is there any easier way to go about this?

For functions you can try to find the `Scope` when `ScopeStatement`s are visited because they have `.loc` (the begining) and `.endloc` information. So if the cursor loc is within the two values, the scope `sc` of the visitor (StatementSemanticVisitor.sc) gives you the available symbols at the cursor pos.

Now technically there might be a problem because I dont see how you can plug the code to make this in `StatementSemanticVisitor`. This would require a kind of signal/callback/event system. Since the `Scope` is not attached to an `AstNode` you CANT use a custom visitor.

On Tuesday, 21 April 2020 at 01:52:49 UTC, Basile B. wrote:
> On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:
>> Hi!
>>
>> I am trying to implement autocomplete in DCD using DMD as a library.
>> [...]
> [...]
> to an `AstNode` you CANT use a custom visitor.

Sorry maybe I've been too opiniated.

in DMD, because of the lazy compilation model, visitors usually don't process a whole module so:

- the scope for a statement is the scope that's passed to the `StatementSemanticVisitor` class created when `statementSemantic()` is called for this statement.
- the scope for an expression is the scope that's passed to the `ExpressionSemanticVisitor` class created when `expressionSemantic()` is called for this expression.
- the scope for a type is the scope that's passed to the `TypeSemanticVisitor` class created when `typeSemantic()` is called for this type.

From your original message I have the impression that you want to retrieve a scope by a node, and you noticed that this is not always possible. The time where a scope is always linked to a node is when its semantic is run, and it's a class member. So maybe (and only maybe) there's something to do with that fact but as said at first glance I don't see how, without a kind of callback system.

On Tuesday, 21 April 2020 at 06:58:47 UTC, Basile B. wrote:
> On Tuesday, 21 April 2020 at 01:52:49 UTC, Basile B. wrote:
>> On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:
>>> Hi!
>>>
>>> I am trying to implement autocomplete in DCD using DMD as a library.
>>> [...]
>> [...]
>> to an `AstNode` you CANT use a custom visitor.
>
> Sorry maybe I've been too opiniated.
>
> in DMD, because of the lazy compilation model, visitors usually don't process a whole module so:
>
> - the scope for a statement is the scope that's passed to the `StatementSemanticVisitor` class created when `statementSemantic()` is called for this statement.
> - the scope for an expression is the scope that's passed to the `ExpressionSemanticVisitor` class created when `expressionSemantic()` is called for this expression.
> - the scope for a type is the scope that's passed to the `TypeSemanticVisitor` class created when `typeSemantic()` is called for this type.
>
> From your original message I have the impression that you want to retrieve a scope by a node, and you noticed that this is not always possible. The time where a scope is always linked to a node is when its semantic is run, and it's a class member. So maybe (and only maybe) there's something to do with that fact but as said at first glance I don't see how, without a kind of callback system.

Maybe that adding this in the compiler globals:

  alias OnStatementSemanticStart = void function(Statement, Scope*);
  alias OnStatementSemanticDone = void function(Statement, Scope*);
  alias OnExpressionSemanticStart = void function(Expression, Scope*);
  alias OnExpressionSemanticDone = void function(Expression, Scope*);

  OnStatementSemanticStart onStatementSemanticStart;
  OnStatementSemanticDone onStatementSemanticDone;
  OnExpressionSemanticStart onExpressionSemanticStart;
  OnExpressionSemanticDone onExpressionSemanticDone;

and add the handlers in statementSemantic() and expressionSemantic()

could help to build a useful enough hash map but without significant memory impact on the compiler. Actually just the "Done" events would be usefull because of the many lowering done during semantics.

On Monday, 20 April 2020 at 16:56:51 UTC, Cristian Creteanu wrote:
> Hi!
>
> I am trying to implement autocomplete in DCD using DMD as a library. After parsing & the semantic analysis of a given source file, given a location (line & column number), I want to retrieve the Scope* that the statement at the given line is part of so that I can get the symbol table.

In addition to what Basile mentioned, you can have a look at VisualD [1], which now uses DMD for semantic analysis and intellisense.

[1] https://github.com/rainers/visuald/blob/master/vdc/semantic.d

--
/Jacob Carlborg

On Tuesday, 21 April 2020 at 07:45:23 UTC, Basile B. wrote:
>   alias OnStatementSemanticStart = void function(Statement, Scope*);
>   alias OnStatementSemanticDone = void function(Statement, Scope*);
>   alias OnExpressionSemanticStart = void function(Expression, Scope*);
>   alias OnExpressionSemanticDone = void function(Expression, Scope*);
>
>   OnStatementSemanticStart onStatementSemanticStart;
>   OnStatementSemanticDone onStatementSemanticDone;
>   OnExpressionSemanticStart onExpressionSemanticStart;
>   OnExpressionSemanticDone onExpressionSemanticDone;
>
> and add the handlers in statementSemantic() and expressionSemantic()


Thank you so much! It worked :D

On Thursday, 23 April 2020 at 09:26:22 UTC, Jacob Carlborg wrote:
> In addition to what Basile mentioned, you can have a look at VisualD [1], which now uses DMD for semantic analysis and intellisense.
>
> [1] https://github.com/rainers/visuald/blob/master/vdc/semantic.d
>
> --
> /Jacob Carlborg

It seems that VisualD uses scopes defined internally rather than the ones used by DMD

On Saturday, 2 May 2020 at 21:36:00 UTC, Cristian Creteanu wrote:
> On Tuesday, 21 April 2020 at 07:45:23 UTC, Basile B. wrote:
>>   alias OnStatementSemanticStart = void function(Statement, Scope*);
>>   alias OnStatementSemanticDone = void function(Statement, Scope*);
>>   alias OnExpressionSemanticStart = void function(Expression, Scope*);
>>   alias OnExpressionSemanticDone = void function(Expression, Scope*);
>>
>>   OnStatementSemanticStart onStatementSemanticStart;
>>   OnStatementSemanticDone onStatementSemanticDone;
>>   OnExpressionSemanticStart onExpressionSemanticStart;
>>   OnExpressionSemanticDone onExpressionSemanticDone;
>>
>> and add the handlers in statementSemantic() and expressionSemantic()
>
>
> Thank you so much! It worked :D

I've seen the PR [1], so it's ok, you manage to get something to work with this ?
If so in DMD I would prefer a `version(callback_API)` because at some point more callbacks might be required, added, but not necessarily tied to the dmd library version.

[1] https://github.com/dlang/dmd/pull/11092

On Sunday, 3 May 2020 at 03:10:34 UTC, Basile B. wrote:
> I've seen the PR [1], so it's ok, you manage to get something to work with this ?
> [1] https://github.com/dlang/dmd/pull/11092

Yes, I did manage to get results out of this approach.

> If so in DMD I would prefer a `version(callback_API)` because at some point more callbacks might be required, added, but not necessarily tied to the dmd library version.

You're right, I will change the version name in the PR.

On Saturday, 2 May 2020 at 21:40:28 UTC, Cristian Creteanu wrote:
> On Thursday, 23 April 2020 at 09:26:22 UTC, Jacob Carlborg wrote:
>> In addition to what Basile mentioned, you can have a look at VisualD [1], which now uses DMD for semantic analysis and intellisense.
>>
>> [1] https://github.com/rainers/visuald/blob/master/vdc/semantic.d
>>
>> --
>> /Jacob Carlborg
>
> It seems that VisualD uses scopes defined internally rather than the ones used by DMD

My bad, VisualD does use ScopeDsymbol instead, so I should've taken a closer look. Thanks for the suggestion, I will pay more attention to VisualD in the future.