On Friday, 19 December 2014 at 02:53:02 UTC, Rikki Cattermole wrote:
> On 19/12/2014 10:19 a.m., Kingsley wrote:
>> On Wednesday, 17 December 2014 at 21:05:05 UTC, Kingsley wrote:
>>>
>>>> Hi Bruno,
>>>>
>>>> Thanks very much. I do have a couple of questions about DDT in
>>>> relation to my plugin.
>>>>
>>>> Firstly - I'm not too familiar with parsing/lexing but at the moment
>>>> the Psi Structure I have implemented that comes from the DDT
>>>> parser/lexer is not in any kind of hierarchy. All the PsiElements are
>>>> available but all at the same level. Is this how the DDT parser
>>>> works? Or is it down to my implementation of the Parser/Lexer that
>>>> wraps it to create some hierarchy.
>>>>
>>>> For intellij it's going to be vastly easier to have a hierarchy with
>>>> nested elements in order to get hold of a structure representing a
>>>> class or a function for example - in order to do things like get the
>>>> start and end lines of a class definition in order to apply code
>>>> folding and to use for searching for classes and stuff.
>>>>
>>>> Secondly - how active it the development of DDT - does it keep up
>>>> with the D2 releases.
>>>>
>>>> --Kingsley
>>>
>>> After doing a bit more research it looks like I have to create the psi
>>> hierarchy myself - my current psi structure is flat because I'm just
>>> converting the DeeTokens into PsiElements directly. I've still got
>>> some experimentation to do. On the plus side I implemented commenting,
>>> code folding but everything else needs a psi hierarchy
>>
>> I've done some more investigation and I do need to build the parser
>> myself in order to create the various constructs. I've made a start but
>> I haven't gotten very far yet because I don't fully understand the
>> correct way to proceed.
>>
>> I also had a look at using the DeeParser - because it already does most
>> of what I want. However the intellij plugin wants a PsiParser which
>> returns an intellij ASTNode in the primary parse method. I can't see an
>> easy way to hook this up with DeeParser because the ParsedResult
>> although had a node method on it - gives back the wrong type of ASTNode.
>>
>> Any pointers on how I might get the DeeParser to interface to an
>> intellij ASTNode would be appreciated.
>
> Read my codebase again, it'll answer a lot of questions. Your parser is different, but what it produces shouldn't be. and yes it supports hierarchies.

Hi

So finally after a lot of wrestling with the internals of intellij I finally managed to get a working parser implementation that produces a psi hierarchy based on the DeeParser from the ddt code.

The main issue was that Intellij only wants you to create a parser using their toolset - which is either with a BNF grammar that you can then generate the parser - or with a hand written parser. Since I'm already using the DDT lexer and there is a perfectly good DDT parser as well - I just wanted to re-use the DDT parser.

However Intellij does not provide any way to create a custom AST/PSI structure or use an external parser. So I basically had to wrap the DeeParse inside the Intellij parser and sync them up programmatically. It's not the most efficient way in the world but it at least works.

In the long term I will write a BNF grammar for Intellij (using their toolkit) but I can see that will take me several months so this is a quick way to get the plugin up and running with all the power of intellij extras without spending several months stuck learning all about the complexities of grammar parsing and lexing.

Thanks very much for you help. Once I get a bit more of the cool stuff done I will release the plugin.

On Sunday, 21 December 2014 at 00:34:06 UTC, Kingsley wrote:
> On Friday, 19 December 2014 at 02:53:02 UTC, Rikki Cattermole wrote:
>> On 19/12/2014 10:19 a.m., Kingsley wrote:
>>> On Wednesday, 17 December 2014 at 21:05:05 UTC, Kingsley wrote:
>>>>
>>>>> Hi Bruno,
>>>>>
>>>>> Thanks very much. I do have a couple of questions about DDT in
>>>>> relation to my plugin.
>>>>>
>>>>> Firstly - I'm not too familiar with parsing/lexing but at the moment
>>>>> the Psi Structure I have implemented that comes from the DDT
>>>>> parser/lexer is not in any kind of hierarchy. All the PsiElements are
>>>>> available but all at the same level. Is this how the DDT parser
>>>>> works? Or is it down to my implementation of the Parser/Lexer that
>>>>> wraps it to create some hierarchy.
>>>>>
>>>>> For intellij it's going to be vastly easier to have a hierarchy with
>>>>> nested elements in order to get hold of a structure representing a
>>>>> class or a function for example - in order to do things like get the
>>>>> start and end lines of a class definition in order to apply code
>>>>> folding and to use for searching for classes and stuff.
>>>>>
>>>>> Secondly - how active it the development of DDT - does it keep up
>>>>> with the D2 releases.
>>>>>
>>>>> --Kingsley
>>>>
>>>> After doing a bit more research it looks like I have to create the psi
>>>> hierarchy myself - my current psi structure is flat because I'm just
>>>> converting the DeeTokens into PsiElements directly. I've still got
>>>> some experimentation to do. On the plus side I implemented commenting,
>>>> code folding but everything else needs a psi hierarchy
>>>
>>> I've done some more investigation and I do need to build the parser
>>> myself in order to create the various constructs. I've made a start but
>>> I haven't gotten very far yet because I don't fully understand the
>>> correct way to proceed.
>>>
>>> I also had a look at using the DeeParser - because it already does most
>>> of what I want. However the intellij plugin wants a PsiParser which
>>> returns an intellij ASTNode in the primary parse method. I can't see an
>>> easy way to hook this up with DeeParser because the ParsedResult
>>> although had a node method on it - gives back the wrong type of ASTNode.
>>>
>>> Any pointers on how I might get the DeeParser to interface to an
>>> intellij ASTNode would be appreciated.
>>
>> Read my codebase again, it'll answer a lot of questions. Your parser is different, but what it produces shouldn't be. and yes it supports hierarchies.
>
> Hi
>
> So finally after a lot of wrestling with the internals of intellij I finally managed to get a working parser implementation that produces a psi hierarchy based on the DeeParser from the ddt code.
>
> The main issue was that Intellij only wants you to create a parser using their toolset - which is either with a BNF grammar that you can then generate the parser - or with a hand written parser. Since I'm already using the DDT lexer and there is a perfectly good DDT parser as well - I just wanted to re-use the DDT parser.

Hi Bruno - would be easy to return the list of tokens included for each node in the DeeParser?
>
> However Intellij does not provide any way to create a custom AST/PSI structure or use an external parser. So I basically had to wrap the DeeParse inside the Intellij parser and sync them up programmatically. It's not the most efficient way in the world but it at least works.
>
> In the long term I will write a BNF grammar for Intellij (using their toolkit) but I can see that will take me several months so this is a quick way to get the plugin up and running with all the power of intellij extras without spending several months stuck learning all about the complexities of grammar parsing and lexing.
>
> Thanks very much for you help. Once I get a bit more of the cool stuff done I will release the plugin.

On 17/12/2014 17:19, Kingsley wrote:
> Secondly - how active it the development of DDT - does it keep up with
> the D2 releases.

Yes, it keeps up, and I plan to keep that up for the foreseeable future. (Since language grammar changes are fairly uncommon, and easy to implement when they happen).

-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros

On 21/12/2014 00:34, Kingsley wrote:
> On Friday, 19 December 2014 at 02:53:02 UTC, Rikki Cattermole wrote:
>> On 19/12/2014 10:19 a.m., Kingsley wrote:
>>> On Wednesday, 17 December 2014 at 21:05:05 UTC, Kingsley wrote:
>>>>
>>>>> Hi Bruno,
>>>>>
>>>>> Thanks very much. I do have a couple of questions about DDT in
>>>>> relation to my plugin.
>>>>>
>>>>> Firstly - I'm not too familiar with parsing/lexing but at the moment
>>>>> the Psi Structure I have implemented that comes from the DDT
>>>>> parser/lexer is not in any kind of hierarchy. All the PsiElements are
>>>>> available but all at the same level. Is this how the DDT parser
>>>>> works? Or is it down to my implementation of the Parser/Lexer that
>>>>> wraps it to create some hierarchy.
>>>>>
>>>>> For intellij it's going to be vastly easier to have a hierarchy with
>>>>> nested elements in order to get hold of a structure representing a
>>>>> class or a function for example - in order to do things like get the
>>>>> start and end lines of a class definition in order to apply code
>>>>> folding and to use for searching for classes and stuff.
>>>>>
>>>>> Secondly - how active it the development of DDT - does it keep up
>>>>> with the D2 releases.
>>>>>
>>>>> --Kingsley
>>>>
>>>> After doing a bit more research it looks like I have to create the psi
>>>> hierarchy myself - my current psi structure is flat because I'm just
>>>> converting the DeeTokens into PsiElements directly. I've still got
>>>> some experimentation to do. On the plus side I implemented commenting,
>>>> code folding but everything else needs a psi hierarchy
>>>
>>> I've done some more investigation and I do need to build the parser
>>> myself in order to create the various constructs. I've made a start but
>>> I haven't gotten very far yet because I don't fully understand the
>>> correct way to proceed.
>>>
>>> I also had a look at using the DeeParser - because it already does most
>>> of what I want. However the intellij plugin wants a PsiParser which
>>> returns an intellij ASTNode in the primary parse method. I can't see an
>>> easy way to hook this up with DeeParser because the ParsedResult
>>> although had a node method on it - gives back the wrong type of ASTNode.
>>>
>>> Any pointers on how I might get the DeeParser to interface to an
>>> intellij ASTNode would be appreciated.
>>
>> Read my codebase again, it'll answer a lot of questions. Your parser
>> is different, but what it produces shouldn't be. and yes it supports
>> hierarchies.
>
> Hi
>
> So finally after a lot of wrestling with the internals of intellij I
> finally managed to get a working parser implementation that produces a
> psi hierarchy based on the DeeParser from the ddt code.
>
> The main issue was that Intellij only wants you to create a parser using
> their toolset - which is either with a BNF grammar that you can then
> generate the parser - or with a hand written parser. Since I'm already
> using the DDT lexer and there is a perfectly good DDT parser as well - I
> just wanted to re-use the DDT parser.
>
> However Intellij does not provide any way to create a custom AST/PSI
> structure or use an external parser. So I basically had to wrap the
> DeeParse inside the Intellij parser and sync them up programmatically.
> It's not the most efficient way in the world but it at least works.
>
> In the long term I will write a BNF grammar for Intellij (using their
> toolkit) but I can see that will take me several months so this is a
> quick way to get the plugin up and running with all the power of
> intellij extras without spending several months stuck learning all about
> the complexities of grammar parsing and lexing.
>
> Thanks very much for you help. Once I get a bit more of the cool stuff
> done I will release the plugin.

Again, I'm not familiar with Intellij internals or the PSI structure, so I don't know how the DDT parser can be adapted to that.

However, I do suspect there should be a way to create a PSI structure from the DDT ASTNode structure.

PS: Sorry for the long delay in replying, I don't often check the digitalmars.D newsgroup/forum, and can sometimes be behind on the posts there. If you want to grab my attention, better to post on digitalmars.D.ide as I keep a closer eye on that newsgroup.

-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros

On 22/12/2014 11:44, Kingsley wrote:
> Hi Bruno - would be easy to return the list of tokens included for each
> node in the DeeParser?

You can create an utility method that does that, if you have a DeeParseResult.

The DeeParseResult has a 'tokenList' member, ordered by the source range. With a node's source range, you can do a binary search in that list to find the corresponding tokens.

For more convenience, I guess DeeParser could be modified so that this information - in the form of a sublist of 'tokenList' - would be stored directly in each ASTNode, in the 'data' field. This way one would not need to provide the DeeParseResult directly.


-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros