https://tree-sitter.github.io/tree-sitter/
https://github.com/tree-sitter/tree-sitter
https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)

Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser. Haven't looked in depth at the grammar yet -- appears to be context free grammar.

*Supposedly Neovim will be using this going forward

On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
> https://tree-sitter.github.io/tree-sitter/
> https://github.com/tree-sitter/tree-sitter
> https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)
>
> Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser.

I wonder how it compares to Pegged.

On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
> https://tree-sitter.github.io/tree-sitter/
> https://github.com/tree-sitter/tree-sitter
> https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)
>
> Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser. Haven't looked in depth at the grammar yet -- appears to be context free grammar.
>
> *Supposedly Neovim will be using this going forward

I tried making a tree-sitter grammar for D a long while ago, it needed some custom C code for stuff like WYSIWYG strings and some other things I think. Another issue was that D's official grammar not quite always reflecting the compiler's actual behavior.

On Monday, 22 February 2021 at 15:56:39 UTC, James Blachly wrote:
> https://tree-sitter.github.io/tree-sitter/
> https://github.com/tree-sitter/tree-sitter
> https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)
>
> Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser. Haven't looked in depth at the grammar yet -- appears to be context free grammar.
>
> *Supposedly Neovim will be using this going forward

tree-sitter is a GLR parser. This makes sense for a general purpose text editor like Atom and Neovim.

It can handle more grammars
It can handle grammars that require fixups in the parser better (e.g. C++)
It can potentially give better syntax error reporting

But does come at a cost, and these benefits might not apply to an editor that handles just one grammar (Like a D IDE)

On 2021-02-22 16:56, James Blachly wrote:
> https://tree-sitter.github.io/tree-sitter/
> https://github.com/tree-sitter/tree-sitter
> https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)
> 
> Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser. Haven't looked in depth at the grammar yet -- appears to be context free grammar.
> 
> *Supposedly Neovim will be using this going forward

I wonder how the text editor/IDE integration is supposed to work. To build the parser you need Node.js and a C compiler. Is the author of the grammar supposed to pre-compile everything and it's loaded as a dynamic library? That would defiantly make it less flexible that the current approach TextMate grammars. TextMate grammars are very easy to modify on the fly and the editor will immediately be able to use the new grammar. No building, no compilation and no loading of dynamic libraries. But I do know how limiting TextMate grammars are.

I guess in theory the editor could ship with Node.js and a C compiler and compile on the fly to have the same flexibility as TextMate grammars.

BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine would work. At least macOS already ships with a JavaScript engine.

-- 
/Jacob Carlborg

On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:
> On 2021-02-22 16:56, James Blachly wrote:
>> https://tree-sitter.github.io/tree-sitter/
>> https://github.com/tree-sitter/tree-sitter
>> https://news.ycombinator.com/item?id=26225298 (currently #1 on HN)
>> 
>> Will be keeping an eye on this. If it gains traction*, it would be nice (and possibly important) to have a Dlang parser. Haven't looked in depth at the grammar yet -- appears to be context free grammar.
>> 
>> *Supposedly Neovim will be using this going forward
>
> I wonder how the text editor/IDE integration is supposed to work. To build the parser you need Node.js and a C compiler. Is the author of the grammar supposed to pre-compile everything and it's loaded as a dynamic library? That would defiantly make it less flexible that the current approach TextMate grammars. TextMate grammars are very easy to modify on the fly and the editor will immediately be able to use the new grammar. No building, no compilation and no loading of dynamic libraries. But I do know how limiting TextMate grammars are.
>
> I guess in theory the editor could ship with Node.js and a C compiler and compile on the fly to have the same flexibility as TextMate grammars.
>
> BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine would work. At least macOS already ships with a JavaScript engine.

tree-sitter is written in C, and all parsers are written in C. Atom is the primary editor using tree-sitter, that uses Node.js (/Electron).

I also forgot to mention the other major benefit/goal to tree-sitter. It is an incremental parser, so the parse tree is updated at every keystoke. (this is mentioned in this threads title).

Is this discussion for parsing in general, or for the D frontend/IDEs?

On Friday, 26 February 2021 at 20:06:46 UTC, Jacob Carlborg wrote:
> I wonder how the text editor/IDE integration is supposed to work. To build the parser you need Node.js and a C compiler. Is the author of the grammar supposed to pre-compile everything and it's loaded as a dynamic library?

Yes.

> That would defiantly make it less flexible that the current approach TextMate grammars. TextMate grammars are very easy to modify on the fly and the editor will immediately be able to use the new grammar. No building, no compilation and no loading of dynamic libraries. But I do know how limiting TextMate grammars are.

Yes, there is definitely an edit-compile-test cycle.

The D grammar I am working on currently takes about 15 seconds to compile.

> I guess in theory the editor could ship with Node.js and a C compiler and compile on the fly to have the same flexibility as TextMate grammars.

The great majority of users won't need to edit the grammar, so I think this is not an important goal to aim for. Parsing speed would probably be better appreciated than the ability to edit the grammar on the fly.

> BTW, I'm wondering if it needs to be Node.js or if any JavaScript engine would work. At least macOS already ships with a JavaScript engine.

I believe the tool chain is built on node/npm.

Grammar authors seem to keep the generated C files in git, so running the JavaScript part isn't needed to get the usable .so file.

For the future, I understand that the plan is to target WebAssembly instead. WebAssembly binaries are portable and it executes sufficiently quickly.