Another thread on a project from the GSoC wiki to summarize all information, spark discussions and inspire students.

The excerpt of this project from the wiki:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication.
The Language Server Protocol [1] (LSP) defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc.

Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature (see the list of Editors [2] and IDEs [3]). Additionally as the existing tools use `libdparse` (used in DLS [4] and DCD [5]), they lack the ability to perform semantic analysis on the AST and can't evaluate `mixin`'s nor D's very powerful Compile-Time Function Evaluation (CTFE).

The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single D Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort.

Recent development in the DMD compiler made it available as a DUB library [7]. This mean that the compiler itself can be re-used and third-party tools don't have to re-implement e.g. D's complex CTFE. This project is about using the DMD compiler as a library to implement a Language Server for D.

How to start:

* Simple example of using DMD as a library [8]
* More complex use cases [9]

[1] https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
[2] https://wiki.dlang.org/Editors
[3] https://wiki.dlang.org/IDEs
[4] https://microsoft.github.io/language-server-protocol
[5] https://github.com/d-language-server/dls
[6] https://github.com/dlang-community/DCD
[7] https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
[8] https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
[9] https://github.com/jacob-carlborg/dlp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think the project itself contains of two parts:

1) Improving DMD as a Library

We are still actively working on this, so expect that you might need to dive into DMD and refactor things.
Here's a recent example:

https://github.com/dlang/dmd/pull/9350

2) Exposing the DMD as a language server

You probably can get a many inspirations here from the existing tools:

- https://github.com/d-language-server/dls
- https://github.com/dlang-community/DCD


How to start
------------

I think the most crucial step is to familiarize yourself with the DMD codebase. It's very non-trivial and I think the best way is to start fixing easier DMD bugs.
Often ICEs are good for beginners as they already point you to the rough location which most likely needs fixing.

A more concrete suggestion from Jacob
-------------------------------------

When it comes to improving DMD as a library, I think one of the key things would be to reduce global state. When the compiler is used as a library it needs to be able to invoke the library multiple times without changing the outcome. Not be dependent of previous state. We have a few workarounds , like [1], to reset the state, but it would be much better if there wasn’t any global state to begin with.

A related thing is that all diagnostic handling (after the parsing phase) goes through a global function which always outputs to stderr. This makes it basically impossible to configure the diagnostic handling which is useful when using the compiler as a library.

Using the `pure` function attribute helps to identify global state. My tool, DLP [2], allows to identify leaf functions (functions that don’t call other functions), which would be a good start to add attributes on. My tool can also serve as an example of using the DMD as a library.

Perhaps adding to DLP, or writing a new tool, that can suggest attributes that can be added to functions could be a good start. It would both help by learning how to use DMD as a library and the tool can then be used to identify which functions can be `pure` and which can’t.


[1] https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
[2] http://github.com/jacob-carlborg/dlp

Community
---------

@community: what language smarts in an editor are the most important to you? What features do you miss in DLS/DCD?

@students: please don't be shy. This is on open discussion and your questions are welcome!

These two bugs can be considered blockers (Win64):

https://issues.dlang.org/show_bug.cgi?id=18810
https://issues.dlang.org/show_bug.cgi?id=18811

The first shouldn't be too much work for a hacky solution. Port file to D, version as required and move to root/.
The second I haven't retested with the precise GC. It may solve it. Otherwise I have got no idea where to go from here.

On 2019-03-13 12:13, Seb wrote:

> @community: what language smarts in an editor are the most important to you? What features do you miss in DLS/DCD?

* Stability. DCD crashes quite often for me. Then there's this problem [1]. I have a locally patched version, which is required for me to even use DCD.

* I would really like to see the accuracy being improved. For example, autocompletion with support for UFCS.

* Not sure if this is supported by LSP but incremental compilation and reporting compile errors would be really cool.

* I'm not sure how feasible this would be but using Eclipse and Java autocompletion works even without any imports. When an autocompletion item is selected the IDE will automatically add the correct include.

For a bunch of more ideas, here's a list of supported features of an old Eclipse plugin, Descent, for D1 [2]:

* Syntax highlighting.
* Awesome, flexible formatter.
* Folding.
* Automatic closing of strings, brackets, chars and comments.
* Code templates.
* Syntax errors and some semantic errors are shown with a reg squiggle.
* Autocompletion which writes the imports for you.
* Source on hover (shift+hover).
* Go to definition (ctrl+hover).
* Some compile-time function evaluation (ctrl+shift+hover).
* Greying of inactive code.
* Project Explorer.
* Outline view.
* Type Hierarchy view.
* Automatic generation of documentation comments (Params, Author, etc.).
* Spell checking.
* TODO tasks.
* Ddoc view.
* Declaration (source) view.
* Highlighting of all occurrences of the current token under the cursor.
* trace.log viewer
* Compile-time debugging
* Runs and debugs programs within the IDE.

[1] https://github.com/dlang-community/DCD/issues/345
[2] http://www.dsource.org/projects/descent

-- 
/Jacob Carlborg

On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:
>It's very non-trivial and I think the best
> way is to start fixing easier DMD bugs.
> Often ICEs are good for beginners as they already point you to

No. Common Seb. Most of the ICEs are in the backend or in CTFE.

On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:
> Another thread on a project from the GSoC wiki to summarize all information, spark discussions and inspire students.
>
> The excerpt of this project from the wiki:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication.
> The Language Server Protocol [1] (LSP) defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc.
>
> Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature (see the list of Editors [2] and IDEs [3]). Additionally as the existing tools use `libdparse` (used in DLS [4] and DCD [5]), they lack the ability to perform semantic analysis on the AST and can't evaluate `mixin`'s nor D's very powerful Compile-Time Function Evaluation (CTFE).
>
> The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single D Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort.
>
> Recent development in the DMD compiler made it available as a DUB library [7]. This mean that the compiler itself can be re-used and third-party tools don't have to re-implement e.g. D's complex CTFE. This project is about using the DMD compiler as a library to implement a Language Server for D.
>
> How to start:
>
> * Simple example of using DMD as a library [8]
> * More complex use cases [9]
>
> [1] https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
> [2] https://wiki.dlang.org/Editors
> [3] https://wiki.dlang.org/IDEs
> [4] https://microsoft.github.io/language-server-protocol
> [5] https://github.com/d-language-server/dls
> [6] https://github.com/dlang-community/DCD
> [7] https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
> [8] https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
> [9] https://github.com/jacob-carlborg/dlp
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> I think the project itself contains of two parts:
>
> 1) Improving DMD as a Library
>
> We are still actively working on this, so expect that you might need to dive into DMD and refactor things.
> Here's a recent example:
>
> https://github.com/dlang/dmd/pull/9350
>
> 2) Exposing the DMD as a language server
>
> You probably can get a many inspirations here from the existing tools:
>
> - https://github.com/d-language-server/dls
> - https://github.com/dlang-community/DCD
>
>
> How to start
> ------------
>
> I think the most crucial step is to familiarize yourself with the DMD codebase. It's very non-trivial and I think the best way is to start fixing easier DMD bugs.
> Often ICEs are good for beginners as they already point you to the rough location which most likely needs fixing.
>
> A more concrete suggestion from Jacob
> -------------------------------------
>
> When it comes to improving DMD as a library, I think one of the key things would be to reduce global state. When the compiler is used as a library it needs to be able to invoke the library multiple times without changing the outcome. Not be dependent of previous state. We have a few workarounds , like [1], to reset the state, but it would be much better if there wasn’t any global state to begin with.
>
> A related thing is that all diagnostic handling (after the parsing phase) goes through a global function which always outputs to stderr. This makes it basically impossible to configure the diagnostic handling which is useful when using the compiler as a library.
>
> Using the `pure` function attribute helps to identify global state. My tool, DLP [2], allows to identify leaf functions (functions that don’t call other functions), which would be a good start to add attributes on. My tool can also serve as an example of using the DMD as a library.
>
> Perhaps adding to DLP, or writing a new tool, that can suggest attributes that can be added to functions could be a good start. It would both help by learning how to use DMD as a library and the tool can then be used to identify which functions can be `pure` and which can’t.
>
>
> [1] https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
> [2] http://github.com/jacob-carlborg/dlp
>
> Community
> ---------
>
> @community: what language smarts in an editor are the most important to you? What features do you miss in DLS/DCD?
>
> @students: please don't be shy. This is on open discussion and your questions are welcome!

Using the actual DMD compiler may not be that great as a language server.

https://code.visualstudio.com/api/language-extensions/language-server-extension-guide#error-tolerant-parser-for-language-server

https://github.com/Microsoft/tolerant-php-parser/blob/master/docs/HowItWorks.md

It might be good to provide auto complete that is aware of things like mixins, but if you want language features when you are working on an incomplete incorrect source file. Not sure how good DMD will be able to handle that.

Whatever happened to the funding for code-d, did nothing ever come out of that?

On Wednesday, 13 March 2019 at 21:03:21 UTC, Basile B. wrote:
> On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:
>>It's very non-trivial and I think the best
>> way is to start fixing easier DMD bugs.
>> Often ICEs are good for beginners as they already point you to
>
> No. Common Seb. Most of the ICEs are in the backend or in CTFE.

I find ICEs to be the easiest to fix because they are often (but not always) simply dereferencing a null, and sometimes (though not always) the fix is just adding a check for null or something of the like. Tracking them down is also easier because the stack trace tells you exactly where the problem occurs.  Other bugs are MUCH harder to track down.

So, I encourage those who want to get started hacking DMD to go for the ICEs.  Not all fit the description above, but I think they're good for beginners.

Mike

On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:
> @community: what language smarts in an editor are the most important to you? What features do you miss in DLS/DCD?

I agree with Jacob. I would add another feature I miss: methods / properties / etc. should be sorted by relevance. For example overriden methods (or new ones) should come before inherited ones. Also frequency should be a sort key. You can see something like this in android studio.

Andrea

On 2019-03-14 00:32, Rubn wrote:

> It might be good to provide auto complete that is aware of things like mixins, but if you want language features when you are working on an incomplete incorrect source file. Not sure how good DMD will be able to handle that.

If that's required, then that should be part of the GSoC project.

-- 
/Jacob Carlborg

On Thursday, 14 March 2019 at 20:03:29 UTC, Jacob Carlborg wrote:
> On 2019-03-14 00:32, Rubn wrote:
>
>> It might be good to provide auto complete that is aware of things like mixins, but if you want language features when you are working on an incomplete incorrect source file. Not sure how good DMD will be able to handle that.
>
> If that's required, then that should be part of the GSoC project.

Which part? Point is using DMD as a language server isn't going to be intuitive, compilers are designed to fail on invalid input but editors need to work for incomplete source files. Either way I feel this is out of scope for a single student to complete in a single summer. Thinking that most of the work is done because they can use DMD is just backwards and haven't ever written tooling tools.

On 2019-03-14 23:34, Rubn wrote:

> Which part? Point is using DMD as a language server isn't going to be intuitive, compilers are designed to fail on invalid input but editors need to work for incomplete source files.

Then DMD needs to be changed to handle incomplete source files. I'm no expert on this topic but wouldn't it be enough if the lexer and parser can handle incomplete source files? That might be feasible to complete during a summer.

> Either way I feel this is out of scope for a single student to complete in a single summer.

I'm thinking it's very ambitious too.

> Thinking that most of the work is done because they can use DMD is just backwards and haven't ever written tooling tools.

Yes. There's no magic to use DMD as a library. Just compile all the necessary files and create a library out of the object files. It's been possible to do this for as long as DMD has existed. We just created a Dub package and some utility functions to package it nicely.

-- 
/Jacob Carlborg