I am working on DLS (https://github.com/d-language-server/dls), trying to change the implementation to use the compiler as a library. The reason behind this is to make DLS be always up to date with the compiler so that it won't provide any obsolete information.

At the moment, libdparse (https://github.com/dlang-community/libdparse) is used instead and I want to get rid of this dependency. However, as DLS doesn't really do much by itself, since it relies on other tools, such as dcd (https://github.com/dlang-community/DCD), dscanner (https://github.com/dlang-community/D-Scanner), dfmt (https://github.com/dlang-community/dfmt), to do its work. Unfortunately, these dependencies also use libdparse, so removing it from DLS wouldn't be as easy as just adding the dmd logic to the DLS code where libdparse is used.

The cleanest way to solve this would be to replace libdparse with dmd in dcd and dscanner, as well. Would this be a good idea? Should I approach this differently?

On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu wrote:
> ?

https://forum.dlang.org/post/wrwbcujfccjjnnlhwgak@forum.dlang.org

On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu wrote:
> I am working on DLS (https://github.com/d-language-server/dls), trying to change the implementation to use the compiler as a library. The reason behind this is to make DLS be always up to date with the compiler so that it won't provide any obsolete information.
>
> At the moment, libdparse (https://github.com/dlang-community/libdparse) is used instead and I want to get rid of this dependency. However, as DLS doesn't really do much by itself, since it relies on other tools, such as dcd (https://github.com/dlang-community/DCD), dscanner (https://github.com/dlang-community/D-Scanner), dfmt (https://github.com/dlang-community/dfmt), to do its work. Unfortunately, these dependencies also use libdparse, so removing it from DLS wouldn't be as easy as just adding the dmd logic to the DLS code where libdparse is used.
>
> The cleanest way to solve this would be to replace libdparse with dmd in dcd and dscanner, as well. Would this be a good idea? Should I approach this differently?

As DLS' developer, I've looked into doing this myself a while back, close to a year ago now.
At that point I wasn't sure if it was really a good idea; even if DMD has technically been usable as a library for quite some time, in practice it wasn't as straightforward as I had hoped.

From what I recall, the necessary changes to DMD would be:
- Removing any remaining global state. I don't know what the status of global state in DMD is nowadays. Although this might not be "necessary", it's always good to know that we can safely repeat calls to some APIs and always get the same output, without interference from some internal variables.
- Being able to fully control the standard input and output. When I tried to use DMD, there were some places where it was impossible to prevent DMD from using stdout IIRC. This is really critical, as standard input/output is the way language servers are used 99% of the time, and thus they need 110% control over stdin/stdout.
- Being able to free all the memory DMD allocates. From what I've read here, DMD doesn't free memory to get a higher compilation speed. However, language servers are long-running background programs, and any kind of memory leak is extremely bad in this situation. Since the apparition of the `-lowmem` flag, maybe this has improved.

Now, I haven't looked at all that since something like March of last year, so things might be better today (and they probably are, i think I recall Walter not being fond of DMD's global state either). I've gotten my hands busy with other projects, so development on DLS has basically halted and I don't expect to get back on it very soon.
My original plan wasn't to convert DCD/D-Scanner/DFMT, but to rewrite all functionality inside DLS from scratch using DMD. On the other hand, converting those tools would certainly benefit the whole community.

Rainers, VisualD's maintainer, has recently released a version of VisualD that can use DMD as a library; maybe its source code could provide you with some useful insight.

Good luck converting all those tools to use DMD, this is quite some work. If you do follow up on converting DLS, don't hesitate to open up issues in its repo if you have questions about how it works, I'll try my best to remember what I did and why!

On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu wrote:
> [...]

I'll only comment this parapgraph since I don't use DLS:

> The cleanest way to solve this would be to replace libdparse with dmd in dcd and dscanner, as well. Would this be a good idea? Should I approach this differently?

You should rather start new tools. dsymbol (so the library used by dcd) has a lot of problems related to import and out-of-order declarations, templates or even expressions are not handled at all (BTW dparse doesn't decompose unaries at all !).

Replacing dparse by DMD-FE in the tools, from my POV, would be like puting a Rolls-Royce turbo reactor in an old Renault 4L car. The way the tools are made is too bent to dparse.

I and at least one other have discussed this in the past (I am one of the "owners" of dlang-community @rikkimax so feel free to ping).

DCD needs rewriting to use dmd-fe.

Keep or improve feature parity while stripping out how it does everything.

The other tools are in the same position, ignore DLS most IDE's don't use it in the D community for now and as you said it has dependencies on the other programs anyway.

On Thursday, 20 February 2020 at 18:58:12 UTC, Cristian Creteanu wrote:
>
> The cleanest way to solve this would be to replace libdparse with dmd in dcd and dscanner, as well. Would this be a good idea? Should I approach this differently?

Not familiar with DLS / DCD source code, but quite familiar with DMD.
Things have definitely gotten better since the first time I attempted this, but it's definitely non-trivial, depending on what you are trying to do.

I can confirm that the global state and the memory allocation are still issues.
I don't know about standard output, has some PRs have been made to use a delegate (https://github.com/dlang/dmd/pull/10072 and https://github.com/dlang/dmd/pull/10711), so things might be better on that front.

Two other large issues:
- DMD is not built to be able to handle edit cycles, only complete & valid source files.
  When working on an IDE, you'll have to deal with incomplete source code (e.g. typing `format("`) and you want to be able to provide completion for parameter, return value, overload sets, etc... Doing so would require large refactorings in DMD which are likely to raise some eyebrows.
- DMD semantic analysis is destructive. Many things are rewritten and some links are lost, and the rewrite might not be valid D code. One example, any `alias this` is just rewritten as a direct call, but bypassing visibility checks on the target, so writing the same code would produce an error.
 Some expressions are rewritten as ComaExpression and use the return of that expression. This is also very evident with `alias`, as they are resolved as early as possible. There are quite a few places in DMD that work around this issue (e.g. `immutable(char)[]` is special-cased to always show as `string` in error message, and the POSIX C++ mangler uses a mix of pre and post-semantic AST to work around that issue as soon as templates are involved).

On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
> I and at least one other have discussed this in the past (I am one of the "owners" of dlang-community @rikkimax so feel free to ping).
>
> DCD needs rewriting to use dmd-fe.
>
> Keep or improve feature parity while stripping out how it does everything.
>
> The other tools are in the same position, ignore DLS most IDE's don't use it in the D community for now and as you said it has dependencies on the other programs anyway.

If you drop dparse and dsymbol then DCD is an empty shell that only handles the requests and their serialization. Why keeping this ? That are other ways of implementing an IPC than sockets, that were never done in DCD (shared memory, standard IO).

On 21/02/2020 6:21 PM, Basile B. wrote:
> On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
>> I and at least one other have discussed this in the past (I am one of the "owners" of dlang-community @rikkimax so feel free to ping).
>>
>> DCD needs rewriting to use dmd-fe.
>>
>> Keep or improve feature parity while stripping out how it does everything.
>>
>> The other tools are in the same position, ignore DLS most IDE's don't use it in the D community for now and as you said it has dependencies on the other programs anyway.
> 
> If you drop dparse and dsymbol then DCD is an empty shell that only handles the requests and their serialization. Why keeping this ? That are other ways of implementing an IPC than sockets, that were never done in DCD (shared memory, standard IO).

Lets not break existing tooling just because we can.

New IO methods can be added later.

On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
> ignore DLS most IDE's don't use it

I'll chime in a bit to disagree on that. I don't know where you're getting this idea from, but DLS can be used with VSCode, Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly not as popular as code-d and VisualD, but it's certainly usable with a whole range of editors and IDE's

On 21/02/2020 8:24 PM, Laurent Tréguier wrote:
> On Friday, 21 February 2020 at 03:42:37 UTC, rikki cattermole wrote:
>> ignore DLS most IDE's don't use it
> 
> I'll chime in a bit to disagree on that. I don't know where you're getting this idea from, but DLS can be used with VSCode, Visual Studio, Sublime text, Atom, Vim/Neovim, Emacs, and Intellij IDEA. It's certainly not as popular as code-d and VisualD, but it's certainly usable with a whole range of editors and IDE's

For the more used IDE's the community from what I have observed use the dedicated plugins for the IDE's to integrate into external resources like DCD.

I was not commenting on if an IDE supports it, but instead on what the priority is.

Keeping in mind that this is in context on rewriting DLS which depends on these very same resources.