I've finally managed to sever its dependencies on the rest of the compiler. You can now build a standalone lexer, or a standalone D parser, or a standalone ImportC parser.

You can use it to build:

1. syntax highligher
2. source code formatter
3. syntax checker
4. your own semantic processor
5. something that annotates D code

Even without 1..5, having it be separate from the rest of the compiler makes it much easier to understand and work with.

https://github.com/dlang/dmd/pull/15091

I wanted to double check something about this, but I think that this will only be temporary unfortunately (assuming it has been successfully split, which we can't know because it hasn't been verified at the CI level).

Both build.d and dub will allow things to bleed through and inter-dependency can be added accidentally.

Dub is simple, as long as its in the same directory structure, it'll be -I'd and hence available (although it may not link).

build.d compiles the frontend (minus the lexer, common and root) in one go. So therefore ImportC parser and the D one, can become inter-dependent. Along with all the semantic analysis ext.

https://github.com/dlang/dmd/blob/master/compiler/src/build.d#L1544

Not quite time to start the celebrations just yet :/

On Monday, 10 April 2023 at 02:02:49 UTC, Walter Bright wrote:
> I've finally managed to sever its dependencies on the rest of the compiler. You can now build a standalone lexer, or a standalone D parser, or a standalone ImportC parser.

Appreciate the effort!

From my brief attempts I've ran into a few problems:

- `Parser` requires `ErrorSink` in its constructor, but `dmd.errorsink` is not in DMD's  `dub.sdl`. So using `Parser` directly doesn't seem to work (errorsink module not found). Maybe it's an easy fix to dub.sdl..?
- On Linux I'm getting linking errors. While on Windows I get a segfault. Probably because I am missing some important initialization somewhere.

-----
import std.stdio;
import dmd.dmodule;
import dmd.identifier;
import dmd.location;

void main()
{
    auto mod = new Module(`C:\DMD\dmd2\src\phobos\std\file.d`, new Identifier("file"), false, false);
    auto loc = Loc("app.d", 0, 0);
    if (!mod.read(loc))
        assert(0, "can't read");
}
-----

Linux:

-----
Linking dparser
/usr/bin/ld: /home/drey/.dub/cache/dmd/~master/+lexer/build/library-debug-linux.posix-x86_64-dmd_v2.102.0-9CFF4D64344FD218B941F4DE6FAF1B94F5E5640E5F33D1921150D7105AD91F2E/libdmd_lexer.a(errors_66_6da.o):(.data._D3dmd6errors17ErrorSinkCompiler7__ClassZ+0x50): undefined reference to `_D3dmd9errorsink9ErrorSink7__ClassZ'
/usr/bin/ld: /home/drey/.dub/cache/dmd/~master/+lexer/build/library-debug-linux.posix-x86_64-dmd_v2.102.0-9CFF4D64344FD218B941F4DE6FAF1B94F5E5640E5F33D1921150D7105AD91F2E/libdmd_lexer.a(errors_88_732.o): in function `_D3dmd6errors18colorHighlightCodeFNbKSQBk6common9outbuffer9OutBufferZv':
/home/drey/.dub/packages/dmd-master/dmd/compiler/src/dmd/errors.d:832: undefined reference to `_D3dmd9errorsink13ErrorSinkNull7__ClassZ'
-----

On Windows it segfaults somewhere in `StringTable` on an invalid pointer, so probably something needs to be initialized.

But the real problem here is the lack of documentation, or a lack of examples of how to use these classes outside of the compiler itself.

It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.

On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:
> It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.

There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package

But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.

On Monday, 10 April 2023 at 12:30:53 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:
>> It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.
>
> There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package
>
> But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.

Oh thanks, that's better. I see there's a call to initDMD(), it's probably what I was missing.

On Monday, 10 April 2023 at 12:30:53 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 11/04/2023 12:17 AM, Andrej Mitrovic wrote:
>> It would be nice to have a very minimal sample dub project that uses these Module / Parser classes.
>
> There is: https://github.com/dlang/dmd/tree/master/compiler/test/dub_package
>
> But yeah, there needs to be working examples for a formatter + auto completion engine. That is both tested and with good documentation; those two are probably the most common use cases for such a tool.

Formatting off an AST directly is quite hard to get perfect.

On 14/04/2023 1:24 AM, Andrej Mitrovic wrote:
> Maybe I could try with DMD `master`. Although `digger` no longer seems to work so I bet it's gonna be a pain building everything from scratch.

You don't need to build dmd, so digger won't be helpful anyway.

The frontend is pretty isolated as far as overall requirements go.

On Thursday, 13 April 2023 at 13:29:01 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 14/04/2023 1:24 AM, Andrej Mitrovic wrote:
>> Maybe I could try with DMD `master`. Although `digger` no longer seems to work so I bet it's gonna be a pain building everything from scratch.
>
> You don't need to build dmd, so digger won't be helpful anyway.

It's been fixed in the meantime: https://github.com/CyberShadow/Digger/issues/98

And yes now I realize this isn't supposed to be used with just a regular D install, as it needs to import into DMD modules which aren't part of the release package.