I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)

Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.

The following dub pre-generation hooks were necessary:

- run src/idgen.d to generate src/id.d
- create verstr.h and SYSCONFDIR.imp
- create and link source/ddmd/ to src/

The last point makes ddmd modules reside in its correct package. I'm using a symbolic link for this, which is the only reason this approach is currently limited to Linux. In the long run, I think the cleanest way would be to place all ddmd files in src/ddmd/ and just leave mars.d and things like idgen.d in the main src/ directory.

For demonstration purposes, here is a dub package to play around with:
http://code.dlang.org/packages/ddmd-experimental

Here is the dub.json that was used:
https://github.com/aneas/ddmd-experimental/blob/master/dub.json

And here is an example program using this API, have fun!
(run with $ chmod +x file.d; ./file.d)
---
#!/usr/bin/env dub
/+ dub.sdl:
name "ddmd-example"
dependency "ddmd-experimental" version="~>2.71.2-b2dub"
+/

import std.path;
import std.stdio;
import std.string;
import ddmd.builtin;
import ddmd.dmodule;
import ddmd.dclass;
import ddmd.dsymbol;
import ddmd.expression;
import ddmd.func;
import ddmd.globals;
import ddmd.id;
import ddmd.identifier;
import ddmd.mtype;
import ddmd.visitor;

void main(string[] args) {
	if(args.length != 2) {
		writeln("prints top-level function and class declarations");
		writeln("usage: ", args[0].baseName, " d-filepath");
		return;
	}

	// Initialization
	global._init();
	global.params.isLinux = true;
	Type._init();
	Id.initialize();
	Module._init();
	Expression._init();
	builtin_init();

	// Read and parse specified module
	auto id = Identifier.idPool(args[1].baseName.stripExtension);
	auto m = new Module(args[1].toStringz, id, false, false);
	m.read(Loc());
	m.parse();

	// Output
	m.accept(new class Visitor {
		extern(C++):
		alias visit = Visitor.visit;
		override void visit(Module m) {
			foreach(i; 0 .. m.members.dim)
				(*m.members)[i].accept(this);
		}
		override void visit(Dsymbol s) {
		}
		override void visit(FuncDeclaration fd) {
			writeln("function ", fd.ident.toString);
		}
		override void visit(ClassDeclaration cd) {
			writeln("class ", cd.ident.toString);
		}
	});
}

On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
> I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)
>
> Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.

Thx, that's interesting from an IDE developer perspective.

However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.

On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
> I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)
>
> [...]

Nice :)
How about submitting this as PR?
Then you get updates for free and more people can work on tweaking the DUB file.

On Monday, 29 August 2016 at 11:27:44 UTC, Basile B. wrote:
> On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
>> I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)
>>
>> Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.
>
> Thx, that's interesting from an IDE developer perspective.
>
> However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.

Right. With these limitations, it might also be useful for writing static analysis tools, coding guidelines checkers, ...

If someone wants to write a tool that tells me which functions and methods are not reachable from main(), or could be declared pure nothrow @nogc, i would be more than happy to use it :)

On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:
> On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
>> I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)
>>
>> [...]
>
> Nice :)
> How about submitting this as PR?
> Then you get updates for free and more people can work on tweaking the DUB file.

I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.

On 29/08/2016 11:54 PM, Alexander Breckel wrote:
> On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote:
>> On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
>>> I just tried to use the DMD frontend as a Dub package in my own
>>> project to parse and analyze D files and it ... just worked. The
>>> dub.json for dmd is fairly small and doesn't require any changes to
>>> the code. This might be common knowledge, but I was completely
>>> unprepared for this :)
>>>
>>> [...]
>>
>> Nice :)
>> How about submitting this as PR?
>> Then you get updates for free and more people can work on tweaking the
>> DUB file.
>
> I'm afraid this is not yet ready for upstream. Dmd would first have to
> move all ddmd files into src/ddmd/ to have a clean package namespace.
> And someone smarter than me should probably adjust the dub.json to
> somehow emphasize that the library contains only the front-end.

With the help of sourceFiles you should be fine.
You will also want sourcePaths to be set to [""] to ensure it doesn't have anything e.g. src in it.

I would recommend you have dub pregenerate some import files and add them via importPaths.

I didn't see your announcement, but... AWESOME!!

This could be the basis for some really good tutorials on making compiler backends etc...

We need more little teaser examples like the one you posted in the beginning of this thread.

PS: I did already check the code out on github because I watch
code.dlang.org (a lot).

Thanks!



On Mon, Aug 29, 2016 at 12:42 PM, Alexander Breckel via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :)
>
> Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated.
>
> The following dub pre-generation hooks were necessary:
>
> - run src/idgen.d to generate src/id.d
> - create verstr.h and SYSCONFDIR.imp
> - create and link source/ddmd/ to src/
>
> The last point makes ddmd modules reside in its correct package. I'm using a symbolic link for this, which is the only reason this approach is currently limited to Linux. In the long run, I think the cleanest way would be to place all ddmd files in src/ddmd/ and just leave mars.d and things like idgen.d in the main src/ directory.
>
> For demonstration purposes, here is a dub package to play around with: http://code.dlang.org/packages/ddmd-experimental
>
> Here is the dub.json that was used: https://github.com/aneas/ddmd-experimental/blob/master/dub.json
>
> And here is an example program using this API, have fun!
> (run with $ chmod +x file.d; ./file.d)
> ---
> #!/usr/bin/env dub
> /+ dub.sdl:
> name "ddmd-example"
> dependency "ddmd-experimental" version="~>2.71.2-b2dub"
> +/
>
> import std.path;
> import std.stdio;
> import std.string;
> import ddmd.builtin;
> import ddmd.dmodule;
> import ddmd.dclass;
> import ddmd.dsymbol;
> import ddmd.expression;
> import ddmd.func;
> import ddmd.globals;
> import ddmd.id;
> import ddmd.identifier;
> import ddmd.mtype;
> import ddmd.visitor;
>
> void main(string[] args) {
>         if(args.length != 2) {
>                 writeln("prints top-level function and class
> declarations");
>                 writeln("usage: ", args[0].baseName, " d-filepath");
>                 return;
>         }
>
>         // Initialization
>         global._init();
>         global.params.isLinux = true;
>         Type._init();
>         Id.initialize();
>         Module._init();
>         Expression._init();
>         builtin_init();
>
>         // Read and parse specified module
>         auto id = Identifier.idPool(args[1].baseName.stripExtension);
>         auto m = new Module(args[1].toStringz, id, false, false);
>         m.read(Loc());
>         m.parse();
>
>         // Output
>         m.accept(new class Visitor {
>                 extern(C++):
>                 alias visit = Visitor.visit;
>                 override void visit(Module m) {
>                         foreach(i; 0 .. m.members.dim)
>                                 (*m.members)[i].accept(this);
>                 }
>                 override void visit(Dsymbol s) {
>                 }
>                 override void visit(FuncDeclaration fd) {
>                         writeln("function ", fd.ident.toString);
>                 }
>                 override void visit(ClassDeclaration cd) {
>                         writeln("class ", cd.ident.toString);
>                 }
>         });
> }
>
>

On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
>

Because of the poor memory management in the compiler, I included a modified GC-stub when I compiled the frontend as a library, so that it can be used in long-running processes:

https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819

You can call gc_annihilate(true) to delete the entire heap.

Obviously you need to keep the library in a separate DLL with its own runtime, and carefully avoid holding references to GC memory across annihilations.

This technique is working pretty smoothly for me so far.

Also to compile it as a DLL you either have to remove main() from mars.d or play games with the C runtime: https://gist.github.com/Cauterite/b190e62891c773703d0de3a1d99df362

Thanks, for the GC stub, that will be great for playing with whether or not a little dmd app crashes after gc_annihilate(true).

Did I understand that right?

R


On Thu, Sep 1, 2016 at 6:16 PM, Cauterite via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote:
>
>>
>>
> Because of the poor memory management in the compiler, I included a modified GC-stub when I compiled the frontend as a library, so that it can be used in long-running processes:
>
> https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819
>
> You can call gc_annihilate(true) to delete the entire heap.
>
> Obviously you need to keep the library in a separate DLL with its own runtime, and carefully avoid holding references to GC memory across annihilations.
>
> This technique is working pretty smoothly for me so far.
>
> Also to compile it as a DLL you either have to remove main() from mars.d or play games with the C runtime: https://gist.github.com/Cauter ite/b190e62891c773703d0de3a1d99df362
>

On Thu, 01 Sep 2016 18:42:13 +0200, Rory McGuire via Digitalmars-d wrote:

> Thanks, for the GC stub, that will be great for playing with whether or not a little dmd app crashes after gc_annihilate(true).
> 
> Did I understand that right?

It sounds like this there's a special function `gc_annihilate` that immediately frees all memory that libdmd has allocated. It doesn't do any reference tracking or mark/sweep. libdmd disables the collector as dmdfe does.

You need to keep dmd in its own DLL. Otherwise, symbols between dmd and your application will be deduplicated or something and either you get a linker error or an arbitrary implementation is selected.

Another safe way of working is to use the GC as usual, but before you call into libdmd, you call GC.disable, and you wait until you've finished using the data structures it returns before calling GC.enable.