I see that DUB has DMD as a library package, but I was not able to understand how to use it.

Is it possible to use DMD as a library within a D program to compile a string to machine code and run the compiled code at runtime?

Thanks,
Saurabh

On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:
> I see that DUB has DMD as a library package, but I was not able to understand how to use it.
>
> Is it possible to use DMD as a library within a D program to compile a string to machine code and run the compiled code at runtime?
>
> Thanks,
> Saurabh

Fundamentally DMD as a library is a front-end. Jitting is to the backend side.
You'll be able to lex and parse the source to get an AST, to perform the semantic passes on this AST and that's all.

Then to run this code you would need to make an AST visitor that will generate the binary code to execute. Even using a specialized library with jitting abilities, such as LLVM-d [1] or libfirm-d [2], this would be *quite* a journey.

[1] https://github.com/MoritzMaxeiner/llvm-d
[2] https://gitlab.com/basile.b/libfirm-d

On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:
> On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:
>> [...]
>
> Fundamentally DMD as a library is a front-end. Jitting is to the backend side.
> You'll be able to lex and parse the source to get an AST, to perform the semantic passes on this AST and that's all.
>
> Then to run this code you would need to make an AST visitor that will generate the binary code to execute. Even using a specialized library with jitting abilities, such as LLVM-d [1] or libfirm-d [2], this would be *quite* a journey.
>
> [1] https://github.com/MoritzMaxeiner/llvm-d
> [2] https://gitlab.com/basile.b/libfirm-d

Thank you. That's enough to get me started tinkering!

Saurabh

On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:
> I see that DUB has DMD as a library package, but I was not able to understand how to use it.
>
> Is it possible to use DMD as a library within a D program to compile a string to machine code and run the compiled code at runtime?
>
> Thanks,
> Saurabh

Another approach:
- include the dmd compiler package with your application
- within your app call the compiler executable and compile the source code to a dll / so
- call the dll / so function

Maybe you will get some trouble with AV software with this approach...

Kind regards
Andre

On Sat, Feb 01, 2020 at 08:01:34PM +0000, Andre Pany via Digitalmars-d-learn wrote: [...]
> Another approach:
> - include the dmd compiler package with your application
> - within your app call the compiler executable and compile the source
> code to a dll / so
> - call the dll / so function
[...]

I've actually done this before in an equation grapher program: the user inputs an equation, the program generates D code to compute the equation, then runs dmd to compile it into a shared library, and opens the shared library and looks up the symbol to execute the compiled code. Dmd is fast enough that this actually works fairly well. When the input to dmd is small, it's so fast you don't even notice it.


T

-- 
An imaginary friend squared is a real enemy.

On Saturday, 1 February 2020 at 20:37:03 UTC, H. S. Teoh wrote:
> On Sat, Feb 01, 2020 at 08:01:34PM +0000, Andre Pany via Digitalmars-d-learn wrote: [...]
>> Another approach:
>> - include the dmd compiler package with your application
>> - within your app call the compiler executable and compile the source
>> code to a dll / so
>> - call the dll / so function
> [...]
>
> I've actually done this before in an equation grapher program: the user inputs an equation, the program generates D code to compute the equation, then runs dmd to compile it into a shared library, and opens the shared library and looks up the symbol to execute the compiled code. Dmd is fast enough that this actually works fairly well. When the input to dmd is small, it's so fast you don't even notice it.
>
>
> T

This approach seems more tractable at present. Would you have any example code lying around for this?

Saurabh

On Sun, Feb 02, 2020 at 03:16:46AM +0000, Saurabh Das via Digitalmars-d-learn wrote:
> On Saturday, 1 February 2020 at 20:37:03 UTC, H. S. Teoh wrote:
[...]
> > I've actually done this before in an equation grapher program: the user inputs an equation, the program generates D code to compute the equation, then runs dmd to compile it into a shared library, and opens the shared library and looks up the symbol to execute the compiled code.  Dmd is fast enough that this actually works fairly well. When the input to dmd is small, it's so fast you don't even notice it.
[...]
> This approach seems more tractable at present. Would you have any example code lying around for this?
[...]

It's very simple. Let's say you have your code in some string called 'code'. Since dmd nowadays can take stdin as input (specify "-" as input filename), all you have to do is to assemble your dmd command and use std.process's awesome API to run it:

	/*
	 * Step 1: Compile the code
	 */
	string code = ...;
	auto cmd = [
		"/usr/bin/dmd", // or wherever your dmd is
		"-O",	// or whatever other flags you need
		"-fPIC", "-shared", // this is important
		"-of" ~ soFilename, // specify output filename
		"-"	// read from stdin
	]

	// This part is a bit involved because we have to spawn the
	// compiler as a child process then write our code string into
	// its stdin.
	// Alternatively, just write your code into a temporary file and
	// pass the filename to dmd, then you can just use
	// std.process.execute() which has a much simpler API.
        import std.process : pipeProcess, Redirect, wait;
        auto pipes = pipeProcess(cmd, Redirect.stdin | Redirect.stdout |
                                      Redirect.stderrToStdout);

	// Send code to compiler
        pipes.stdin.write(code);
        pipes.stdin.flush();
        pipes.stdin.close();

	// Read compiler output (optional)
        auto app = appender!string();
        enum chunkSize = 4096;
        pipes.stdout.byChunk(chunkSize)
                    .copy(app);

        // Wait for compiler to finish
        auto status = wait(pipes.pid);
        auto output = app.data;
        if (status != 0)
            throw new Exception("Failed to compile code:\n" ~ output);

	/*
	 * Step 2: Load the compiled library.
	 */
	// This is for Posix; replace with Windows equivalent if you're
	// on Windows.
	auto libhandle = dlopen(soFilename.toStringz, RTLD_LAZY | RTLD_LOCAL);
	if (libhandle is null) ... /* handle error here */

	// Look up entry point by symbol.
	string entryPoint = ...; /* symbol of library entry point */
	alias FunType = int function(string); // your function signature here
	auto funptr = cast(FunType) dlsym(libhandle, entryPoint.toStringz);

	/*
	 * Step 3: Use the compiled code.
	 */

	// Call the compiled function with whatever arguments.
	int result = funptr("my input");

	...

	// Cleanup once you're done with the library.
	dlclose(libhandle);
	std.file.remove(soFilename);


T

-- 
First Rule of History: History doesn't repeat itself -- historians merely repeat each other.

On Sunday, 2 February 2020 at 06:03:01 UTC, H. S. Teoh wrote:
> On Sun, Feb 02, 2020 at 03:16:46AM +0000, Saurabh Das via Digitalmars-d-learn wrote:
>> [...]
> [...]
>> > [...]
> [...]
>> [...]
> [...]
>
> It's very simple. Let's say you have your code in some string called 'code'. Since dmd nowadays can take stdin as input (specify "-" as input filename), all you have to do is to assemble your dmd command and use std.process's awesome API to run it:
>
> [...]

Thanks. Will try this out.

Saurabh

On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:
> On Friday, 31 January 2020 at 11:19:37 UTC, Saurabh Das wrote:
>> I see that DUB has DMD as a library package, but I was not able to understand how to use it.
>>
>> Is it possible to use DMD as a library within a D program to compile a string to machine code and run the compiled code at runtime?
>>
>> Thanks,
>> Saurabh
>
> Fundamentally DMD as a library is a front-end. Jitting is to the backend side.
> You'll be able to lex and parse the source to get an AST, to perform the semantic passes on this AST and that's all.
>
> Then to run this code you would need to make an AST visitor that will generate the binary code to execute. Even using a specialized library with jitting abilities, such as LLVM-d [1] or libfirm-d [2], this would be *quite* a journey.
>
> [1] https://github.com/MoritzMaxeiner/llvm-d
> [2] https://gitlab.com/basile.b/libfirm-d

about [1] (llvm) I've made a better binding this weekend:

https://gitlab.com/basile.b/llvmd-d

Seriouly I cant believe that at some point in the past I translated by hand. dstep can handle big C libraries.

On Monday, 10 February 2020 at 12:31:03 UTC, Basile B. wrote:
> On Friday, 31 January 2020 at 14:25:30 UTC, Basile B. wrote:
>> [...]
>
> about [1] (llvm) I've made a better binding this weekend:
>
> https://gitlab.com/basile.b/llvmd-d
>
> Seriouly I cant believe that at some point in the past I translated by hand. dstep can handle big C libraries.

ah ah ah, I meant

https://gitlab.com/basile.b/llvm-d