On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
> I wonder if it is possible to somehow turn D in to a scripting language that can be run inside D?
>
> The point? To have the same uniform syntax for quickly developing scripts that can then be easily transferred, if desired, in to a complete binary.
>
> e.g., suppose I am working in some type of analysis software. Use a Dscript like feature to develop and test different analysis algorithms quickly(rather than using the compile and execute model)... then once everything is working, move the code to a D file and compile it directly.
>
> Since the syntax would be identical(or nearly so) it would be quite easy to copy and paste... unlike, say, using lua or some other non-D like scripting language.

You might find the conversation on this thread interesting:
http://forum.dlang.org/thread/tssitxrniaxacpktjsat@forum.dlang.org

On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
> I wonder if it is possible to somehow turn D in to a scripting language that can be run inside D?

On a game project I'm working on at home, I've done:

- Hot reloading via a DLL
- A build script that runs in the background, detects file changes, and automatically rebuilds
- A code structure that keeps build times to a minimum (currently 1.8s)

All these combined, and D feels pretty script-like. The setup is far from ideal, since it imposes some limitations on the language (only limited use of classes/finalizers, be careful with static data, some changes to the program state struct require a restart, etc). It also took a significant amount of work to get it up and running, requiring several changes to druntime.

But writing an entire feature with the game still running, followed by testing and iterating, all without closing the game, is pretty great. Way nicer than at work, where I have 1-15 minute rebuilds, with a restart for every change.

On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
> I wonder if it is possible to somehow turn D in to a scripting language that can be run inside D?
>
> The point? To have the same uniform syntax for quickly developing scripts that can then be easily transferred, if desired, in to a complete binary.
>
> e.g., suppose I am working in some type of analysis software. Use a Dscript like feature to develop and test different analysis algorithms quickly(rather than using the compile and execute model)... then once everything is working, move the code to a D file and compile it directly.
>
> Since the syntax would be identical(or nearly so) it would be quite easy to copy and paste... unlike, say, using lua or some other non-D like scripting language.

Stefan Koch has written a good part of an interpreter for D AST, no? And I guess the lexing and parsing stage doesn't take so long, whereas not having to link saves much time.

So is there any way to repurpose his work on ctfe to have an interpreter that you can call at run time, set context for and get return values back?

On Friday, 2 June 2017 at 16:13:03 UTC, Laeeth Isharc wrote:
> On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
>> [...]
>
> Stefan Koch has written a good part of an interpreter for D AST, no? And I guess the lexing and parsing stage doesn't take so long, whereas not having to link saves much time.
>
> So is there any way to repurpose his work on ctfe to have an interpreter that you can call at run time, set context for and get return values back?

No there is not.
First it's woefully incomplete and secondly it's tightly bound to dmd, and it's semantic phases

On Fri, Jun 02, 2017 at 05:22:20PM +0000, Stefan Koch via Digitalmars-d-learn wrote:
> On Friday, 2 June 2017 at 16:13:03 UTC, Laeeth Isharc wrote:
> > On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
> > > [...]
> > 
> > Stefan Koch has written a good part of an interpreter for D AST, no? And I guess the lexing and parsing stage doesn't take so long, whereas not having to link saves much time.
> > 
> > So is there any way to repurpose his work on ctfe to have an interpreter that you can call at run time, set context for and get return values back?
> 
> No there is not.  First it's woefully incomplete and secondly it's tightly bound to dmd, and it's semantic phases

Hmm. I wonder if there's a way for, say, ldc to export llvm bytecode of a given D program, such that another program can load this bytecode and interpret it at runtime?  Is it possible to use llvm in such a way?


T

-- 
People tell me that I'm skeptical, but I don't believe them.

On Friday, 2 June 2017 at 17:22:20 UTC, Stefan Koch wrote:
> On Friday, 2 June 2017 at 16:13:03 UTC, Laeeth Isharc wrote:
>> On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
>>> [...]
>>
>> Stefan Koch has written a good part of an interpreter for D AST, no? And I guess the lexing and parsing stage doesn't take so long, whereas not having to link saves much time.
>>
>> So is there any way to repurpose his work on ctfe to have an interpreter that you can call at run time, set context for and get return values back?
>
> No there is not.
> First it's woefully incomplete and secondly it's tightly bound to dmd, and it's semantic phases

Maybe it's incomplete today, but some day it will be or could be finished. And whilst I understand the difficulty of working with the dmd codebase as it's structured today, why is it intrinsically a problem that your work is bound to dmd?

T - interesting idea about ldc though that's a bit slower than dmd.

On Friday, 2 June 2017 at 17:50:30 UTC, H. S. Teoh wrote:
> On Fri, Jun 02, 2017 at 05:22:20PM +0000, Stefan Koch via Digitalmars-d-learn wrote:
>> On Friday, 2 June 2017 at 16:13:03 UTC, Laeeth Isharc wrote:
>> > [...]
>> 
>> No there is not.  First it's woefully incomplete and secondly it's tightly bound to dmd, and it's semantic phases
>
> Hmm. I wonder if there's a way for, say, ldc to export llvm bytecode of a given D program, such that another program can load this bytecode and interpret it at runtime?  Is it possible to use llvm in such a way?
>
>
> T

Yes it is possible but you will be astonished how slow this will be.
It is probably more viable run dmd and have it compile into a shared library.

On Friday, 2 June 2017 at 15:55:53 UTC, Lewis wrote:
> On Friday, 2 June 2017 at 02:06:27 UTC, Mike B Johnson wrote:
>> I wonder if it is possible to somehow turn D in to a scripting language that can be run inside D?
>
> On a game project I'm working on at home, I've done:
>
> - Hot reloading via a DLL
> - A build script that runs in the background, detects file changes, and automatically rebuilds
> - A code structure that keeps build times to a minimum (currently 1.8s)
>
> All these combined, and D feels pretty script-like. The setup is far from ideal, since it imposes some limitations on the language (only limited use of classes/finalizers, be careful with static data, some changes to the program state struct require a restart, etc). It also took a significant amount of work to get it up and running, requiring several changes to druntime.
>
> But writing an entire feature with the game still running, followed by testing and iterating, all without closing the game, is pretty great. Way nicer than at work, where I have 1-15 minute rebuilds, with a restart for every change.

Would you mind, when you get some time, to write up a more detailed analysis of the problems you had to overcome to get it to work? Possibly we could get some type of library solution that just "works" with very little change and restriction?

After all, dll's effectively already solve the problem, in some sense... except they are generally not meant to be reloaded on demand. Solving the issues that reloading causes, I think, would be the bulk of the problem?

On Friday, 2 June 2017 at 20:47:31 UTC, Mike B Johnson wrote:
> Would you mind, when you get some time, to write up a more detailed analysis of the problems you had to overcome to get it to work? Possibly we could get some type of library solution that just "works" with very little change and restriction?

For sure. I actually want to post the source code at some point, but the changes I made are very much set up specifically for this project. I'll sift through at some point and see if I can gather together something worth posting.

The main problem with hot-reloading DLLs stems from the fact that every new D DLL comes with its own copy of druntime. This means a second GC, a second registry of active threads, a second registry of typeinfos, and so on. If there were a way to load a D DLL that automatically used the executable's copy of druntime, most of the hacks I had to make could be removed.

For reference, my general approach is that I have an EXE and a DLL. We run the EXE, which makes a copy of the DLL and loads the copy. Once per frame, the EXE calls GameUpdateAndRender() on the DLL, which does everything needed to step the game forward one frame. After each call to GameUpdateAndRender(), the EXE checks to see if the original DLL has changed. If so, we unload the DLL, make a copy of the new one, and load the copy. We then continue calling GameUpdateAndRender() as usual. All game state that needs to survive a hot reload is stored directly (or referenced indirectly) from a giant struct call the GameState. The EXE keeps a void* to the GameState, that way the GC doesn't collect anything inside it.

The main issues I remember off the top of my head:

- I have to use the GC proxy to redirect DLL GC calls over to the EXE's GC. This is because when I hot reload, the DLL's GC dies, so it can't be holding on to anything that needs to survive the hot reload (and I want pretty much everything to survive a hot reload).

- D has some built-in support for redirecting GC calls. However when you load a DLL, by the time you get a chance to call gc_setProxy(), it's too late, and druntime has already done a few allocations into the DLL GC. I had to change the order of a couple parts of initialization (in runtime.d I think?), and in my own code defer calling dll_attach_process() until after DllMain(). That way I get a chance to set up the GC proxy, then initialize druntime, which will trigger a few allocations that now go to the right place.

- I haven't experimented thoroughly with threading, but I think I have to do something similar. Since we're using the EXE's GC, if the EXE's druntime doesn't know about the existence of a thread spawned through the DLL's druntime, then we run into problems. So I redirect all thread spawning over to the EXE.

- With threading, we have to be careful that we don't tear down the DLL from underneath a thread that might still be running code on it. We have two options. 1. Join all threads before unloading the DLL.
2. The thread needs to be running a procedure that originates in the EXE, and then calls into the DLL, returning periodically back to the EXE. While execution is back in the EXE, we take that moment to reload the DLL.

Currently I'm doing the former, but might move to the latter if I end up needing long-lived threads for performance reasons. Note that callbacks originating from C code running on a separate thread also have this same restriction, and need to be handled careful.

- The different druntimes have different typeinfos for each class. This can be problematic for a few reasons, and I don't have as total a grasp on what situations related to classes are guaranteed to cause a crash. In particular, finalizers seem to cause problems, so I have disabled them entirely in druntime. In my own code, I almost exclusively use structs, and any instances of classes I do use are never stored long-term, and are never allowed to survive a hot reload.

I think those are all the D-specific gotchas, but I'll double-check my notes when I get home. In addition, there are common gotchas to this approach that would exist in other languages (C/C++ for example). Examples:

- Static variables will die in a hot reload, so be cautious using them.
- Storing a char* or string that references string in the data segment of the DLL will cause problems when we hot reload. To avoid this, I have a templated function that, on hot reload, iterates over my entire gamestate and copies each string. Since hot reloading is a debug-only utility for me, this is acceptable.
- Code changes to the gamestate struct will require a restart. Adding a new field to the gamestate does not, since I have some padding room at the end of the struct where I can add new fields during a hot reload.
- Don't hold on to function pointers through a hot reload, as functions addresses are liable to be reshuffled in the new DLL.
- Probably some others I'm forgetting.

In terms of making a library solution, I'm not sure what the best approach would be. But allowing users to compile a "light" D DLL that auto-patches itself back to the calling EXE's druntime would drastically reduce the number of hacks needed to make this work. So I guess that would be the main hurdle to overcome?

On Fri, 2017-06-02 at 12:27 +0000, Adam D. Ruppe via Digitalmars-d- learn wrote:
> On Friday, 2 June 2017 at 09:39:51 UTC, Russel Winder wrote:
> > With D and Go, both of which claim compilation so fast you do not notice:
> 
> 1. Edit
> 2. run rdmd
> 

There is no rdmd on Fedora Rawhide. There is only ldc2. (Indeed Debian Sid has no rdmd either by default, you have to use d-apt.)

> Especially if you only expose to the "script" in D the same functions you'd expose to, say, a javascript script, that compile can be under a tenth of a second.
> 
> But 3/10 of a second isn't bad either and that's about what you'd get with an out of the box setup...

Not enough is made of rdmd. rdmd needs to be separated from dmd so it works with gdc and ldc2.


-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder