There are many programs out there that use some sort of scripting capabilities.

I've been wanting to write an app that exposes a scripting like environment D, lua, or some other fast language to the user.

But there are two requirements:

1. The user has access to all objects created in app. e.g., if I create a class X in the app then the user can instantiate X in the script without having to recreate it. (the script then, is sort of an extension of the app)

I don't mind actually having to expose X to the script, but it should be easy.

2. It should be fast. When it is "recompiled" it shouldn't take more than a few seconds. It also shouldn't require a re-initialization of everything.

Can D do this easily?

On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:
> There are many programs out there that use some sort of scripting capabilities.
>
> I've been wanting to write an app that exposes a scripting like environment D, lua, or some other fast language to the user.
>
> But there are two requirements:
>
> 1. The user has access to all objects created in app. e.g., if I create a class X in the app then the user can instantiate X in the script without having to recreate it. (the script then, is sort of an extension of the app)
>
> I don't mind actually having to expose X to the script, but it should be easy.
>
> 2. It should be fast. When it is "recompiled" it shouldn't take more than a few seconds. It also shouldn't require a re-initialization of everything.
>
> Can D do this easily?

You know, you can use D as a scripting language. On Linux, you do something like put

#!/bin/rdmd

at the beginning of the file, mark it is executable, and then you can run it. I don't know what you need to put on the top to make it work on Windows.

These days, I write most of my scripts in D.

- Jonathan M Davis

On Tuesday, 6 October 2015 at 20:02:48 UTC, Jonathan M Davis wrote:
> On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:
>> There are many programs out there that use some sort of scripting capabilities.
>>
>> I've been wanting to write an app that exposes a scripting like environment D, lua, or some other fast language to the user.
>>
>> But there are two requirements:
>>
>> 1. The user has access to all objects created in app. e.g., if I create a class X in the app then the user can instantiate X in the script without having to recreate it. (the script then, is sort of an extension of the app)
>>
>> I don't mind actually having to expose X to the script, but it should be easy.
>>
>> 2. It should be fast. When it is "recompiled" it shouldn't take more than a few seconds. It also shouldn't require a re-initialization of everything.
>>
>> Can D do this easily?
>
> You know, you can use D as a scripting language. On Linux, you do something like put
>
> #!/bin/rdmd
>
> at the beginning of the file, mark it is executable, and then you can run it. I don't know what you need to put on the top to make it work on Windows.
>
> These days, I write most of my scripts in D.
>
> - Jonathan M Davis


But this isn't what I want, only part.

Essentially I want to write an app that allows the user to "inteface" with the internals of the app, to control things and such.

Half the app is graphics and provides the backbone, sets up the graphics, deals with all that mess that the user doesn't need to deal with. The user, though, can control specific things such as the colors, objects, and such in the graphics scene. But instead of providing relatively static settings for the user(mess with sliders and such), I would like them to be able to specify what they want in "code".

e.g.,
User script code:

Graphics.Objects["Ship"].X = 10*cos(time);
Graphics.Objects["Ship"].Y = 10*sin(time);


...

So the user has the ability to access the "internals". Here Graphics is a sort of container for all the graphics, Objects is a map of all the objects in the scene.

But ultimately the code should be "compiled" in to the backbone of the app so it runs as fast as possible(not re-interpreted each scene).

One can think of it like this: The user provides code, the code is compiled as a function and the function called by the app. As long as one can link up the objects this should not be difficult.

If I were to do this by hand I'd have to write or use an interpreter or compiler(probably too slow) and provide a mapping of all the objects I want to expose(I'll have to "export" them from the app).

I could use something like LuaD, as I think it has the ability to parse strings as lua code, and the lua code can directly interface with the app. So this would not be that difficult to implement. The questions here are, is it fast enough and can it provide the simplicity I'd want.

I think C# has some type of compiler that allows one, in real time, to re-compile source code chunks and execute them as part of the program(not as separate entities that have no access to the underlying code).

This process would work but surely is too slow:

1. Treat the script code as pure D code.
2. Provide object references(pointers) to the objects I want to expose to the D code(as function arguments or through a lookup table or whatever)
3. compile the code into a dll.
4. Call the code in the dll per frame(or how ever I need to).


This would work great except the recompiling part and dll interfacing would surely be too slow. Since the user code could be modified often(modify a line, recompile), I need the changes to happen as fast as possible. If one wanted to modify the above code to

Graphics.Objects["Ship"].X = 10*sin(time);
Graphics.Objects["Ship"].Y = 10*cos(time);

I don't want the user to wait 5 mins while the app does all the work listed. Even, if it's 50 seconds it's still too slow.

I'd also like to use D as the scripting language since it would probably flow better(although, maybe lua and python could work too).

My app essentially has a visual scene representing the 3d graphics, and a "script editor" that allows the user to interact and modify the scene using code.

The scripting is crucial as it what makes the app what it is.

On Tuesday, 6 October 2015 at 21:41:18 UTC, Jacob wrote:
>> On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:
>>> 
>>> I've been wanting to write an app that exposes a scripting like environment D, lua, or some other fast language to the user.

PyD works nicely and I have used it enough to feel comfortable, but fast it isn't.

Isn't LuaJIT the obvious answer for you ?  You can use the FFI and generate C headers, or use LuaD.  I am struggling a bit with LuaD at the moment, so I can't tell you for sure it's perfectly usable, but you have the backdrop of knowing that the worst case of dropping down to C style is still pretty easy.  Note that the FFI allows you to wrap foreign methods nicely for constructors etc.  it even deals with templated types !

LuaJIT is shockingly fast, and Lua itself seems the perfect scripting language.

I am replying now as I have the exact same problem in a different domain myself, and tentatively going with Lua even though I know Python and PyD better.  LuaD was complaining and trying to wrap some private methods, enums etc.  my meta programming is at an early stage, and I am trying to fix these teething problems.  But it shouldn't be too bad - I am not far off, I think.

Also it currently harmlessly segfaults on exit due to a change in compiler behaviour interacting with LuaObject destructor (Ponce may have the answer) and creating a D module for Lua doesn't work for me (segfaults) on Arch Linux 64 although embedding Lua is fine.  I am not worried about it because I know I have C API as last resort, and it's coming along nicely anyway.

>>> 1. The user has access to all objects created in app. e.g., if I create a class X in the app then the user can instantiate X in the script without having to recreate it. (the script then, is sort of an extension of the app)
>>>
>>> I don't mind actually having to expose X to the script, but it should be easy.

Should be easy.
>>>
>>> 2. It should be fast. When it is "recompiled" it shouldn't take more than a few seconds. It also shouldn't require a re-initialization of everything.
>>>
>>> Can D do this easily?

You won't need to precompile Lua but if you do it will be quick.

>
> Half the app is graphics and provides the backbone, sets up the graphics, deals with all that mess that the user doesn't need to deal with. The user, though, can control specific things such as the colors, objects, and such in the graphics scene. But instead of providing relatively static settings for the user(mess with sliders and such), I would like them to be able to specify what they want in "code".

Yes - exactly...  Back end power, correctness, efficiency with a light scripting interface.
>

> But ultimately the code should be "compiled" in to the backbone of the app so it runs as fast as possible(not re-interpreted each scene).

LuaJIT will be very fast.  I think calling back to C might not be something you want to do in an inner loop though, so structure accordingly.  It's a drop in replacement for Lua 5.1, which is the version supported by LuaD so just change the library you link against.  Beware that you might need to compile LuaJIT  on Linux with don't disable stack frame and some other weirdness on Windows.  And that LuaD is usable but docs could be more complete and it's a little rough around edges.

> I could use something like LuaD, as I think it has the ability to parse strings as lua code, and the lua code can directly interface with the app. So this would not be that difficult to implement. The questions here are, is it fast enough and can it provide the simplicity I'd want.

I think yes and yes.  Can, but you might spend an afternoon or two grumbling at it before you figure it out.

> 1. Treat the script code as pure D code.
> 2. Provide object references(pointers) to the objects I want to expose to the D code(as function arguments or through a lookup table or whatever)
> 3. compile the code into a dll.
> 4. Call the code in the dll per frame(or how ever I need to).
>
>
> This would work great except the recompiling part and dll interfacing would surely be too slow. Since the user code could be modified often(modify a line, recompile), I need the changes to happen as fast as possible. If one wanted to modify the above code to

Have a look at D REPL or Jupyter notebook for what's possible with compiling and dynamically loading libraries.  Strikes me as a no brainer to use Lua.  But not because of compilation speed - for D that should be a handful of seconds at most (Phobos compiles in four seconds)

> I'd also like to use D as the scripting language since it would probably flow better(although, maybe lua and python could work too).
>
> My app essentially has a visual scene representing the 3d graphics, and a "script editor" that allows the user to interact and modify the scene using code.
>
> The scripting is crucial as it what makes the app what it is.

Let us know what you decide and how it goes.

On Tuesday, 6 October 2015 at 19:42:08 UTC, Jacob wrote:
> There are many programs out there that use some sort of scripting capabilities.
>
> I've been wanting to write an app that exposes a scripting like environment D, lua, or some other fast language to the user.
>
> But there are two requirements:
>
> 1. The user has access to all objects created in app. e.g., if I create a class X in the app then the user can instantiate X in the script without having to recreate it. (the script then, is sort of an extension of the app)
>
> I don't mind actually having to expose X to the script, but it should be easy.
>
> 2. It should be fast. When it is "recompiled" it shouldn't take more than a few seconds. It also shouldn't require a re-initialization of everything.
>
> Can D do this easily?

I can see two ways about this:

1. Use D

Can you include your application's source code with the application?

If so, what I think could work is simply import modules belonging to your app from your plugin's source code, but don't link with the app object files.

The main complication with this approach is that you would need to either include a D compiler with your app, or expect the user to have one installed on their system. Note that DMD is not redistributable (you can't include DMD with your program), but this doesn't apply to LDC/GDC.

This might also get tricky on Windows, as you'll probably need to create or generate a .DEF file (and accompanying import library) which lists all exports/imports shared between your program and plugins. This will include all class methods as mangled names.

D compile times are quite low already, so I don't think this would be an issue.

2. Use a scripting language, e.g. Lua

D's compile-time reflection and code-generation abilities certainly make it possible to expose functions and classes to scripts. You could add a mixin to an exposed class, which would generate all the necessary run-time type information to expose the class to the scripting language. I don't know if any existing library already provides this in an easy-to-use manner, though.

Perhaps a good starting point would be LuaD:
https://github.com/JakobOvrum/LuaD

On Tuesday, 6 October 2015 at 22:05:59 UTC, Laeeth Isharc wrote:
> LuaJIT will be very fast.  I think calling back to C might not be something you want to do in an inner loop though, so structure accordingly.

I meant other way around.  You don't want to call out to Lua there, but I don't think that's your intent anyway.

https://gist.github.com/spion/3049314

it took a lot of work just for C++ to beat LuaJIT on this little benchmark.