Laeeth Isharc 
Posted in reply to PiSquared
| On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
> D has the ability to embed most languages in to it through scripting. There is matplotlibD already, I have modified the code to be able to run arbitrary scripts and also work with Lua. Any language can be used through the method of:
>
> 1. Writing a D string representing the target language's program to a file or stdin of the interpreter.
>
> 2. Before executing above, first modify the input to replace special tokens with D data. This allows one to insert D data in to the target program. This allows one to use D to generate the data which may be much faster and can be used in other ways.
>
> [See below for an example]
>
> These methods should work with all languages since it simply emulates coding manually. It becomes a powerful feature because one and leverage the target language as if one was writing directly in the languages.
>
> More importantly, Visual D and Visual Studio allow one to load many target languages in to a D project and get almost full benefit such as intelligence(debugging does not work but I imagine it could with some work).
>
>
> Furthermore, with some work, interopt can be made to work really well by transferring data between the programs using a client server protocol which can be done through files or global memory(depending on the languages level of OS access).
>
>
> With the proper design 1. D can be used along side and as a host for most languages. The most common will work fine. With a bit of work it should be work extremely well. This allows D to be leverage and piggy back on these languages and will bring users of those languages to D. D can be used for what it is good at and the holes it has can be filled by some other language. 2. The design is not specific to D as most languages have this ability, but with D it generally can be quite easy.
>
> The only downside is that of "marshaling" the data, this can be reduced significantly by not using text to transfer data but by having appropriate data marshalers that can work in memory and relate data between D and the target language correctly.
>
>
> Further more, the target language needs some capacity too send data back to the D program.
>
> I have created a server and a client that do this. It is used for writing latex code in D. This allows me to also use python's scientific plotting libraries along with tikz by using a common D library. Tikz is slow, D+mlab is fast.
>
> The latex code, for example, can send data back to the server using a client proxy and this can trigger different results when the latex file is being compiled by latex.
>
> It is all quite simple but it allows me to debug D programs in D without issue as all the plumbing on the target language is relatively hidden.
>
>
> I would like you guys to consider taking this proposal seriously: Create a proper D design that allows one to leverage any target language in a unified way. Make it robust, powerful, feature rich.
>
> It will bring many people to use D if done properly. With a proper IDE D could act as a centerpiece to use many languages together.
>
> One could use a gui in lua, a plotting library in python, and latex dox generation.
>
> All done rather seamless.
>
>
>
> here is a lua target, bare bones with no fancy features and essentially ripped from matplotlibD(which essentially does the same with python, of which I have a version that is essentially the same as below):
>
>
> module LuaD;
> import std.stdio;
>
> enum DDataToken = `"@D$`;
>
> alias immutable bool LuaBool;
> alias immutable (void*) LuaNone;
>
> LuaBool False = false;
> LuaBool True = true;
> LuaNone None = null;
>
> // A header to use for common lua files
> string luaHeader = `
> `;
>
> string luaFooter = ``;
>
>
> string d2lua(T)(T v)
> {
>
> int x = 3242;
> import std.conv, std.traits, std.array, std.algorithm, std.format: format;
> static if (is(T : LuaNone))
> return "None";
>
> else static if (is(T : bool))
> return v ? "True" : "False";
>
> else static if (is(T : string))
> return format("\"%s\"", v);
>
> else
> {
> static if (isArray!T)
> return v.map!(a=>to!string(a)~",").join;
>
> return format("%s", v);
> }
> }
>
>
>
> string lua_path = `C:\Lua\lua.exe`;
>
> // executes lua on the program
> void lua(string program)
> {
>
> import std.process: environment, pipeProcess, wait, Redirect;
>
> if (!lua_path)
> lua_path = environment.get("LuaD", "lua");
>
> auto pipes = pipeProcess(lua_path, Redirect.stdin | Redirect.stderr);
>
> pipes.stdin.writeln(program);
> pipes.stdin.writeln("exit()");
> pipes.stdin.close();
>
> auto result = wait(pipes.pid);
>
> if (result != 0) {
> string error;
> foreach (line; pipes.stderr.byLine)
> error ~= line ~ "\n";
> throw new Exception("\n\nERROR occurred in Lua:\n" ~ error);
> }
> }
>
>
>
> // converts all "@$..." strings to D variables by converting
> auto luaD(bool ret = false)(string program)
> {
>
> if (program.length < 5) return "";
>
> import std.string, std.range, std.algorithm;
>
>
> string r;
> for(int i = 0; i < program.length; i++)
> {
> if (i < program.length - 5)
> {
> if (program[i..i+DDataToken.length] == DDataToken)
> {
> i += DDataToken.length;
> string id;
> while(i < program.length && program[i] != '"')
> id ~= program[i++];
>
> r ~= `"~d2lua(`~id~`)~"`;
> continue;
> }
> }
> if (program[i] == '"') r ~= "\\";
> r ~= program[i];
> }
>
> static if (ret) { return `"`~r~`"`; }
>
> return `lua("`~r~`");`;
> }
>
>
>
> Then a simple lua program `test.lua`:
>
> print("@D$TestString")
>
>
>
> one calls this program in D:
>
> auto TestString = "This is a D string";
>
> // Will automatically substitute TestString
> mixin(luaD(import(test.lua));
>
>
>
>
>
> and the lua program will be executed with the D data.
>
>
> I have used such things for computing data in D and then plotting them using some of pythons plotting libraries(matplotlib, mlab, mayavi, etc).
>
>
> It is not as optimal since large datasets can cause problems since they can take up so much textual space but this can be alleviated by writing them to binary files, or ideally, using memory transfers.
>
>
> With a good robust design D can be the centerpiece of language hosting bringing a very large number of users. Python, for some odd reason, has a lot of scientific plotting libraries but, as most people know, is not very fast. Imagine using D for the algorithms and python for the plotting. That is precisely what I have done and it works well(debugging errors is the hardest since no IDE but that could change). In visual D I get full syntax coloring, intellisense, and documentation for python and lua.
>
> Imagine F#, C#, vbs, C++, Haskell, even assembly and many other languages being usable with this method. With a good "marshaling" library it one would could have limited performance degradation. For apps that do not have performance issues, say typical gui's one can use a language with good gui frameworks and D for the business end. With D's meta capabilities one could probably export most of the bridging automatically in to the gui.
>
>
> Done well it could be a boon to D and bring in many new people and $$$. As far as I'm aware, no language or framework exists to do this well and easily yet it is not a difficult problem.
Yes, and for example in C# and F# I think Roslyn could take you a long way since you can compile dotnet code at D runtime.
As is well known, C++ is an interpreted language where you can instantiate templates at runtime.
I'm joking, but it can be if you want it to be and CERN do this at considerable scale with Cling, which allows you to instantiate templates at runtime and introspect, create instance of classes and call them. Some interesting early results in calling cling from D, but I was waiting till things were a bit more polished to talk about them. The next stage might be to run cling on clang to help with dpp development. Libclang looks amazing at first sight but it is missing a lot.
I think there are two reasons why interoperability matters. One is external libraries but another is that when you are exploring using D then it's often a bit challenging if you have C++ code and need to use it or implement part of a new feature in D. If replacing working code you don't ideally want to do a complete rewrite but replace small pieces at a time and have working code all along the way. That might not be easy in many cases.
I think RoI is often a less important barrier than limits to upfront pain. If you use D you are already an unusual sort of person because the discomfort is front loaded and how do you know it will be worth it to persevere? Atila downloaded the compiler and back then it segfaulted when he tried to compile hello world! Most reasonable people would have given up then.
But the dollar return on being reasonable is time varying and for some people at some times it's not necessarily a bad thing to hire people who are a bit unreasonable. Perhaps all progress depends on people not being reasonable.
So if it becomes easier to use libraries and internal code written in other languages then it's easy to make small beginnings. Lots of large projects begin as little beginnings and so making it easy to start work has value for the ecosystem I think.
I agree that sometimes you don't need to do things that are that clever to be quite productive. But one probably can do a bit better than what you suggest even if for any individual it's more than good enough.
It's almost easy to use python, C, Lua,R libraries from D but not quite. Pyd is showing its age right now and LuaD isn't that well maintained. It's not super easy to find embedr and it's not trivially easy to get up and running.
I think lots of that is documentation, maintenance,small improvements and blog posts.
Dpp is pretty solid a lot of the time. On Linux anyway but on Windows not yet, judging by a conversation I had yesterday. I asked Adam Ruppe to look into that and I think in time it will just work for what it works for. It's a shame it doesn't dependably yet at least handle Linux headers without choking. I think that's an SAoC project if I recall. We could do a bit more to programmatically just try wrapping every header we can get our hands on and then try to programmatically analyse the results.
Ultimately there is a limit with macros, maybe with using libclang and maybe how far you can get with generating strings in D rather than AST to AST transformation. The latter for C isn't a trivial project - DoD spent millions on C2Rust though on the other hand they are not going to be able to do things on the cheap. C to D ought to be quite doable though. Our focus has been more on the cpp side for now but in time could do a bit more with C.
With PyD under autowrap there is a project called PyNIH. Steady progress with focus first on Python calling D but that helps with the reverse.
Embedding C# in D has stalled a bit - see dflat and a project to do so via Mono also. It's not difficult work technically but just a bit of a grind.
We did some very basic work on embedding Julia in D. That also isn't difficult but time is limited.
The basic idea is to use Symmetry Integration Language as a substrate where you can write plugins in any sensible language and call it from any sensible language for us. We have to solve our own problems first but in the process of doing so probably it will help others.
One more thing - work Sebastian Koppe did shows a proof of concept of D compiling to web assembly and using Javascript libraries. He will generate the wrapping code from Typescript bindings in time.
From work I did before I don't think embedding Java would be so difficult but we just don't use Java that much.
So I think the biggest constraint right now is the ability to organise and direct the work in some cases (like C#) and in others for other people to submit improvements. But in time I think interoperability will just keep getting better.
|
Permalink
Reply
