Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.

What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.

Thoughts?

-Craig

Craig Black wrote:

> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
> 
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?
> 
> -Craig

I like this idea but it sounds a bit tricky to implement (not to mention possible license issues where the compiler is made a part of the programs it compiles).

Johan Granberg wrote:
> Craig Black wrote:
> 
>> Since Walter is in the habit of adding super advanced features to D, I
>> thought it possible that this one might eventually make it in.  As always,
>> please let me know if I'm way off base.
>>
>> What if we could invoke the compiler from an application at run time and
>> generate a function that would be instantiated on the heap?  The function
>> would be defined by a text string and would be invoked via a function
>> pointer or delegate.  This would allow a script-like capability with
>> fully-optimized performance provided by the D compiler.  I think the D
>> compiler would be ideal for this since it is so fast.  Since the method
>> would be allocated on the heap dynamically, it could also be removed when
>> it is done being used.

I've actually been doing something very similar to this already, but I just write some source code to a file, compile it to a shared library, and load it :P.
Note, In my case, the generated source is C instead of D because http://dsource.org/projects/ddl/ says it's not compatible with current compilers, and the code I generate doesn't use any advanced features anyway. But once DDL is updated it should be quite possible to do this with D code as well.

This does of course require that the user has the compiler installed, but in my case I'm the only one running it, so that's not a problem here :).

This stuff is actually pretty easy to do as long as you're OK with using the compiler as an external program (and keep any source needed for compilation[1] around, obviously).


[1]: i.e. .h files for C/C++, .d/.di for D

> I like this idea but it sounds a bit tricky to implement (not to mention
> possible license issues where the compiler is made a part of the programs
> it compiles).

I think many Common Lisp compilers already do something like this, and license issues don't seem to be a problem. Just consider it an expanded runtime library :P.

Ok, ignore the bootstrap issues for a second, and consider: If the compiler were primarily implemented as a set of functions in the standard library, then we could view "static compilation" as CTFE of those functions...while "dynamic compilation" (a.k.a. runtime compiler) as ordinary, runtime execution.

Russ

Craig Black wrote:
> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
> 
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?
> 
> -Craig

Craig Black wrote:
> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
> 
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?
> 
> -Craig
> 
> 

I like this idea (and have thought about it myself, I haven't brought it up yet, because I'm still unsure what the best approach would be).

Michael Ambrash, was doing some something of similar nature for Pixomatic (a very fast software renderer that looks like DirectX 7). I'm pretty sure his compiler would not map directly to DMD because it needed to be the fastest possible however he does point out some interesting ideas.   The cool thing is, that its setup to optimise for the system it is running on (kinda like java's virtual machine) of which (in Ambrash's case) there are thousands of combinations.

For many case, we could change the code for that users environment write the algorithm in the most otimal way for the given inputs and change dymamic data to constants ect...

System cache can have a dramatic impact on performance which is why code-modifcation techniques have been avoided in the past.  Ambrash found that one frame was enough to allow the newly generated code to propgage into cache.

If you were to encapsulate the code in a function pointer or something I guess, you could have 2 stages.  1) compile the function and 2) run it.  That way in step 1, DMD could push it into cache.

Here are the articles for anyone who is interested.
http://www.ddj.com/184405765
http://www.ddj.com/184405807
http://www.ddj.com/184405848

Anyhow, we do have DDL.  I'm not exactly sure how the compiler/syntax could do a much better job yet (I'm sure its possible).

-Joel

Frits van Bommel wrote:
> Johan Granberg wrote:
> 
>> Craig Black wrote:
>>
>>>
>>> What if we could invoke the compiler from an application at run time and
>>> generate a function that would be instantiated on the heap?  The function
>>> would be defined by a text string and would be invoked via a function
>>> pointer or delegate.
> 
> 
> I've actually been doing something very similar to this already, but I just write some source code to a file, compile it to a shared library, and load it :P.

Darn you all! I've been thinking of both of these ideas for some time. <G> I like both ideas by the way.

However I wouldn't use text, I'd go with generated AST or something like that, maybe a mesh of basic blocks. I don't known exactly what, but it seems that the nature of memory being lots different than a text file would allow for some more fun (powerful) ways of doing things.

janderson wrote:
> Craig Black wrote:
>> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
>>
>> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
>>
>> Thoughts?
>>
>> -Craig
>>
>>
> 
> I like this idea (and have thought about it myself, I haven't brought it up yet, because I'm still unsure what the best approach would be).
> 
> Michael Ambrash, was doing some something of similar nature for Pixomatic (a very fast software renderer that looks like DirectX 7). I'm pretty sure his compiler would not map directly to DMD because it needed to be the fastest possible however he does point out some interesting ideas.   The cool thing is, that its setup to optimise for the system it is running on (kinda like java's virtual machine) of which (in Ambrash's case) there are thousands of combinations.
> 
> For many case, we could change the code for that users environment write the algorithm in the most otimal way for the given inputs and change dymamic data to constants ect...
> 
> System cache can have a dramatic impact on performance which is why code-modifcation techniques have been avoided in the past.  Ambrash found that one frame was enough to allow the newly generated code to propgage into cache.
> 
> If you were to encapsulate the code in a function pointer or something I guess, you could have 2 stages.  1) compile the function and 2) run it.  That way in step 1, DMD could push it into cache.
> 
> Here are the articles for anyone who is interested.
> http://www.ddj.com/184405765
> http://www.ddj.com/184405807
> http://www.ddj.com/184405848
> 
> Anyhow, we do have DDL.  I'm not exactly sure how the compiler/syntax could do a much better job yet (I'm sure its possible).
> 
> -Joel

Humm, why do I always spell Abrash's name wrong  :(

Craig Black wrote:
> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
> 
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?

Plenty.  I've tried my best to approach this kind of functionality for some time in library space.  However I have my doubts as to having this included in D proper - since D is so minimal at it's core, it becomes very much a library support problem.

(I'll leave the DSP/DDL/Flectioned discussion for another thread).

I can say from experience that doing this from within library space is perfectly feasable, and yields some great results, at the cost of additional memory at runtime and a small (one-time per module) delay for compilation and runtime binding.  I would have sooner considered drafting a runtime compiler if DMD wasn't so fast.

-- 
- EricAnderton at yahoo

Craig Black schrieb:
> Since Walter is in the habit of adding super advanced features to D, I thought it possible that this one might eventually make it in.  As always, please let me know if I'm way off base.
> 
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?
> 
> -Craig
> 
> 

I remember the softwire project which does such things. It is currently hosted at

https://gna.org/projects/softwire

I also thought that the Win32 port of TinyCC could compile directly to the heap.

KlausO

Craig Black wrote:
> What if we could invoke the compiler from an application at run time and generate a function that would be instantiated on the heap?  The function would be defined by a text string and would be invoked via a function pointer or delegate.  This would allow a script-like capability with fully-optimized performance provided by the D compiler.  I think the D compiler would be ideal for this since it is so fast.  Since the method would be allocated on the heap dynamically, it could also be removed when it is done being used.
> 
> Thoughts?


Yep. You will like Forth

Andreas