Okay, the article and the library have both been updated, with the little tweaks suggested (here versus hear, doc code online) and support for Linux (running Windows' DMD and SCPPN in Linux, that is); that proved a little bit tricky so there's a section in the article on it now. Note that that's specifically Linux - FreeBSD and BeOS would require a different syscall, but I'm already burned out from Linux's poor documentation so I'm ignoring that for now, if not forever.

Oh, and: I don't know why SCPPN (DigitalMars C) is twice as fast as DMD in Windows and many times slower in Linux. It asks for some really weird information (like GetTimeZoneInformation) that I just brush off rather than implement, so maybe I'm sending it on some loopy journey that costs millions of cycles. I might investigate that later.

Dejan Lekic Wrote:

> Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I would (seriously) like to see this in Phobos one day (hopefully Phobos developers are reading this thread). It deserves to go into Phobos, definitely.

Sure. What I'd prefer actually would be language integration, so that it can sort out the details of integrating runtime types into the rest of the type hierarchy, and it would be able to statically test the template. That can be done automatically (So some template parameter can't be const-folded? Okay, we'll compile it when we know what the value is!) or with subtle hints like a dynamic attribute (that, and here's the very important part, act like regular template parameters if they ARE const). But that seems unlikely as it would put a comparatively massive burden on the runtime* and make porting to new environments quite a bit more difficult. I haven't used GDC, but if it's as terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it suitable for dynamic compilation.

* In which case, hey, it's optional for restrictive environments!

Burton Radons wrote:
> Oh, and: I don't know why SCPPN (DigitalMars C) is twice as fast as
> DMD in Windows and many times slower in Linux. It asks for some
> really weird information (like GetTimeZoneInformation) that I just
> brush off rather than implement, so maybe I'm sending it on some
> loopy journey that costs millions of cycles. I might investigate that
> later.

The compiler calls ctime() in order to set the macros __DATE__, __TIME__, and __TIMESTAMP__.

Updated the library. It now compiles in D 1.0, D 2.0, and Tango without changes. Tango's a pain for me to test so it might drift and break before the article's finished.

Burton Radons wrote:
> Dejan Lekic Wrote:
> 
>> Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I would (seriously) like to see this in Phobos one day (hopefully Phobos developers are reading this thread). It deserves to go into Phobos, definitely.
> 
> Sure. What I'd prefer actually would be language integration, so that it can sort out the details of integrating runtime types into the rest of the type hierarchy, and it would be able to statically test the template. That can be done automatically (So some template parameter can't be const-folded? Okay, we'll compile it when we know what the value is!) or with subtle hints like a dynamic attribute (that, and here's the very important part, act like regular template parameters if they ARE const). But that seems unlikely as it would put a comparatively massive burden on the runtime* and make porting to new environments quite a bit more difficult. I haven't used GDC, but if it's as terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it suitable for dynamic compilation.
> 
> * In which case, hey, it's optional for restrictive environments!

I wouldn't want it transparently doing runtime code generation. I think the library solution, as ugly as it is, is ideal since it lets people know "yeah, this is a weighty operation".

(Web-News decided the best place to put a reply to a post in digitalmars.D.announce that was marked as being for digitalmars.D.announce was actually digitalmars.D. I've noticed it doesn't like it when I take more than eight hours or so to write a reply; I guess it's using cookies in silly ways.)

Robert Fraser Wrote:

> Burton Radons wrote:
> > Dejan Lekic Wrote:
> > 
> >> Mr. Radons, this is *AMAZING* ! Thank you for sharing it with us! I would (seriously) like to see this in Phobos one day (hopefully Phobos developers are reading this thread). It deserves to go into Phobos, definitely.
> > 
> > Sure. What I'd prefer actually would be language integration, so that it can sort out the details of integrating runtime types into the rest of the type hierarchy, and it would be able to statically test the template. That can be done automatically (So some template parameter can't be const-folded? Okay, we'll compile it when we know what the value is!) or with subtle hints like a dynamic attribute (that, and here's the very important part, act like regular template parameters if they ARE const). But that seems unlikely as it would put a comparatively massive burden on the runtime* and make porting to new environments quite a bit more difficult. I haven't used GDC, but if it's as terrifyingly slow as GCC (at no fault of GDC I'll add) I wouldn't consider it suitable for dynamic compilation.
> > 
> > * In which case, hey, it's optional for restrictive environments!
> 
> I wouldn't want it transparently doing runtime code generation. I think the library solution, as ugly as it is, is ideal since it lets people know "yeah, this is a weighty operation".

D has absolutely no presence in limited-memory environments and I don't see that ever happening, not just because of its design but because those environments are dying out (micro-limited-memory environments are here to say). Maybe someone with a 2 GHz processor and a gigabyte of RAM wants to control every single byte, but their concern is invalid. Where they should be enjoying their riches they're sitting there biting their nails over nothing like a rich man obsessed with thieves, while Java and C# gladly take a few dozen megabytes for simple programs. Even if I did think it worth courting them it would be pointless; D has garbage collection and that's the end of the story.

Burton Radons wrote:
> Bjoern Wrote:
> 
>> I will read your article a third time, but I think it means : No need for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast scripting on-the-fly-compiled language. (using eval)
>>
>> Sorry I must be wrong!
> 
> It can take a big chunk out of any situation where you need to use a switch to select between templates, but it'll have little effect on anything which uses templating and CTFE for expressiveness. Or maybe it will - it's a diverse concept that I haven't fully explored.

The back-end of my BLADE code uses CTFE to generates asm for floating point array operations at compile-time; but the same CTFE can be used without modification for code generation at run-time. The advantage at run-time (or at least, at install time) is that you know the CPU, number of cores, and cache size.

I think that in your article you should include a brief comparison with the JIT techniques of Java/.NET. To a naive reading, it sounds similar; but they are fundamentally different. .NET's last-minute conversion of intermediate form to native code doesn't buy you very much. JIT source code generation is something else entirely. Using your library, you can have JIT selection of the _algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of thing.

Don Clugston Wrote:

> Burton Radons wrote:
> > Bjoern Wrote:
> > 
> >> I will read your article a third time, but I think it means : No need for CTFE anymore, DDL is dead. Furthermore we can use D as ultra fast scripting on-the-fly-compiled language. (using eval)
> >>
> >> Sorry I must be wrong!
> > 
> > It can take a big chunk out of any situation where you need to use a switch to select between templates, but it'll have little effect on anything which uses templating and CTFE for expressiveness. Or maybe it will - it's a diverse concept that I haven't fully explored.
> 
> The back-end of my BLADE code uses CTFE to generates asm for floating point array operations at compile-time; but the same CTFE can be used without modification for code generation at run-time. The advantage at run-time (or at least, at install time) is that you know the CPU, number of cores, and cache size.

This can also work at a slightly higher level, where your object file has multiple versions of a function for different feature sets and the custom linker chooses the ideal one at runtime. Another use would be to make the inliner work better; I've noticed that sometimes it's better to avoid asm statements because they prevent the optimiser from inlining the function and applying further optimisations to it. So you could compile the code with a regular D function, then link whatever doesn't inline the regular function to a custom-written asm function, improving execution speed in both scenarios.

However that won't be helpful if the compiler's only reasoning about inlining is the number of statements in the function; I haven't really tested this.

> I think that in your article you should include a brief comparison with the JIT techniques of Java/.NET. To a naive reading, it sounds similar; but they are fundamentally different. .NET's last-minute conversion of intermediate form to native code doesn't buy you very much. JIT source code generation is something else entirely. Using your library, you can have JIT selection of the _algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of thing.

I don't know much about Sun's Java or Microsoft's .NET JITs; I'll read about them when I have the time.

From my initial perusal, dynamic recompilation is certainly something you could do, where you profile code then generate a more optimal version of the function based upon how it's being used or the environment. However, I'd like a more solid case for where that could be useful in D before putting it in the article.

Burton Radons wrote:

>> I think that in your article you should include a brief comparison with the JIT techniques of Java/.NET. To a naive reading, it sounds similar; but they are fundamentally different. .NET's last-minute conversion of intermediate form to native code doesn't buy you very much. JIT source code generation is something else entirely. Using your library, you can have JIT selection of the _algorithm_. FFTW (www.fftw.org) is an interesting example of this sort of thing.
> 
> I don't know much about Sun's Java or Microsoft's .NET JITs; I'll read about them when I have the time.
> 
> From my initial perusal, dynamic recompilation is certainly something you could do, where you profile code then generate a more optimal version of the function based upon how it's being used or the environment. However, I'd like a more solid case for where that could be useful in D before putting it in the article.


Just out of curiousity, isn't such a dynamic recompilation technique used is some emulator systems (CPU instruction translators like the old 680x0 emulators for PowerPC systems).  I believe some of these ideas were used to compile, profile, and cache runs of instructions (saving the optimal results)... or something to that effect.

-JJR