So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output. Something like the string imports but instead of opening and reading a text file, it run an executable, waits for it to finish, and grabs its output.

It would get really cool if you could pass this executable some args and then mix in its out put into your own code. It could be used similarly to how CTFE are used but with out the overhead of trying to compile that function and what not and with out the limitations on what it can do.

I could imagine all sorts of things that would be possible with this that is currently not.

Not sure if this is something that could be implemented easily, but seems like something that could be done and something that would be really cool.

On Fri, Jul 12, 2013 at 1:42 PM, Tofu Ninja <emmons0@purdue.edu> wrote:

> So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output. Something like the string imports but instead of opening and reading a text file, it run an executable, waits for it to finish, and grabs its output.
>
> It would get really cool if you could pass this executable some args and then mix in its out put into your own code. It could be used similarly to how CTFE are used but with out the overhead of trying to compile that function and what not and with out the limitations on what it can do.
>
> I could imagine all sorts of things that would be possible with this that is currently not.
>
> Not sure if this is something that could be implemented easily, but seems like something that could be done and something that would be really cool.
>

I would definitely love that, and I don't see why this should be hard.
It should be enabled by a compiler switch (dmd -enable_exec) for obvious
safety reasons though (just as string import requires -J).

This enables or simplifies a number of things:

* ctfeWriteln becomes trivial (this feature has been asked a while ago)
* CTFE in situations where current CTFE fails
* do things that would temprarily require malloc
* easily customize compilation process (user would be able to add logging /
profile info without touching the dmd compiler)
* accessing environment variables during compilation

seems to me all that is needed is optionally redirecting stdin/stdout of a forked process inside dmd compilation, with the exec C function family.

On 7/12/2013 1:42 PM, Tofu Ninja wrote:
> So I had an idea recently, wouldn't it be cool to have the ability to call an
> executable at compile time and capture its output. Something like the string
> imports but instead of opening and reading a text file, it run an executable,
> waits for it to finish, and grabs its output.
>
> It would get really cool if you could pass this executable some args and then
> mix in its out put into your own code. It could be used similarly to how CTFE
> are used but with out the overhead of trying to compile that function and what
> not and with out the limitations on what it can do.
>
> I could imagine all sorts of things that would be possible with this that is
> currently not.
>
> Not sure if this is something that could be implemented easily, but seems like
> something that could be done and something that would be really cool.

This is actually done in the makefile that builds DMD. A program (optabgen) is compiled and then run. Optabgen's output is several .c files which are then compiled into DMD's binary. The programs impcnvgen and idgen do similar things.

It is a very powerful technique.

On Fri, Jul 12, 2013 at 2:54 PM, Walter Bright <newshound2@digitalmars.com>wrote:

> On 7/12/2013 1:42 PM, Tofu Ninja wrote:
>
>> So I had an idea recently, wouldn't it be cool to have the ability to
>> call an
>> executable at compile time and capture its output. Something like the
>> string
>> imports but instead of opening and reading a text file, it run an
>> executable,
>> waits for it to finish, and grabs its output.
>>
>> It would get really cool if you could pass this executable some args and
>> then
>> mix in its out put into your own code. It could be used similarly to how
>> CTFE
>> are used but with out the overhead of trying to compile that function and
>> what
>> not and with out the limitations on what it can do.
>>
>> I could imagine all sorts of things that would be possible with this that
>> is
>> currently not.
>>
>> Not sure if this is something that could be implemented easily, but seems
>> like
>> something that could be done and something that would be really cool.
>>
>
> This is actually done in the makefile that builds DMD. A program (optabgen) is compiled and then run. Optabgen's output is several .c files which are then compiled into DMD's binary. The programs impcnvgen and idgen do similar things.
>
> It is a very powerful technique.
>

I think the OP was refering to something different:
ability to call an arbitrary executable / shell command during compile time
of a D function, whereas optabgen is during compiling dmd itself:

what we want to have is this:

----
import("some_source_file.txt"); //currently this is our only interaction
with outside world during compile time
import std.process; //or std.ctfeprocess ?

string getStringAtCompileTime(string command){
  if(!__ctfe) assert(0);
  //below is currently impossible, but would be great to have
  Tuple!(int,string,string) result=systemCaptureStdinStdout(command);
//likewise without redirecting stdin/stdout
  assert(!result[0]);
  return result[1];
}

void main(){
  enum host = getStringAtCompileTime("env | grep HOST");
}
----

On 7/12/2013 3:12 PM, Timothee Cour wrote:
> I think the OP was refering to something different:
> ability to call an arbitrary executable / shell command during compile time of a
> D function, whereas optabgen is during compiling dmd itself:

It's still the same idea - using external programs to generate source code.


> what we want to have is this:

I do understand that. I'm just saying that this can currently (but awkwardly) be done in the makefile.

On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.

How long until D compilers are able to read mail? :-)

There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.

If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.

On Sat, 13 Jul 2013 00:36:21 +0200, Peter Alexander wrote:

> On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
>> So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> 
> How long until D compilers are able to read mail? :-)
> 
> There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.
> 
> If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.

Other compilers allow symbols/macros to be defined on the command line, e.g. gcc's `-D` flag;  it'd be nice for DMD to define a similar mechanism for defining enums, e.g.

---D code---
  void main() { writeln(VERSION_NUMBER); }
------------

dmd main.d -DVERSION_NUMBER=1.2

Currently I have to use my makefiles to output a whole file which defines the various build constants.

On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
>> So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
>
> How long until D compilers are able to read mail? :-)
>
> There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.
>
> If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.

I think this is in line with what you want though, it allows the things that really should be separate to be separate in another executable, the only thing that it is really bring into the language is that you wont have to rely on a makefile as much, which in my opinion is a good thing.

On Fri, Jul 12, 2013 at 3:36 PM, Peter Alexander < peter.alexander.au@gmail.com> wrote:

> On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
>
>> So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
>>
>
> How long until D compilers are able to read mail? :-)
>
> There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.
>
> If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.
>

regarding added complexity: the only thing this adds is 1 function (calling an executable, with option to redirect stdin/out/err). And yes, that could read mail as you joked if the user called such a program inside his D function, but that would require ZERO change in compiler, apart from that ONE function to support calling external programs.

we need this for the same reason we need CTFE: try using makefiles to achieve what CTFE does. Using a separate build is way less convenient and doesn't allow complex interactions, as it requires the process to be sequential: do other stuff THEN compile with dmd. Whereas integrating it inside compilation would allow interdependent computations offloaded to an external process.

With makefile-like build, you'd have to have potentially many layers of interaction between compiling and calling external programs. Not only would that be much slower (dmd has overhead), but would also require parsing files each time.

On Fri, Jul 12, 2013 at 03:35:30PM -0700, Walter Bright wrote:
> On 7/12/2013 3:12 PM, Timothee Cour wrote:
> >I think the OP was refering to something different: ability to call an arbitrary executable / shell command during compile time of a D function, whereas optabgen is during compiling dmd itself:
> 
> It's still the same idea - using external programs to generate source code.

This idea isn't new. lex/yacc (or their modern incarnations flex/bison) come to mind. The usage is a bit clunky, but the essence is the same.


> >what we want to have is this:
> 
> I do understand that. I'm just saying that this can currently (but awkwardly) be done in the makefile.

At what point does the balance shift from having such an ability built-in, vs. just using OS-level facilities for combining different programs? For example, one could pipe source through a program that performs arbitrary transformations on it, then pipe the result through the compiler. Or one can write a program that generates arbitrary source code and pipe that into the compiler.

Presently, such things are easily handled by a modern build system (of which makefiles are a rather clunky implementation thereof).


T

-- 
This is not a sentence.