On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> >
> >How long until D compilers are able to read mail? :-)

Zawinski's law of software envelopment:

	Every program attempts to expand until it can read mail. Those
	programs which cannot so expand are replaced by ones which can.

:-)


> >There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.
> >
> >If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.
> 
> I think this is in line with what you want though, it allows the things that really should be separate to be separate in another executable, the only thing that it is really bring into the language is that you wont have to rely on a makefile as much, which in my opinion is a good thing.

Makefiles are really a poor example of build systems. A proper build system can do such things in a much better, consistent way.

Makefiles, for example, can't correctly recompile your program if you change compiler flags. And complex interdependencies, like you said, require painful manual maintenance, otherwise they tend to screw up and produce inconsistent results.

For an example of a modern build system, see http://gittup.org/tup/.


T

-- 
Ph.D. = Permanent head Damage

On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:

> On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> > On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> > >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> > >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> > >
> > >How long until D compilers are able to read mail? :-)
>
> Zawinski's law of software envelopment:
>
>         Every program attempts to expand until it can read mail. Those
>         programs which cannot so expand are replaced by ones which can.


forget about reading mail, we currently don't even have hello world! (we
have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)



> :-)
>
>
> > >There's many obvious applications of this proposed feature, but I say such things should be delegated to the build process. Just run those executables using a makefile, or whatever build system you use.
> > >
> > >If you need compile time inputs from your D code then just run a separate build to extract them. Yes it's more work and less convenient, but I think we need to be careful with how much is added to the D compilers. We don't want to turn them into operating systems.
> >
> > I think this is in line with what you want though, it allows the things that really should be separate to be separate in another executable, the only thing that it is really bring into the language is that you wont have to rely on a makefile as much, which in my opinion is a good thing.
>
> Makefiles are really a poor example of build systems. A proper build system can do such things in a much better, consistent way.
>
> Makefiles, for example, can't correctly recompile your program if you change compiler flags. And complex interdependencies, like you said, require painful manual maintenance, otherwise they tend to screw up and produce inconsistent results.
>
> For an example of a modern build system, see http://gittup.org/tup/.
>
>
Using a build system would likely be non-portable (see current mess even in dmd source code, which requires posix.ma / win32.mak / win64.mak which is not DRY and hard to maintain), and suffer from the points I mentioned in my previous post.

We don't need yet another build system if we have "exec" available at CT.

Let's put it another way: if I or someone else made a pull request for CTFE "exec", would it have a chance of being accepted?

On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:
> On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:
> 
> > On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> > > On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> > > >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> > > >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> > > >
> > > >How long until D compilers are able to read mail? :-)
> >
> > Zawinski's law of software envelopment:
> >
> >         Every program attempts to expand until it can read mail. Those
> >         programs which cannot so expand are replaced by ones which can.
> 
> 
> forget about reading mail, we currently don't even have hello world! (we
> have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)

Why should the compiler turn into a hello world program?


[...]
> > Makefiles, for example, can't correctly recompile your program if you change compiler flags. And complex interdependencies, like you said, require painful manual maintenance, otherwise they tend to screw up and produce inconsistent results.
> >
> > For an example of a modern build system, see http://gittup.org/tup/.
> >
> >
> Using a build system would likely be non-portable (see current mess even in dmd source code, which requires posix.ma / win32.mak / win64.mak which is not DRY and hard to maintain), and suffer from the points I mentioned in my previous post.

Again, makefiles are a poor example of build systems. There *are* build systems that allow you to write portable build scripts, such as SCons. They're not perfect, but makefiles are a far cry from what they can do.


> We don't need yet another build system if we have "exec" available at CT.

That doesn't replace a proper build system. What if your external program itself is a D program that needs to be compiled first? What if you need to run an external program like povray in order to generate images (say, animation frames) that are string-imported into your program? Should DMD have the capability of running arbitrary programs that produce arbitrary output and be able to sort out which programs must run first and which output should be imported into the D program?

So we're no longer talking about the D compiler, but about the D operating system. The real "DOS". :-P Which, mind you, would be a good thing IMO, but I think it's a bit of a stretch from where we are currently.


> Let's put it another way: if I or someone else made a pull request for CTFE "exec", would it have a chance of being accepted?

I'm not one to decide, you have to convince Walter.

And I'm not saying this idea is good or bad, I'm just saying that you can already do such things without compiler support. At this point, I'd rather we focus on stabilizing the language and improving its implementation, than to continue adding new features that reinvent the wheel.


T

-- 
For every argument for something, there is always an equal and opposite argument against it. Debates don't give answers, only wounded or inflated egos.

On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:

> On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:
> > On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh@quickfur.ath.cx>
> wrote:
> >
> > > On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> > > > On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> > > > >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> > > > >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> > > > >
> > > > >How long until D compilers are able to read mail? :-)
> > >
> > > Zawinski's law of software envelopment:
> > >
> > >         Every program attempts to expand until it can read mail. Those
> > >         programs which cannot so expand are replaced by ones which can.
> >
> >
> > forget about reading mail, we currently don't even have hello world! (we
> > have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)
>
> Why should the compiler turn into a hello world program?
>


For logging / debugging purposes, we often need to be able to print at compile time some variables, but ctfeWriteln doesn't even work. And again, pragma(msg) can't replace it.



> [...]
> > > Makefiles, for example, can't correctly recompile your program if you change compiler flags. And complex interdependencies, like you said, require painful manual maintenance, otherwise they tend to screw up and produce inconsistent results.
> > >
> > > For an example of a modern build system, see http://gittup.org/tup/.
> > >
> > >
> > Using a build system would likely be non-portable (see current mess even in dmd source code, which requires posix.ma / win32.mak / win64.mak which is not DRY and hard to maintain), and suffer from the points I mentioned in my previous post.
>
> Again, makefiles are a poor example of build systems. There *are* build systems that allow you to write portable build scripts, such as SCons. They're not perfect, but makefiles are a far cry from what they can do.
>
>
> > We don't need yet another build system if we have "exec" available at CT.
>
> That doesn't replace a proper build system.


I wrote a D based makefile-like tool in D (which uses direct acyclic graph to find compilation order). It's much cleaner in D for obvious reasons.


> What if your external program itself is a D program that needs to be compiled first?


call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)


> What if you need to run an external program like povray in order to
> generate
> images (say, animation frames) that are string-imported into your
> program?


I don't see how that's different:
void main(){
enum s=exec("program generating images");
import(s);
//use the string import
//or even simply: use s directly, depending on conditions
}



> Should DMD have the capability of running arbitrary programs
> that produce arbitrary output and be able to sort out which programs
> must run first and which output should be imported into the D program?
>

The order would be simply program order, as in the example above: "enum
s=exec(...)" is run during CTFE before import(s), as it comes first.


> So we're no longer talking about the D compiler, but about the D operating system. The real "DOS". :-P Which, mind you, would be a good thing IMO, but I think it's a bit of a stretch from where we are currently.
>

No, all it requires in terms of dmd modification is CTFE exec. The user can use it to do complex stuff if he so wishes.

CTFE already runs very complex program (involving std.algorithm and whatnot), so there's already a justification for running complex programs during CTFE. Allowing that one simple function will enable many more things.


> > Let's put it another way: if I or someone else made a pull request for CTFE "exec", would it have a chance of being accepted?
>
> I'm not one to decide, you have to convince Walter.
>
> And I'm not saying this idea is good or bad, I'm just saying that you can already do such things without compiler support.


we can't even print variables during CTFE. Using a build system to do that is way overkill.

On Fri, Jul 12, 2013 at 05:39:45PM -0700, Timothee Cour wrote:
> On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:
> 
> > On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:
> > > On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh@quickfur.ath.cx>
> > wrote:
> > >
> > > > On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> > > > > On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> > > > > >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> > > > > >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> > > > > >
> > > > > >How long until D compilers are able to read mail? :-)
> > > >
> > > > Zawinski's law of software envelopment:
> > > >
> > > >         Every program attempts to expand until it can read mail.
> > > >         Those programs which cannot so expand are replaced by
> > > >         ones which can.
> > >
> > >
> > > forget about reading mail, we currently don't even have hello world! (we
> > > have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)
> >
> > Why should the compiler turn into a hello world program?
> >
> 
> 
> For logging / debugging purposes, we often need to be able to print at compile time some variables, but ctfeWriteln doesn't even work. And again, pragma(msg) can't replace it.

This is a bug / limitation in CTFE, not a justification for invoking arbitrary programs from the compiler.


[...]
> > > We don't need yet another build system if we have "exec" available at CT.
> >
> > That doesn't replace a proper build system.
> 
> 
> I wrote a D based makefile-like tool in D (which uses direct acyclic graph to find compilation order). It's much cleaner in D for obvious reasons.

I'm interested. Is the code on github?


> > What if your external program itself is a D program that needs to be compiled first?
> 
> 
> call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)

So basically you're saying DMD should be expanded into a D shell?


[...]
> > So we're no longer talking about the D compiler, but about the D operating system. The real "DOS". :-P Which, mind you, would be a good thing IMO, but I think it's a bit of a stretch from where we are currently.
> >
> 
> No, all it requires in terms of dmd modification is CTFE exec. The user can use it to do complex stuff if he so wishes.
> 
> CTFE already runs very complex program (involving std.algorithm and whatnot), so there's already a justification for running complex programs during CTFE. Allowing that one simple function will enable many more things.

In the old days, somebody wrote a 1-line universal C program that could do literally *anything*. All you have to do is to specify, in C syntax, what you want the program to do. The program's source code is:

	#include "/dev/tty"

:-)  (Sadly, this no longer works on modern systems.)

Seriously, though, I think adding an exec function is total overkill for working around CTFE limitations. If that's what we're trying to solve, we should be looking at how to improve CTFE, not adding a nuclear warhead to dmd.


> > > Let's put it another way: if I or someone else made a pull request for CTFE "exec", would it have a chance of being accepted?
> >
> > I'm not one to decide, you have to convince Walter.
> >
> > And I'm not saying this idea is good or bad, I'm just saying that you can already do such things without compiler support.
> 
> 
> we can't even print variables during CTFE. Using a build system to do that is way overkill.

If that's the root problem, then we should fix/improve CTFE. I'm not convinced that this should be grounds for adding an exec function to dmd. It seems to be complete overkill just for working around CTFE limitations.

But maybe Walter has a different opinion about this. :)


T

-- 
Let's call it an accidental feature. -- Larry Wall

On Fri, Jul 12, 2013 at 6:14 PM, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:

> On Fri, Jul 12, 2013 at 05:39:45PM -0700, Timothee Cour wrote:
> > On Fri, Jul 12, 2013 at 5:10 PM, H. S. Teoh <hsteoh@quickfur.ath.cx>
> wrote:
> >
> > > On Fri, Jul 12, 2013 at 05:00:29PM -0700, Timothee Cour wrote:
> > > > On Fri, Jul 12, 2013 at 4:51 PM, H. S. Teoh <hsteoh@quickfur.ath.cx>
> > > wrote:
> > > >
> > > > > On Sat, Jul 13, 2013 at 12:51:03AM +0200, Tofu Ninja wrote:
> > > > > > On Friday, 12 July 2013 at 22:36:22 UTC, Peter Alexander wrote:
> > > > > > >On Friday, 12 July 2013 at 20:42:50 UTC, Tofu Ninja wrote:
> > > > > > >>So I had an idea recently, wouldn't it be cool to have the ability to call an executable at compile time and capture its output.
> > > > > > >
> > > > > > >How long until D compilers are able to read mail? :-)
> > > > >
> > > > > Zawinski's law of software envelopment:
> > > > >
> > > > >         Every program attempts to expand until it can read mail.
> > > > >         Those programs which cannot so expand are replaced by
> > > > >         ones which can.
> > > >
> > > >
> > > > forget about reading mail, we currently don't even have hello world!
> (we
> > > > have pragma(msg) but no ctfewriteln, which pragma(msg) can't do)
> > >
> > > Why should the compiler turn into a hello world program?
> > >
> >
> >
> > For logging / debugging purposes, we often need to be able to print at compile time some variables, but ctfeWriteln doesn't even work. And again, pragma(msg) can't replace it.
>
> This is a bug / limitation in CTFE, not a justification for invoking arbitrary programs from the compiler.
>

there would be so many other things to fix (what if I want to access environment variables? or know whether a program / library is available?). Whereas simply allowing CTFE exec unleashes arbitrarily complex use cases. The choice would be up to the user to make use of it or not.


[...]
> > > > We don't need yet another build system if we have "exec" available at CT.
> > >
> > > That doesn't replace a proper build system.
> >
> >
> > I wrote a D based makefile-like tool in D (which uses direct acyclic graph to find compilation order). It's much cleaner in D for obvious reasons.
>
> I'm interested. Is the code on github?


Not yet, code isn't very presentable at the moment but API is easy to use; user can mix both pure D and a makefile-like DSL syntax, so it makes simple things simple and complex things possible. It's DRY (nothing has to be repeated twice, which makes maintenance easy). I can push it to github if there's interest, but my point is that doing such things in D is easier than one might think.

> > What if your external program itself is a D program that needs to be
> > > compiled first?
> >
> >
> > call exec("dmd mydeps.d"); (or exec("make -all") or the D based makefile)
>
> So basically you're saying DMD should be expanded into a D shell?
>

No I never intended that. the call to exec("dmd...") would be a regular statement in a regular D file to be compiled. All I'm suggesting is making at least some of the functions in std.process CTFE-able.


> [...]
> > > So we're no longer talking about the D compiler, but about the D operating system. The real "DOS". :-P Which, mind you, would be a good thing IMO, but I think it's a bit of a stretch from where we are currently.
> > >
> >
> > No, all it requires in terms of dmd modification is CTFE exec. The user can use it to do complex stuff if he so wishes.
> >
> > CTFE already runs very complex program (involving std.algorithm and whatnot), so there's already a justification for running complex programs during CTFE. Allowing that one simple function will enable many more things.
>
> In the old days, somebody wrote a 1-line universal C program that could do literally *anything*. All you have to do is to specify, in C syntax, what you want the program to do. The program's source code is:
>
>         #include "/dev/tty"
>
> :-)  (Sadly, this no longer works on modern systems.)
>
> Seriously, though, I think adding an exec function is total overkill for working around CTFE limitations. If that's what we're trying to solve, we should be looking at how to improve CTFE, not adding a nuclear warhead to dmd.
>

I disagree. From the compiler implementation perspective, it's very easy (only requires making some of std.process CTFE-able), yet unleashes lots of potential use cases. Versus what you're suggesting which is to provide implementation for each individual CTFE limitation one at a time.

Being lazy is good :-)

A similar argument could be made for runtime: if we didn't have std.process, we'd have to reimplement or wrap "in D" all missing features. But thanks to std.process I can easily add many missing D functions, eg for readLink:
----
string realPath(string file){
import std.process;
return runPython(`import os;
a=os.path.realpath(`~file.escapeShellFileName~`); print(a);`);
}
string runPython(string code){
import std.process;
string command=`python -c `~code.escapeShellFileName;
return command.executeShell.outputThrows;
}
----

bam, that was easy.

I see no point in arbitrarily impose restrictions on future use cases.



> > > > Let's put it another way: if I or someone else made a pull request for CTFE "exec", would it have a chance of being accepted?
> > >
> > > I'm not one to decide, you have to convince Walter.
> > >
> > > And I'm not saying this idea is good or bad, I'm just saying that you can already do such things without compiler support.
> >
> >
> > we can't even print variables during CTFE. Using a build system to do that is way overkill.
>
> If that's the root problem, then we should fix/improve CTFE. I'm not convinced that this should be grounds for adding an exec function to dmd. It seems to be complete overkill just for working around CTFE limitations.
>
> But maybe Walter has a different opinion about this. :)
>

ping, Walter ?

On 7/12/2013 5:00 PM, Timothee Cour wrote:
> Let's put it another way: if I or someone else made a pull request for CTFE
> "exec", would it have a chance of being accepted?

A big problem with it would be the equivalent of the "SQL Injection Exploit". Since the compiler can now execute arbitrary code, someone passing around malicious source code could do anything to your system.

On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:
>
> A big problem with it would be the equivalent of the "SQL Injection Exploit". Since the compiler can now execute arbitrary code, someone passing around malicious source code could do anything to your system.

Assuming that the user is compiling the code in order to run it (which does seem to be the most common case, at least in my experience), the user is already running arbitrary code. I don't really see how this would create a greater security risk than what already exists.

That said, I'm not completely sold on this idea, either, at least not while there are more important issues to solve. It could be nice at times, but I'm having a hard time coming up with a usage case where this would really be much more convenient than just using the build system.

On 7/12/2013 11:52 PM, BLM768 wrote:
> On Saturday, 13 July 2013 at 04:23:56 UTC, Walter Bright wrote:
>>
>> A big problem with it would be the equivalent of the "SQL Injection Exploit".
>> Since the compiler can now execute arbitrary code, someone passing around
>> malicious source code could do anything to your system.
>
> Assuming that the user is compiling the code in order to run it (which does seem
> to be the most common case, at least in my experience), the user is already
> running arbitrary code. I don't really see how this would create a greater
> security risk than what already exists.

People can be endlessly creative at this kind of thing. I'm not at all sure you and I have thought of every possible exploit.

Am 13.07.2013 01:43, schrieb H. S. Teoh:
> On Fri, Jul 12, 2013 at 03:35:30PM -0700, Walter Bright wrote:
>> On 7/12/2013 3:12 PM, Timothee Cour wrote:
>>> I think the OP was refering to something different: ability to call
>>> an arbitrary executable / shell command during compile time of a D
>>> function, whereas optabgen is during compiling dmd itself:
>>
>> It's still the same idea - using external programs to generate source
>> code.
>
> This idea isn't new. lex/yacc (or their modern incarnations flex/bison)
> come to mind. The usage is a bit clunky, but the essence is the same.
>
>
>>> what we want to have is this:
>>
>> I do understand that. I'm just saying that this can currently (but
>> awkwardly) be done in the makefile.
>
> At what point does the balance shift from having such an ability
> built-in, vs. just using OS-level facilities for combining different
> programs? For example, one could pipe source through a program that
> performs arbitrary transformations on it, then pipe the result through
> the compiler. Or one can write a program that generates arbitrary source
> code and pipe that into the compiler.
>
> Presently, such things are easily handled by a modern build system (of
> which makefiles are a rather clunky implementation thereof).
>
>
> T
>

In most modern build systems you can easily achieve that via plugins.

Does DUB accept plugins?

--
Paulo