It was (correctly) pointed out that textual import breaks Bud and Rebuild. It's also been pointed out that textual import may be an unexpected vector for security problems.

Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.

For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.

Walter Bright wrote:
> It was (correctly) pointed out that textual import breaks Bud and Rebuild. It's also been pointed out that textual import may be an unexpected vector for security problems.
> 
> Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.
> 
> For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.

I think that's great. Many advanced/powerful language features are unusual and may make tools unhappy. A classic is Java's runtime reflection, which wreaks havoc with pretty much all static analysis tools.


Andrei

Walter Bright wrote:
> It was (correctly) pointed out that textual import breaks Bud and Rebuild. It's also been pointed out that textual import may be an unexpected vector for security problems.
> 
> Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.
> 
> For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.

I can't speak to the security aspect, but I don't think that -J would be helpful to rebuild. For example, if you have a file a.d:

mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));


It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely complex situation. Each of them will only work in some scenario.

What rebuild sees is just a.d .  -Jpath would suggest that a.d depends on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for it to know which short of actually resolving the templates.


The solution? Right now, there's no way to get the list of dependant files without compiling one. Since rebuild gets meta-data out of files, it would then have to compile it again, which is why I'm not doing it that way. I'd like to see something like this:

$ [g]dmd -p -files foo.d
file foo.d foo
file /opt/something/src/phobos/object.d object
file /opt/something/src/phobos/std/path.d std.path


That way, I could simply run the compiler, and get the list of actual files (not just imports) from its output. However, the compiler wouldn't actually compile anything (-p meaning parse-only), and it would list all the /files/ used (not just the imports).


Any way you'd add this feature? :)

 - Gregor Richards

On Wed, 07 Feb 2007 15:38:40 -0800, Gregor Richards wrote:

> Walter Bright wrote:
>> It was (correctly) pointed out that textual import breaks Bud and Rebuild. It's also been pointed out that textual import may be an unexpected vector for security problems.
>> 
>> Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.
>> 
>> For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.
> 
> I can't speak to the security aspect, but I don't think that -J would be
> helpful to rebuild.
> ...
> The solution? Right now, there's no way to get the list of dependant
> files without compiling one. Since rebuild gets meta-data out of files,
> it would then have to compile it again, which is why I'm not doing it
> that way. I'd like to see something like this:
> 
> $ [g]dmd -p -files foo.d
> file foo.d foo
> file /opt/something/src/phobos/object.d object
> file /opt/something/src/phobos/std/path.d std.path
> 
> That way, I could simply run the compiler, and get the list of actual files (not just imports) from its output. However, the compiler wouldn't actually compile anything (-p meaning parse-only), and it would list all the /files/ used (not just the imports).
> 
> Any way you'd add this feature? :)

I totally agree with Gregor. The -J switch might give us a hint that the files /on the command line/ must be recompiled, but it still doesn't make it easier to find out what other files are needed in the application. One purpose of Bud, and I assume Rebuild, is to allow the coder to just enter one name on the command line and get all the other required files magically included in the compilation process. The alternative is to require the coder to maintain makefiles and/or type every required file onto the command line.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
8/02/2007 10:42:00 AM

Gregor Richards wrote:
> I can't speak to the security aspect, but I don't think that -J would be helpful to rebuild. For example, if you have a file a.d:
> 
> mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));
> 
> 
> It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely complex situation. Each of them will only work in some scenario.
> 
> What rebuild sees is just a.d .  -Jpath would suggest that a.d depends on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for it to know which short of actually resolving the templates.

That's right, so when it sees -J, it just always rebuilds it (not just if one of its dependencies is newer).


> The solution? Right now, there's no way to get the list of dependant files without compiling one. Since rebuild gets meta-data out of files, it would then have to compile it again, which is why I'm not doing it that way. I'd like to see something like this:
> 
> $ [g]dmd -p -files foo.d
> file foo.d foo
> file /opt/something/src/phobos/object.d object
> file /opt/something/src/phobos/std/path.d std.path
> 
> 
> That way, I could simply run the compiler, and get the list of actual files (not just imports) from its output. However, the compiler wouldn't actually compile anything (-p meaning parse-only), and it would list all the /files/ used (not just the imports).
> 
> 
> Any way you'd add this feature? :)

It's a good idea.

Walter Bright wrote:
> Gregor Richards wrote:
> 
>> I can't speak to the security aspect, but I don't think that -J would be helpful to rebuild. For example, if you have a file a.d:
>>
>> mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));
>>
>>
>> It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely complex situation. Each of them will only work in some scenario.
>>
>> What rebuild sees is just a.d .  -Jpath would suggest that a.d depends on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for it to know which short of actually resolving the templates.
> 
> 
> That's right, so when it sees -J, it just always rebuilds it (not just if one of its dependencies is newer).

I think we're talking about slightly different scenarios. If it simply mixes in the file, then this would work perfectly. So would the below if it listed files that were imported with import(filename)

But, if it mixes in "import <module name>;", then I'll need not only to compile a.d, but whatever module is imported via mixin("import").  To compile this next module, the feature I listed below would be necessary.


> 
> 
>> The solution? Right now, there's no way to get the list of dependant files without compiling one. Since rebuild gets meta-data out of files, it would then have to compile it again, which is why I'm not doing it that way. I'd like to see something like this:
>>
>> $ [g]dmd -p -files foo.d
>> file foo.d foo
>> file /opt/something/src/phobos/object.d object
>> file /opt/something/src/phobos/std/path.d std.path
>>
>>
>> That way, I could simply run the compiler, and get the list of actual files (not just imports) from its output. However, the compiler wouldn't actually compile anything (-p meaning parse-only), and it would list all the /files/ used (not just the imports).
>>
>>
>> Any way you'd add this feature? :)
> 
> 
> It's a good idea.

Fantastic :)

 - Gregor Richards

Walter Bright wrote:
> Gregor Richards wrote:
>> I can't speak to the security aspect, but I don't think that -J would be helpful to rebuild. For example, if you have a file a.d:
>>
>> mixin(SomeRidiculouslyComplicatedTemplate!(WithRidiculousArguments));
>>
>>
>> It imports foo/b.d, foo/c.d or foo/d.d depending on some bizarrely complex situation. Each of them will only work in some scenario.
>>
>> What rebuild sees is just a.d .  -Jpath would suggest that a.d depends on at least one of foo/b.d, foo/c.d or foo/d.d, but there's no way for it to know which short of actually resolving the templates.
> 
> That's right, so when it sees -J, it just always rebuilds it (not just if one of its dependencies is newer).

That still doesn't tell it what the dependencies *are*. In the above scenario, one of foo/{b,c,d}.d needs to be built & linked as well (as well as all modules it depends on, and so on).
Which, I presume, the feature below was requested.

>> The solution? Right now, there's no way to get the list of dependant files without compiling one. Since rebuild gets meta-data out of files, it would then have to compile it again, which is why I'm not doing it that way. I'd like to see something like this:
>>
>> $ [g]dmd -p -files foo.d
>> file foo.d foo
>> file /opt/something/src/phobos/object.d object
>> file /opt/something/src/phobos/std/path.d std.path
>>
>>
>> That way, I could simply run the compiler, and get the list of actual files (not just imports) from its output. However, the compiler wouldn't actually compile anything (-p meaning parse-only), and it would list all the /files/ used (not just the imports).
>>
>>
>> Any way you'd add this feature? :)
> 
> It's a good idea.

Good to hear :)

Derek Parnell wrote:
> 
> 
> I totally agree with Gregor. The -J switch might give us a hint that the
> files /on the command line/ must be recompiled, but it still doesn't make
> it easier to find out what other files are needed in the application. One
> purpose of Bud, and I assume Rebuild, is to allow the coder to just enter
> one name on the command line and get all the other required files magically
> included in the compilation process. The alternative is to require the
> coder to maintain makefiles and/or type every required file onto the
> command line.
> 


All that build tool needs is to maintain a list of *everything* that was looked at. The first time around (clean build) everything has to be built anyway so you just walk around building things. until everything is built, then link every .o you just built. After that you look at the list of things that you used last time, and if any of them change you rebuild everything that got you there.

the only thing hindering this I can think of is an apparent desirer to have the build process be state-less (it isn't now as we leave around lots of .o files).

On Wed, 07 Feb 2007 13:40:16 -0800, Walter Bright <newshound@digitalmars.com> wrote:

>It was (correctly) pointed out that textual import breaks Bud and Rebuild. It's also been pointed out that textual import may be an unexpected vector for security problems.
>
>Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.
>
>For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.

Will relative paths to subdirectories of 'path' be allowed in the import expression (a behavior similar to ordinary imports)?

import ("file") - 'path'/file
import ("subdir/file") - 'path'/subdir/file
import ("/etc/passwd") - error
import ("../file") - error

Max Samukha wrote:
> On Wed, 07 Feb 2007 13:40:16 -0800, Walter Bright
> <newshound@digitalmars.com> wrote:
>> Both can be resolved by only allowing textual import if a command switch, say, -Jpath, is given. 'path' gives the location of where to look for the file; and the file will be restricted to being under that path. No -Jpath, and textual import won't be allowed.
>>
>> For Bud and Rebuild, if there's no -J, they know there are no textual imports, so they work as before. With -J, they always recompile.
> 
> Will relative paths to subdirectories of 'path' be allowed in the
> import expression (a behavior similar to ordinary imports)?
> 
> import ("file") - 'path'/file
> import ("subdir/file") - 'path'/subdir/file
> import ("/etc/passwd") - error
> import ("../file") - error

No paths at all will be allowed in the import string. They'll have to be supplied via the -Jpath command line switch. It's probably more conservative than necessary, but:

1) introducing host system dependent path separators makes the source code non-portable

2) it gives the 'buildmaster' an easy way to find out and control what files are imported

3) I think it's best to err on the side of conservatism here