Steven Schveighoffer wrote:
> dmd: damn, I guess I'll need to check with the downloader, hey downloader

That's a really good idea. It could be a command line argument that you set to any external program you want. The dmd distribution might bundle a default downloader, so it just works out of the box, but is still a separate program.

> I haven't seen it.

Here's the code: http://arsdnet.net/dcode/build.d

The idea behind it:

   try: dmd -v <args>

   watch for missing import file errors

   if there is a missing file, download the file from a server to
   the local directory. Use the name dmd provided.

   once it's downloaded, add that file to the dmd command line.

   goto try;

Am 14.06.2011 21:34, schrieb Robert Clipsham:
> On 14/06/2011 20:07, Andrei Alexandrescu wrote:
>> On 6/14/11 1:22 PM, Robert Clipsham wrote:
>>> On 14/06/2011 14:53, Andrei Alexandrescu wrote:
>>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>>
>>>> Destroy.
>>>>
>>>>
>>>> Andrei
>>>
>>> This doesn't seem like the right solution to the problem - the correct solution, in my opinion, is to have a build tool/package manager handle this, not the compiler.
>>>
>>> Problems I see:
>>> * Remote server gets hacked, everyone using the library now
>>> executes malicious code
>>
>> This liability is not different from a traditional setup.
> 
> Perhaps, but with a proper package management tool this can be avoided with sha sums etc, this can't happen with a direct get. Admittedly this line of defense falls if the intermediate server is hacked.
> 

Signing the files/hashes with GPG helps (as long as the developers
private key isn't on the server).

On 6/14/11 2:24 PM, Robert Clipsham wrote:
> On 14/06/2011 20:14, Andrei Alexandrescu wrote:
>> On 6/14/11 1:58 PM, Steven Schveighoffer wrote:
>>> I think it should be split as follows:
>>>
>>> dmd: determine *what* to download (i.e. I need to import module x.y.z)
>>
>> It can't (unless it does the download too) due to transitive
>> dependencies.
>
> It can work with the build tool though - build foo -> depends on bar.
> Build tool gets bar -> build bar -> depends baz, etc.

Yes, that's what Adam's tool does. Langsam, langsam, aber sicher...

Andrei

On 6/14/11 2:25 PM, Steven Schveighoffer wrote:
> On Tue, 14 Jun 2011 15:14:28 -0400, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>
>> On 6/14/11 1:58 PM, Steven Schveighoffer wrote:
>>> I think it should be split as follows:
>>>
>>> dmd: determine *what* to download (i.e. I need to import module x.y.z)
>>
>> It can't (unless it does the download too) due to transitive
>> dependencies.
>
> dmd: I need module foo.awesome. Where is it?
> filesystem: nope, don't have it
> dmd: damn, I guess I'll need to check with the downloader, hey
> downloader you have that?
> downloader: hm... oh, yeah! I'll get it for you
> filesystem: got it
> dmd: ok, now what's in foo.awesome? Oh, hm... foo.awesome needs
> bar.gnarly. Let me guess filesystem...
> filesystem: yeah, I suck today, go ask downloader
> ...
>
> How hard is that? I mean the actual downloading of files is pretty
> straightforward, at some point the problem reduces to "download a file".
> Why do we have to reinvent *that* wheel.

It's not hard, in fact that's almost how we want to implement it: a straight function that wraps a call to wget. The difference is that you'd migrate the function into a separate utility, and I think that's a good idea. (Walter prefers it inside the compiler.)

>> Not sure I grok 3 and 4, but as far as I can tell the crux of the
>> matter is that dependencies are already embedded in .d files. That's
>> why I think it's simpler to just let dmd take care of them all instead
>> of maintaining dependency description files in separation from the .d
>> files.
>
> And it would, why wouldn't it? I think you may not be getting something
> here...
>
>> The umpteen billion tools don't know what it takes to download and
>> build everything starting from one or a few root modules. They could
>> execute the download, yes (and of course we'll use such a library for
>> that), but we need a means to drive them.
>
> dmd would drive them.
>
>> I think Adam's tool is a good identification and alternative solution
>> for the problem that the pragma solves.
>
> I haven't seen it. Just thinking out loud...

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=138556


Andrei

On 6/14/11 2:34 PM, Robert Clipsham wrote:
> On 14/06/2011 20:07, Andrei Alexandrescu wrote:
>> On 6/14/11 1:22 PM, Robert Clipsham wrote:
>>> On 14/06/2011 14:53, Andrei Alexandrescu wrote:
>>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>>
>>>> Destroy.
>>>>
>>>>
>>>> Andrei
>>>
>>> This doesn't seem like the right solution to the problem - the correct
>>> solution, in my opinion, is to have a build tool/package manager handle
>>> this, not the compiler.
>>>
>>> Problems I see:
>>> * Remote server gets hacked, everyone using the library now
>>> executes malicious code
>>
>> This liability is not different from a traditional setup.
>
> Perhaps, but with a proper package management tool this can be avoided
> with sha sums etc, this can't happen with a direct get. Admittedly this
> line of defense falls if the intermediate server is hacked.

You may want to update the proposal with the appropriate security artifacts.

[snip]
> I don't have a problem with automatically downloading source during a
> first build, I do see a problem with getting the compiler to do it
> though. I don't believe the compiler should have anything to do with
> getting source code, unless the compiler also becomes a package manager
> and build tool.

Would you agree with the setup in which the compiler interacts during compilation with an external executable, placed in the same dir as the compiler, and with this spec?

dget "url"

Gets "url" and prints the local dir to stdout, or fails and prints an error message to stderr.

Then the matter is to write dget - in D!

I feel this is going somewhere.


Andrei

On 6/14/11 2:41 PM, Daniel Gibson wrote:
> Am 14.06.2011 21:34, schrieb Robert Clipsham:
>> On 14/06/2011 20:07, Andrei Alexandrescu wrote:
>>> On 6/14/11 1:22 PM, Robert Clipsham wrote:
>>>> On 14/06/2011 14:53, Andrei Alexandrescu wrote:
>>>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>>>
>>>>> Destroy.
>>>>>
>>>>>
>>>>> Andrei
>>>>
>>>> This doesn't seem like the right solution to the problem - the correct
>>>> solution, in my opinion, is to have a build tool/package manager handle
>>>> this, not the compiler.
>>>>
>>>> Problems I see:
>>>> * Remote server gets hacked, everyone using the library now
>>>> executes malicious code
>>>
>>> This liability is not different from a traditional setup.
>>
>> Perhaps, but with a proper package management tool this can be avoided
>> with sha sums etc, this can't happen with a direct get. Admittedly this
>> line of defense falls if the intermediate server is hacked.
>>
>
> Signing the files/hashes with GPG helps (as long as the developers
> private key isn't on the server).

Could you please add a subsection to the trust model discussing such a possibility?

Thanks,

Andrei

On 6/14/11 2:42 PM, Adam D. Ruppe wrote:
> Steven Schveighoffer wrote:
>> dmd: damn, I guess I'll need to check with the downloader, hey downloader
>
> That's a really good idea. It could be a command line argument that
> you set to any external program you want. The dmd distribution might
> bundle a default downloader, so it just works out of the box, but
> is still a separate program.

I, too, think this is awesome. Best of all we get to write that tool in D.

Andrei

BTW, I don't think it should be limited to just passing a url to the helper program.

I'd do it something like this:

dget module.name url_from_pragma


Basically, the compiler shouldn't withhold info - tell the
get program everything it might need, to keep it's options open
in how to do it's job.

If a url isn't given, the dget program is free to figure it out by some alternative means if it can, given the module name.

Am 14.06.2011 22:27, schrieb Andrei Alexandrescu:
> On 6/14/11 2:41 PM, Daniel Gibson wrote:
>> Am 14.06.2011 21:34, schrieb Robert Clipsham:
>>> On 14/06/2011 20:07, Andrei Alexandrescu wrote:
>>>> On 6/14/11 1:22 PM, Robert Clipsham wrote:
>>>>> On 14/06/2011 14:53, Andrei Alexandrescu wrote:
>>>>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>>>>
>>>>>> Destroy.
>>>>>>
>>>>>>
>>>>>> Andrei
>>>>>
>>>>> This doesn't seem like the right solution to the problem - the correct
>>>>> solution, in my opinion, is to have a build tool/package manager
>>>>> handle
>>>>> this, not the compiler.
>>>>>
>>>>> Problems I see:
>>>>> * Remote server gets hacked, everyone using the library now
>>>>> executes malicious code
>>>>
>>>> This liability is not different from a traditional setup.
>>>
>>> Perhaps, but with a proper package management tool this can be avoided with sha sums etc, this can't happen with a direct get. Admittedly this line of defense falls if the intermediate server is hacked.
>>>
>>
>> Signing the files/hashes with GPG helps (as long as the developers
>> private key isn't on the server).
> 
> Could you please add a subsection to the trust model discussing such a possibility?
> 
> Thanks,
> 
> Andrei

Done

On Tue, 14 Jun 2011 21:14:28 +0200, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 6/14/11 1:58 PM, Steven Schveighoffer wrote:
>> I think it should be split as follows:
>>
>> dmd: determine *what* to download (i.e. I need to import module x.y.z)
>
> It can't (unless it does the download too) due to transitive dependencies.

So pipe dmd's output to the build tool, dmd waits for the build tool to
feed it some info on stdin, and then continues, using the new info to
locate the downloaded files?

This way, the build tool is free to grab stuff using git, svn, from
additional library folders on the local computer, or even generate
random modules to feed to dmd.

-- 
  Simen