On 6/15/11 3:47 PM, Nick Sabalausky wrote:
> "Andrei Alexandrescu"<SeeWebsiteForEmail@erdani.org>  wrote in message
> news:itagdr$29mt$1@digitalmars.com...
>> On 6/15/11 8:33 AM, Steven Schveighoffer wrote:
>>> I can't really think of any other issues.
>>
>> Allow me to repeat: the scheme as you mention it is unable to figure and
>> load dependent remote libraries for remote libraries. It's essentially a
>> flat scheme in which you know only the top remote library but nothing
>> about the rest.
>>
>> The dip takes care of that by using transitivity and by relying on the
>> presence of dependency information exactly where it belongs - in the
>> dependent source files.
>
> Dependency information is already in the source: The "import" statement.
>
> The actual path to the depndencies does not belong in the source file - that
> *is* a configuration matter, and cramming it into the source only makes
> configuring harder.

Why? I mean I can't believe it just because you are saying it. On the face of it, it seems that on the contrary, there's no more need for crummy little configuration files definition, discovery, adjustment, parsing, etc. Clearly such are needed in certain situations but I see no reason on why they must be the only way to go.


Andrei

On 6/15/11 3:48 PM, Mike Wey wrote:
> First i didn't read all of the posts in this thread, so some of these
> might already be answered.
>
> In the first paragraph the DIP talks about Automatic downloading of
> *libraries* while all the posts here talk about downloading files.
> This is also reflected in the "Package case" paragraph since the
> compiler / separate tool will first try to download a .di file.
> Which generally is a d import or header file, which doesn't need to
> include the implementation, so the compiled library should also be
> downloaded or linking would fail, right?

That is correct. We need to address the scenario in which a .di file requires the existence of a .a/.lib file.

> Also the proposal doesn't do anything with versioning, while larger
> updates will probably get a different url, bug fixes might still
> introduce regressions that silently break an application that uses the
> library.

I think this is a policy matter that depends on the URLs published by the library writer.

> And now you'll have to track down witch library introduced the
> bug, and more importantly your app broke overnight and while you didn't
> change anything. (other that recompiling)
>
> To find out how downloading the files would work i did some tests with
> GtkD.
>
> Building GtkD itself takes 1m56.
> Building an Helloworld app that uses the prebuild library takes 0m01.
>
> The Helloworld app need 133 files from GtkD.
> Building the app and the files it needs takes 0m24.
>
> The source of the HelloWord application can be found here:
> http://www.dsource.org/projects/gtkd/browser/trunk/demos/gtk/HelloWorld.d
>

Thanks for the measurements. So my understanding is that the slow helloworld essentially compiles those 133 files from GtkD in addition to helloworld itself?


Thanks,

Andrei

On 06/15/2011 11:10 PM, Andrei Alexandrescu wrote:
> On 6/15/11 3:48 PM, Mike Wey wrote:
>> First i didn't read all of the posts in this thread, so some of these
>> might already be answered.
>>
>> In the first paragraph the DIP talks about Automatic downloading of
>> *libraries* while all the posts here talk about downloading files.
>> This is also reflected in the "Package case" paragraph since the
>> compiler / separate tool will first try to download a .di file.
>> Which generally is a d import or header file, which doesn't need to
>> include the implementation, so the compiled library should also be
>> downloaded or linking would fail, right?
>
> That is correct. We need to address the scenario in which a .di file
> requires the existence of a .a/.lib file.
>
>> Also the proposal doesn't do anything with versioning, while larger
>> updates will probably get a different url, bug fixes might still
>> introduce regressions that silently break an application that uses the
>> library.
>
> I think this is a policy matter that depends on the URLs published by
> the library writer.

But a different url for every bugfix would be difficult to maintain.

>
>> And now you'll have to track down witch library introduced the
>> bug, and more importantly your app broke overnight and while you didn't
>> change anything. (other that recompiling)
>>
>> To find out how downloading the files would work i did some tests with
>> GtkD.
>>
>> Building GtkD itself takes 1m56.
>> Building an Helloworld app that uses the prebuild library takes 0m01.
>>
>> The Helloworld app need 133 files from GtkD.
>> Building the app and the files it needs takes 0m24.
>>
>> The source of the HelloWord application can be found here:
>> http://www.dsource.org/projects/gtkd/browser/trunk/demos/gtk/HelloWorld.d
>>
>
> Thanks for the measurements. So my understanding is that the slow
> helloworld essentially compiles those 133 files from GtkD in addition to
> helloworld itself?

Yes, thats correct.

>
>
> Thanks,
>
> Andrei

-- 
Mike Wey

On Wed, 15 Jun 2011 14:37:29 -0400, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 6/15/11 12:35 PM, Steven Schveighoffer wrote:
>> I propose the following:
>
> Excellent. I'm on board with everything. Could you please update the DIP reflecting these ideas?
>

Updated, also added it to the DIP index.

-Steve

On 6/15/11 8:33 PM, Steven Schveighoffer wrote:
> On Tue, 14 Jun 2011 09:53:16 -0400, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>
>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>
>> Destroy.
>
> I put this as replies in several threads, but I'll throw it out there as
> its own thread:
>
> * You already agree that having the fetching done by a separate program
> (possibly written in d) makes the solution cleaner (i.e. you are not
> infiltrating the code that actually does compiling with code that does
> network fetching).
>
> * I think specifying the entire url in the pragma is akin to specifying
> the full path of a given module on your local disk. I think it's not the
> right place for it, the person who is building the code should be
> responsible for where the modules come from, and import should continue
> to specify the module relative to the include path.
>
> * A perfect (IMO) way to configure the fetch tool is by using the same
> mechanism that configures dmd on how to get modules -- the include path.
> For instance -Ihttp://xxx.yyy.zzz/package can be passed to the compiler
> or put into the dmd.conf.
>
> * DMD already has a good mechanism to specify configuration and you
> would barely have to change anything internally.
>
> Here's how it would work. I'll specify how it goes from command line to
> final (note the http path is not a valid path, it's just an example):
>
> dmd -Ihttp://www.dsource.org/projects/dcollections/import testproj.d
>
> 1. dmd recognizes the url pattern and stores this as an 'external' path
> 2. dmd reads the file testproj.d and sees that it imports
> dcollections.TreeMap
> 3. Using it's non-external paths, it cannot find the module.
> 4. It calls:
> dget -Ihttp://www.dsource.org/projects/dcollections/import
> dcollections.TreeMap
> 5. dget checks its internal cache to see if the file
> dcollections/TreeMap.[d|di] already exists -- not found
> 6. dget uses internal logic to generate a request to download either
> a. an entire package which contains the requested import (preferred)
> b. just the specific file dcollections/TreeMap.d
> 7. Using the url as a key, it stores the TreeMap.d file in a cache so it
> doesn't have to download it again (can be stored globally or local to
> the user/project)
> 8. Pipes the file to stdout, dmd reads the file, and returns 0 for success
> 9. dmd finishes compiling.

So if I have a library with three modules, a.d, b.d, c.d, which depend on another library, I should put that pragma(importpath) on each of them with the same url?

Or maybe I could create a fake d file with that pragma, and make the three modules import it so I just specify it once.

On Wed, 15 Jun 2011 23:23:43 -0400, Ary Manzana <ary@esperanto.org.ar> wrote:

> On 6/15/11 8:33 PM, Steven Schveighoffer wrote:
>> On Tue, 14 Jun 2011 09:53:16 -0400, Andrei Alexandrescu
>> <SeeWebsiteForEmail@erdani.org> wrote:
>>
>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>
>>> Destroy.
>>
>> I put this as replies in several threads, but I'll throw it out there as
>> its own thread:
>>
>> * You already agree that having the fetching done by a separate program
>> (possibly written in d) makes the solution cleaner (i.e. you are not
>> infiltrating the code that actually does compiling with code that does
>> network fetching).
>>
>> * I think specifying the entire url in the pragma is akin to specifying
>> the full path of a given module on your local disk. I think it's not the
>> right place for it, the person who is building the code should be
>> responsible for where the modules come from, and import should continue
>> to specify the module relative to the include path.
>>
>> * A perfect (IMO) way to configure the fetch tool is by using the same
>> mechanism that configures dmd on how to get modules -- the include path.
>> For instance -Ihttp://xxx.yyy.zzz/package can be passed to the compiler
>> or put into the dmd.conf.
>>
>> * DMD already has a good mechanism to specify configuration and you
>> would barely have to change anything internally.
>>
>> Here's how it would work. I'll specify how it goes from command line to
>> final (note the http path is not a valid path, it's just an example):
>>
>> dmd -Ihttp://www.dsource.org/projects/dcollections/import testproj.d
>>
>> 1. dmd recognizes the url pattern and stores this as an 'external' path
>> 2. dmd reads the file testproj.d and sees that it imports
>> dcollections.TreeMap
>> 3. Using it's non-external paths, it cannot find the module.
>> 4. It calls:
>> dget -Ihttp://www.dsource.org/projects/dcollections/import
>> dcollections.TreeMap
>> 5. dget checks its internal cache to see if the file
>> dcollections/TreeMap.[d|di] already exists -- not found
>> 6. dget uses internal logic to generate a request to download either
>> a. an entire package which contains the requested import (preferred)
>> b. just the specific file dcollections/TreeMap.d
>> 7. Using the url as a key, it stores the TreeMap.d file in a cache so it
>> doesn't have to download it again (can be stored globally or local to
>> the user/project)
>> 8. Pipes the file to stdout, dmd reads the file, and returns 0 for success
>> 9. dmd finishes compiling.
>
> So if I have a library with three modules, a.d, b.d, c.d, which depend on another library, I should put that pragma(importpath) on each of them with the same url?

With the updated proposal (please see the DIP now), you can do -I to specify the import path on the command line.  Otherwise, yes, you have to duplicate it.

> Or maybe I could create a fake d file with that pragma, and make the three modules import it so I just specify it once.

As long as the fake d file imports the files you need publicly, it should be pulled in, yes.  But the import pragma only affects imports from the current file.  I think that seems right, because you don't want to worry about importing files that might affect your import paths.  I look at it like version statements.

-Steve

"Andrei Alexandrescu" <SeeWebsiteForEmail@erdani.org> wrote in message news:itb6os$161f$1@digitalmars.com...
> On 6/15/11 3:47 PM, Nick Sabalausky wrote:
>> "Andrei Alexandrescu"<SeeWebsiteForEmail@erdani.org>  wrote in message news:itagdr$29mt$1@digitalmars.com...
>>> On 6/15/11 8:33 AM, Steven Schveighoffer wrote:
>>>> I can't really think of any other issues.
>>>
>>> Allow me to repeat: the scheme as you mention it is unable to figure and load dependent remote libraries for remote libraries. It's essentially a flat scheme in which you know only the top remote library but nothing about the rest.
>>>
>>> The dip takes care of that by using transitivity and by relying on the presence of dependency information exactly where it belongs - in the dependent source files.
>>
>> Dependency information is already in the source: The "import" statement.
>>
>> The actual path to the depndencies does not belong in the source file -
>> that
>> *is* a configuration matter, and cramming it into the source only makes
>> configuring harder.
>
> Why? I mean I can't believe it just because you are saying it. On the face of it, it seems that on the contrary, there's no more need for crummy little configuration files definition, discovery, adjustment, parsing, etc. Clearly such are needed in certain situations but I see no reason on why they must be the only way to go.
>

I do have reasons, but TBH I really don't have any more time or energy for these uphill debates right now.

On 2011-06-14 15:53, Andrei Alexandrescu wrote:
> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>
> Destroy.
>
>
> Andrei

Instead of complaining about others ideas (I'll probably do that as well :) ), here's my idea: https://github.com/jacob-carlborg/orbit/wiki/Oribt-Package-Manager-for-D

I'm working on both of the tools mentioned in the above link. The ideas for the package manager are heavily based on Rubygems.

-- 
/Jacob Carlborg

On 2011-06-14 16:09, Vladimir Panteleev wrote:
> On Tue, 14 Jun 2011 16:53:16 +0300, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>
>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>
> Why this is a bad idea:
> 1) It hard-codes URLs in source code. Projects often move to other
> code-hosting services. PHP, Python, Perl, not sure about Ruby all have a
> central website which stores package metadata.
> 2) It requires that the raw source code be available via HTTP. Not all
> code hosting services allow this. GitHub will redirect all HTTP requests
> to HTTPS.
> 3) It only solves the problem for D modules, but not any other possible
> dependencies.
>
> I understand that this is a very urgent problem, but my opinion is that
> this half-arsed solution will only delay implementing and cause
> migration problems to a real solution, which should be able to handle
> svn/hg/git checkout, proper packages with custom build scripts,
> versioning, miscellaneous dependencies, publishing, etc.

I agree with this, see my ideas in an answer to the original post.

-- 
/Jacob Carlborg

On 2011-06-14 17:31, Andrei Alexandrescu wrote:
> On 6/14/11 10:27 AM, Graham Fawcett wrote:
>> On Tue, 14 Jun 2011 08:53:16 -0500, Andrei Alexandrescu wrote:
>>
>>> http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11
>>>
>>> Destroy.
>>>
>>>
>>> Andrei
>>
>> What's the expected payload format? A text/plain D source-file? A zip or
>> tar archive?
>
> Text for now.

If the tool will download individual text files it will be quite ineffective, it's like comparing a svn checkout with a git clone. The git clone is a lot more effective. It would be better to download an archive of some sort.

>> If an archive, what's the required directory layout? Should
>> "dsource.foo.baz" be required to be in "/dsource/foo/baz.d" within the
>> archive?
>
> I agree we need to address these in the future, and also binary
> distributions (e.g. .di files + .a/.lib files).
>
>> And if not an archive, how to reasonably handle multi-file packages?
>
> Consider a library "acme" consisting of three files: widgets.d,
> gadgets.d, fidgets.d in "http://acme.com/d/". It also depends on the
> external library monads on "http://nad.mo/d".
>
> // User code:
> pragma(lib, acme, "http://acme.com/d/");
> import acme.widgets;
> ... use ...
>
> // widgets.d
> // Assume it depends on other stuff in the same lib
> // and on monads.d
> pragma(lib, monads, "http://nad.mo/d/");
> import acme.gadgets, acme.fidgets, monads.io;
>
> This is all that's needed for the compiler to download and compile
> everything needed.
>
>
> Andrei


-- 
/Jacob Carlborg