I don't think this is a new idea but it would be pretty awesome.

So the idea is that the compiler could check dub for libraries that it can't find and automatically download and integrate them. It would essentially make all of dub seem like it was a part of the standard lib and make integrating dub into projects seamless.

Something like "import derelict.opengl3.gl3;" could automatically trigger dub if the import was not found on any local path.

It also might need some special syntax for version specifications.

It would require some work on both sides. Obviously the compiler would need to change but dub would have to as well, I don't think currently there is any way to look up a dub package by an import statement alone.



Pros:
Greatly increase ease of dub use.
Greatly increase libraries that "seem" to be a part of the std lib.
Would solve the large vs small stdlib dilemma.

Cons:
Could degrade perceived quality of stdlib if bad dub packages got in.
Would mean some things could not be compiled without an internet connection.

My answers to those cons would be

1)
It might require some policing to make sure quality stays high.
Maybe even with votes on what dub packages could automatically be picked up by this similar to how packages get into phobos but with a much lower bar.

Also maybe a different import syntax for these imports to clearly differentiate them so that we can draw a line between the standard lib and dub. (personally I don't like that idea, I like the idea of seamless integration)

2)
Normal code would still compile like normal, this would only open up the option of automatic library download for basically the 99% of people who always have an internet connection.




So yeah...
Thoughts?

On 27/10/2014 3:33 p.m., Tofu Ninja wrote:
> I don't think this is a new idea but it would be pretty awesome.
>
> So the idea is that the compiler could check dub for libraries that it
> can't find and automatically download and integrate them. It would
> essentially make all of dub seem like it was a part of the standard lib
> and make integrating dub into projects seamless.
>
> Something like "import derelict.opengl3.gl3;" could automatically
> trigger dub if the import was not found on any local path.
>
> It also might need some special syntax for version specifications.
>
> It would require some work on both sides. Obviously the compiler would
> need to change but dub would have to as well, I don't think currently
> there is any way to look up a dub package by an import statement alone.
>
>
>
> Pros:
> Greatly increase ease of dub use.
> Greatly increase libraries that "seem" to be a part of the std lib.
> Would solve the large vs small stdlib dilemma.
>
> Cons:
> Could degrade perceived quality of stdlib if bad dub packages got in.
> Would mean some things could not be compiled without an internet
> connection.
>
> My answers to those cons would be
>
> 1)
> It might require some policing to make sure quality stays high.
> Maybe even with votes on what dub packages could automatically be picked
> up by this similar to how packages get into phobos but with a much lower
> bar.
>
> Also maybe a different import syntax for these imports to clearly
> differentiate them so that we can draw a line between the standard lib
> and dub. (personally I don't like that idea, I like the idea of seamless
> integration)
>
> 2)
> Normal code would still compile like normal, this would only open up the
> option of automatic library download for basically the 99% of people who
> always have an internet connection.
>
>
>
>
> So yeah...
> Thoughts?

Just an idea, but we could take a leaf out of lua's book in this case.
Lua has the require statement. Which includes external scripts and is highly configurable in lua code.

With skeleton[0] I have my repository/file downloaded hooked into require. It uses a very specific format. E.g. skeleton@rikkimax means a repository for username rikkimax, but it can go one step further skeleton@rikkimax/file.d for a singular file.
There's more to it then just that but small recap.

For singular files import could be used quite well for this format.
import repo@user/foo.bar;

For that module the file loader handler within dmd could be switched out to that repository.

But that's not really dub support. So next step:
import repo@user/p1/foo.bar;

Repository for user, in directory p1 with foo.bar module.
Location of dub library is in p1.

That's a little better.
But it doesn't explain how to get support between dmd and dub.

First the reading of files (either passed in via cli or via import's) would have to be abstracted out in dmd. Basically who handles the import statement.
Next dub would need a new compiler support which would act as a callback instead of directly running dmd. Except when compiling dependencies. It would also need to use the existing settings of the host dmd call (to emulate as if dub was doing dependency compilation).

Okay now this is getting a little scary for feature set of dmd not to mention dmd requiring D program gah yeah no.
So, shared library plugins for dmd to enable this as an optional feature? Sounds like a good idea.

Of course we can go one step further and add in support for e.g. gists, pastebin, dpaste and of course plain text files stored on a web server.

Lastly, versions.

@version(`>=0.2.1`)
import repo@user;

[0] https://github.com/rikkimax/skeleton

i think it would be better to develop something like "universal interface" for this. so we can configure compiler to exec external program which does all the things. this way it wouldn't be tied to dub.

by the way, i believe that something like this can be done with external wrapper like rdmd. dubdmd, for example, which analyses imports the same way rdmd does and invokes dub for all missing libraries, and then invokes dmd with necessary options added.

On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
> Cons:
> Could degrade perceived quality of stdlib if bad dub packages got in.

Bad for security.

On Mon, 27 Oct 2014 03:00:49 +0000
via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
> > Cons:
> > Could degrade perceived quality of stdlib if bad dub packages
> > got in.
> 
> Bad for security.
this feature can be made explicit opt-in with proper message from the compiler.

On Monday, 27 October 2014 at 03:00:50 UTC, Ola Fosheim Grøstad wrote:
> On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
>> Cons:
>> Could degrade perceived quality of stdlib if bad dub packages got in.
>
> Bad for security.

My response to that is that any library you ever download is bad for security (including dmd and phobos). We need to draw the line somewhere for things we trust and things we don't trust, personally I draw the line where it best suits me to get things done. If giving up some small about of security allowed me to automatically integrate dub packages into my projects, I would happily give it up. :)

Also it is why I suggested that it could be policed.

On Monday, 27 October 2014 at 03:15:45 UTC, Tofu Ninja wrote:
> On Monday, 27 October 2014 at 03:00:50 UTC, Ola Fosheim Grøstad
>>
>> Bad for security.
>
> My response to that is that any library you ever download is bad for security (including dmd and phobos).

I currently run dmd on a separate user account…

> We need to draw the line somewhere for things we trust and things we don't trust, personally I draw the line where it best suits me to get things done. If giving up some small about of security allowed me to automatically integrate dub packages into my projects, I would happily give it up. :)

That's ok for a personal user account, but not for a work account IMO.

Then again, I prefer to fetch directly from repos manually and only use dub-like features for languages that run in a VM.

Another point is that if you make fetching libraries too easy it means bloat starts creeping in. OK for a scripting language, but for a system level language…? You risk ending up with Tango-bloat, where you cannot include anything without pulling inn all kinds of dependencies.

> Also it is why I suggested that it could be policed.

But the D community is too small for that atm.

On Mon, 27 Oct 2014 03:24:46 +0000
via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> Another point is that if you make fetching libraries too easy it means bloat starts creeping in. OK for a scripting language, but for a system level language…?
D is not only system-level language. D's metaprogramming features makes it very high-level too. writing CLI scripts with D and rdmd is a joy. and i never was a big fan of writing CLI scripts in C, despite the TCC's ability to to this.

On Monday, 27 October 2014 at 02:58:15 UTC, ketmar via Digitalmars-d wrote:
> i think it would be better to develop something like "universal
> interface" for this. so we can configure compiler to exec external
> program which does all the things. this way it wouldn't be tied to dub.

The universal interface to such a things could simply be an exe that gets called when an import is not found asking for the path of the import. The exe can do all sorts of things(like downloading from dub) before it finally returns the path to the import.

It would need someway to pass extra information to it though(like version specifiers) because I think it wouldn't work if it was embedded in the import statement it self(i mean as part of the import path which would not match with the actual module name).

> by the way, i believe that something like this can be done with
> external wrapper like rdmd. dubdmd, for example, which analyses imports
> the same way rdmd does and invokes dub for all missing libraries, and
> then invokes dmd with necessary options added.

It probably could, I didn't even think about that. But I think you're "universal" interface idea is a bit better and it seems like something that could very easily be added to rdmd or dmd itself in a jiffy.

On Mon, 27 Oct 2014 03:34:34 +0000
Tofu Ninja via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> It probably could, I didn't even think about that. But I think you're "universal" interface idea is a bit better and it seems like something that could very easily be added to rdmd or dmd itself in a jiffy.
i'm dreaming about having this as CTFE interface. one, for example, could write special module like 'core.import_resolver', DMD will load and parse it when it encounters missing import, and then evaluate some of it's well-defined API funcions that can prepare command line for the necessary tool or even execute that tool and parse it's output.

sure, this is not safe and not secure, so such such things as "allow import resolver to execute external programs" must be explicitly enabled with some CLI switch.