View mode: basic / threaded / horizontal-split · Log in · Help
February 16, 2013
The DUB package manager
With the recent talk about Orbit, I thought it is time to also announce
the package manager that we have been working out based on the simple
VPM system that has always been in vibe.d. I don't really like stepping
into competition with Jacob here (*), but the approach is different
enough that I think it should be put on the table.

Some may already have noticed it as it's mentioned already on the vibe.d
website and is currently hosted on the same domain as the old VPM registry:

http://registry.vibed.org/


DUB has two important development goals:

- Simplicity:

  Making a DUB package, as well as using one as a dependency should be
  as simple as possible to facilitate broad usage, also and especially
  among language newcomers. Procedural build scripts often scare away
  people, although their added complexity doesn't matter for bigger
  projects. I think they should be left as an option rather than the
  default.

  Turning a library/application into a DUB package can be as simple as
  adding a package.json file with the following content (mysql-native
  is automatically made available during the build in this example):

  {
       "name": "my-library",
       "dependencies": {"mysql-native": ">=0.0.7"}
  }

  If the project is hosted on GitHub, it can be directly registered on
  the registry site and is then available for anyone to use as a
  dependency. Alternatively, it is also possible to use a local
  directory as the source for a particular package (e.g. for closed
  source projects or when working on both the main project and the
  dependency at the same time).

- Full IDE support:

  Rather than focusing on performing the build by itself or tying a
  package to a particular build tool, DUB translates a general
  build receipt to any supported project format (it can also build
  by itself). Right now VisualD and MonoD are supported as targets and
  rdmd is used for simple command line builds. Especially the IDE
  support is really important to not simply lock out people who prefer
  them.


Apart from that we have tried to be as flexible as possible regarding
the way people can organize their projects (although by default it
assumes source code to be in "source/" and string imports in "views/",
if those folders exist).

There are still a number of missing features, but apart from those it is
fully usable and tested on Windows, Linux, and Mac OS.


GitHub repository:
https://github.com/rejectedsoftware/dub
https://github.com/rejectedsoftware/dub-registry

Preliminary package format documentation:
http://registry.vibed.org/package-format


(*) Originally I looked into using Orbit as the package manager for
vibe.d packages, but it just wasn't far enough at the time and had some
traits that I wasn't fully comfortable with.
February 16, 2013
Re: The DUB package manager
On 2013-02-16 18:10, Sönke Ludwig wrote:
> With the recent talk about Orbit, I thought it is time to also announce
> the package manager that we have been working out based on the simple
> VPM system that has always been in vibe.d. I don't really like stepping
> into competition with Jacob here (*), but the approach is different
> enough that I think it should be put on the table.
>
> Some may already have noticed it as it's mentioned already on the vibe.d
> website and is currently hosted on the same domain as the old VPM registry:
>
> http://registry.vibed.org/
>
>
> DUB has two important development goals:
>
>   - Simplicity:
>
>     Making a DUB package, as well as using one as a dependency should be
>     as simple as possible to facilitate broad usage, also and especially
>     among language newcomers. Procedural build scripts often scare away
>     people, although their added complexity doesn't matter for bigger
>     projects. I think they should be left as an option rather than the
>     default.
>
>     Turning a library/application into a DUB package can be as simple as
>     adding a package.json file with the following content (mysql-native
>     is automatically made available during the build in this example):
>
>     {
>          "name": "my-library",
>          "dependencies": {"mysql-native": ">=0.0.7"}
>     }

Using a full blow language can look pretty declarative as well:

name = "my-library";
dependencies = ["mysql-native": ">=0.0.7"];

I don't see why something like the above would scare away people. It's 
even less code than the JSON above.

>     If the project is hosted on GitHub, it can be directly registered on
>     the registry site and is then available for anyone to use as a
>     dependency. Alternatively, it is also possible to use a local
>     directory as the source for a particular package (e.g. for closed
>     source projects or when working on both the main project and the
>     dependency at the same time).
>
>   - Full IDE support:
>
>     Rather than focusing on performing the build by itself or tying a
>     package to a particular build tool, DUB translates a general
>     build receipt to any supported project format (it can also build
>     by itself). Right now VisualD and MonoD are supported as targets and
>     rdmd is used for simple command line builds. Especially the IDE
>     support is really important to not simply lock out people who prefer
>     them.

I think it looks like you're tying the user to a particular build tool. 
I don't think it's the business of the package manager to build 
software. That's the job of a build tool. The package manager should 
just invoke the build tool. You just need to support a few build tools, 
like rdmd, make and then also support shell script.

Things like setting compiler flags does really not belong in a package 
manger.

I have not looked at the source code for DUB at all. In general how is 
it organized. Can it be used as a library to easily build, say a GUI or 
directly integrate into an IDE?

-- 
/Jacob Carlborg
February 16, 2013
Re: The DUB package manager
On 2013-02-16 18:10, Sönke Ludwig wrote:

> Preliminary package format documentation:
> http://registry.vibed.org/package-format

BTW, "x64" is not a predefined version identifier.

-- 
/Jacob Carlborg
February 16, 2013
Re: The DUB package manager
Am 16.02.2013 19:10, schrieb Jacob Carlborg:
> 
> I think it looks like you're tying the user to a particular build tool.
> I don't think it's the business of the package manager to build
> software. That's the job of a build tool. The package manager should
> just invoke the build tool. You just need to support a few build tools,
> like rdmd, make and then also support shell script.

Not at all, the idea is to have a number of "build script" generators so
everyone can choose whatever fits best, otherwise a generic rdmd/dmd
based build works out of the box with no need to install an additional
tool. Invoking a generic external tool is easy enough to add and already
planned, so this should not be a limiting factor in any way.

> 
> Things like setting compiler flags does really not belong in a package
> manger.

What makes you think so? Just because of your definition of "package
manager" or for a concrete reason? There are things like setting up
import paths to dependent projects that only the package manager can do,
because it knows their locations (and it can be desirable for many
reasons to not install dependencies into a predefined place). Bringing
package management and build receipt together is convenient and natural
here.

You could also say that it is a meta-build tool (along the lines of
cmake) with package support if you want.

> 
> I have not looked at the source code for DUB at all. In general how is
> it organized. Can it be used as a library to easily build, say a GUI or
> directly integrate into an IDE?
> 

There should be no problem with that. The API will need to be cleaned up
a bit, but generally it's a shallow "app.d" file that does command line
processing and a number of modules that do the actual work (in a package
"dub").
February 16, 2013
Re: The DUB package manager
Am 16.02.2013 19:11, schrieb Jacob Carlborg:
> On 2013-02-16 18:10, Sönke Ludwig wrote:
> 
>> Preliminary package format documentation:
>> http://registry.vibed.org/package-format
> 
> BTW, "x64" is not a predefined version identifier.
> 

Thanks, you are right, that was a typo.
February 16, 2013
Re: The DUB package manager
On Saturday, 16 February 2013 at 17:10:33 UTC, Sönke Ludwig wrote:
> Apart from that we have tried to be as flexible as possible 
> regarding
> the way people can organize their projects (although by default 
> it
> assumes source code to be in "source/" and string imports in 
> "views/",
> if those folders exist).

Many projects put their source code in src/ instead of source/ .
Can DUB support this ?
February 16, 2013
Re: The DUB package manager
Am 16.02.2013 19:10, schrieb Jacob Carlborg:
> 
> Using a full blow language can look pretty declarative as well:
> 
> name = "my-library";
> dependencies = ["mysql-native": ">=0.0.7"];
> 
> I don't see why something like the above would scare away people. It's
> even less code than the JSON above.
> 

I was thinking more along the lines of make, cmake, D based build
scripts in the form as proposed in the past here and others. Such
procedural build files often tend to get bloated over time and hard to
work with. While this is not necessarily a problem with the tools
itself, it can still make people look for alternatives.

That said, if a scripting language is used (almost) purely to provide a
declarative system as in your example, it doesn't have to be bad at all.
The question that arguably remains is just if the added flexibility is
actually useful or necessary and if such a thing "pulls its own weight"
(WRT implementation and API complexity).
February 16, 2013
Re: The DUB package manager
Am 16.02.2013 20:07, schrieb SomeDude:
> On Saturday, 16 February 2013 at 17:10:33 UTC, Sönke Ludwig wrote:
>> Apart from that we have tried to be as flexible as possible regarding
>> the way people can organize their projects (although by default it
>> assumes source code to be in "source/" and string imports in "views/",
>> if those folders exist).
> 
> Many projects put their source code in src/ instead of source/ .
> Can DUB support this ?

Yes, it supports a "sourcePath" field to customize that.
February 16, 2013
Re: The DUB package manager
On 2013-02-16 19:39, Sönke Ludwig wrote:

> Not at all, the idea is to have a number of "build script" generators so
> everyone can choose whatever fits best, otherwise a generic rdmd/dmd
> based build works out of the box with no need to install an additional
> tool. Invoking a generic external tool is easy enough to add and already
> planned, so this should not be a limiting factor in any way.

Ok, I see. But it feels wrong to me that it should generate a build 
script. I think the package should already contain a build script.

> What makes you think so? Just because of your definition of "package
> manager" or for a concrete reason? There are things like setting up
> import paths to dependent projects that only the package manager can do,
> because it knows their locations (and it can be desirable for many
> reasons to not install dependencies into a predefined place). Bringing
> package management and build receipt together is convenient and natural
> here.

I'm thinking that there should be a build tool that drives everything. 
The build script contains package dependencies. The build tool will ask 
the package manager to get linker/import paths and libraries for the 
dependencies.

But I guess that basically the same as how DUB is working. But the 
package manager drives everything instead of the build tool.

> You could also say that it is a meta-build tool (along the lines of
> cmake) with package support if you want.

> There should be no problem with that. The API will need to be cleaned up
> a bit, but generally it's a shallow "app.d" file that does command line
> processing and a number of modules that do the actual work (in a package
> "dub").

Good, that's how it's supposed to be organized.

-- 
/Jacob Carlborg
February 16, 2013
Re: The DUB package manager
On Sat, 2013-02-16 at 20:15 +0100, Jacob Carlborg wrote:
[…]
> I'm thinking that there should be a build tool that drives everything. 
> The build script contains package dependencies. The build tool will ask 
> the package manager to get linker/import paths and libraries for the 
> dependencies.
[…]

First a plug for people to think of SCons and Waf, not to mention
Gradle, when starting to rattle of build tools.

The SBT folks are using Scala in quite interesting ways to enable Scala
to define project specifications with all the non-inferable
dependencies.

The Gradle folks have already done almost everything the SBT folks are
doing, but using Groovy rather than Scala.

The Go folk have done something interesting in that they have merged the
whole concept of configuration and build by doing everything over DVCS.
You put your sources in Git, Mercurial, Bazaar (they really should
include Fossil as well but…) and these can be Got and the modules
created within the standard structure local hierarchy.  The have a
single command that performs all activity. D has rdmd but compared to
Go's go command it cannot do a lot.

I would suggest now is the time to think outside the box, analogous to
the ways the Gradle and SBT folk have on the JVM and the way the Go folk
have for native builds of statically linked code.

Instead of thinking in terms of compile, link, modules, dependencies,
what is the workflow that makes D the compelling language for building
Fortran/C/C++/D systems. This is suggesting that the milieu to attack is
the one currently being won by Python in the computationally intensive
areas of bioinformatics and scientific computing.

Can D be the project specification language using convention over
configuration. Project directories are in a standard form (with
exceptions describable) with dependencies either inferred by scanning
the project sources or specified in a trivial way in the project
specification file.

Thus I suggest that it is not that the build tool is embedded in the
package manager but that package and dependency management is part of
the build system.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
« First   ‹ Prev
1 2 3 4 5
Top | Discussion index | About this forum | D home