Graham Fawcett 
| Hi BCS,
On Fri, 07 May 2010 01:54:17 +0000, BCS wrote:
n>
> I haven't watched the video (not enough bandwidth) but I'd like a well done package tool as well. Thinking in the direction of a spec, what do you like about the Go system?
I'll summarize the video (which, by the way, contains everything I know about the Go package system!). The screencast shows a developer writing a short application with a reusable function in it. He factors the function out into a separate module in a subdirectory, then (with the aid of a copied Makefile) installs the new module into a shared location on the host, so it can be reused by other apps. Next, he pushes the new module into a github repository, and reinstalls the module from github like so:
goinstall github.com/nf/mytools # (github user nf, package mytools)
The source is checked out of github, compiled and installed in the central Go 'pkg' directory (both sources and objects are stored there). He commits some new changes to github, and reinstalls the updated module using the same goinstall command.
Finally, he returns to his original application, and replaces the Go equivalent of "import mytools;" with "import github.com/nf/mytools;" and the application now refers to the version of mytools that was installed from Github.
> Some things I'd like:
>
> -It must work offline (via sneeker net) and from trivial local static mirrrors/archives -It should support executing bash commands, D executables or something that does most of what people use that kind of things for. (Some kind of sandbox/security feature might be needed)
Stepping away from Go specifics, I like the idea of a system that does some of the following things. (just a brain-dump, no particular order):
(by the way, I'm overloading the term "package" here, at times to mean a package in the D sense, but usually to mean a downloadable and installable unit.)
* It knows how to build and install packages from an arbitrary source,
specified by a URL: a git/hg/svn repository, a tarball on the
filesystem, a zipfile on a Web server. Exotic sources/formats could
possibly be added with plugins.
* It keeps the ceremony of sharing code to a minimum. It should be
easy to publish a simple library package, without reading an excess
of documentation or writing an excess of package metadata.
* It separates the concerns of installing packages and managing a
central repository of packages. As an analogy, consider 'dpkg' vs.
'apt-get' on Debian systems: one knows about the mechanics of
package installation and removal, the other can search a central
catalogue, automatically install dependencies, etc. While 'd-get'
would be very nice, 'd-dpkg' is the essential building block. This
is important: the core package tool should not try to do more than
absolutely necessary to simply fetch and install code.
* It enables (but does not provide) apt-get-like use by providing ways
to (optionally) declare package dependencies (platform, D-version,
other needed packages), and include metadata about the package
(author, website, etc.) but does not require a complex package
description for simple packages. (An important usability question
is, what is the minimum package description that should be required
of a package's author? Can we work with no metadata at all?)
* It should be easy to have both user and global package directories.
It shouldn't be necessary to 'have root' to install a package.
* It doesn't depend on Make or another build tool, at least not for
basic library packages. (From the DSSS docs, it seems that DSSS was
able to manage dependency graphs, and do the right thing with a
simple 'dsss compile and install' command.) Having said that, a
package with complex build requirements ought to be buildable with a
Makefile, perhaps requiring a 'd-build' target in the Makefile that
builds the package into a state that meets our package system's
installation requirements.
* It makes it easy for library consumers to integrate the code. After
I've installed some third-party libaries, I'd like a tool that can
manage (or support) the build of an application that uses those
libraries -- for example, by figuring out which object files need to
be linked in, based on the modules I've imported. This could be a
tool that just calculates the necessary compiler and linker flags,
and/or a tool that runs the compiler/linker for me with those flags
in place, and/or a tool that builds a suitable Makefile for me,
based on my project's dependencies. (Arguably, this feature is
outside the scope of a simple fetch-and-install system; but if it's
not easy to use those fetched packages, then the overall value of
the system is weakened.)
* Nice to have: a way to allow multiple versions of the same package
to be installed locally, and a corresponding way to specify which
version of the package my application is using.
* Nice to have: a way to specify dependencies between third-party
packages, and dependencies on external libraries (not necessarily a
way to install those external dependencies, just a declaration).
* Nice to have: some support for running unit tests, especially a
basic 'pre-install' unit test to confirm that all dependencies are
in place (if possible).
* Nice to have: a way to publish to a central feed of "known D
packages". Less formal than dsource.org: not a user site, but a
machine-readable one, and possibly distributed/mirrorable. As an
example, the Clojure community has a "clojars.org" site which lets
anyone publish Clojure packages there. Clojars has very limited
support for searching/browsing, it's really more of a key/value
store: it is intended for use by automated tools (it's a Maven
repository, as I understand it). Clojars actually maintains an
archive of those packages (as jar files), this might/might not be a
good idea. I think the "feed" is the key thing.
* Centralizing the feed-of-packages would also enable another useful
feature: a way of mapping a canonical package name (and optionally,
version) to the URL of the source for that package. (Possibly
multiple URLs for the same source.) This is headed into 'apt-get'
territory, but minimizing the responsibility of the central
repository to "here's where to find package x.y.z" without any other
assurances (e.g., that x.y.z will build properly on your system).
Conventions about package naming (should my DSP package be called
"dsp", or "fawcett.dsp", "algorithms.signals.dsp", etc.) and
package-name dispute resolution are orthogonal and should be left
out of the scope of the system.
These next thoughts are outside the scope; but a good design of the core tool won't get in the way of future progress in these areas:
* Thinking ahead: ensure the system can be used by continuous
integration systems, so that e.g. a buildbot could read the feed for
new "known D packages", and attempt to install them on various
D-version and OS permutations, reporting the results. (I'm not
suggesting that the buildbot is part of the package management
system, just that it should be easy for buildbot maintainers to be
aware of new package versions so they can be tested.) The same
facility could be used by developers who want to keep a "mirror" of
some or all known D packages (even though those packages might
reside upstream on several different hosts).
* Thinking ahead: eventually we hope for a nice site where people can
browse, review, grade, and comment upon packages. An opt-in system
where consumers agree to publish their installation activity (like
Debian's 'popularity contest').
* Thinking ahead: easy message-digests and digital signatures for
packages, enabling a truly distributed package-publishing system.
Anyone can host a d-libraries mirror, and a signed "central feed"
document can publish the known hashes for those packages to avoid
man-in-the-middle attacks.
* Thinking ahead: I really like the way github.com shows a 52-week
activity graph on projects that are hosted there. I don't think we
can do exactly that, but it would be nice to be able to generate a
report for all packages, showing how frequently they are updated and
maintained.
* I personally favour a system that works with D2 only, but I am new
here. :) Otherwise, the implications of D1/D2 and Phobos/Tango need
to be considered by more seasoned heads than mine.
And for now, I'm all thought out. :) Your thoughts?
Best,
Graham
|
Permalink
Reply
