On Wed, 11 Sep 2013 21:10:46 +0800
Lionello Lunesu <lionello@lunesu.remove.com> wrote:

> On 9/11/13 5:01, Brad Anderson wrote:
> > I vote yes but only if Sönke feels it is ready. I suspect he has a few things he'll probably want done before this happens (the potential switch from JSON to SDL comes to mind).
> 
> SD-what?! Why would alienate people even more than we already do?
> 
> L.


----------------------------
{
	"name": "myproject",
	"description": "A little web service of mine.",
	"authors": ["Peter Parker", "Joe Contrib"],

	"dependencies": {
		"vibe-d": ">=0.7.11",
		"mylib:component1": "~master",
		"mylib:component2": "~master"
	},

	"subPackages": [
		{
			"name": "component1",
			"targetType": "library",
			"sourcePaths": ["source/component1"]
		},
		{
			"name": "component2",
			"targetType": "library",
			"sourcePaths": ["source/component2"]
		}
	]
}
----------------------------

vs:

----------------------------
name "myproject",
description "A little web service of mine."
authors "Peter Parker" "Joe Contrib"

dependencies {
	vibe-d ">=0.7.11"
	mylib:component1 "~master"
	mylib:component2 "~master"
}

subPackage {
	name "component1"
	targetType "library"
	sourcePaths "source/component1"
}

subPackage {
	name "component2"
	targetType "library"
	sourcePaths "source/component2"
}
----------------------------

That's why.

Besides, the JSON form wouldn't be going away anyway, it'd still be kept.

On Wednesday, 11 September 2013 at 13:39:02 UTC, Nick Sabalausky wrote:
> On Wed, 11 Sep 2013 21:10:46 +0800
> Lionello Lunesu <lionello@lunesu.remove.com> wrote:
>
>> On 9/11/13 5:01, Brad Anderson wrote:
>> > I vote yes but only if Sönke feels it is ready. I suspect he has a
>> > few things he'll probably want done before this happens (the
>> > potential switch from JSON to SDL comes to mind).
>> 
>> SD-what?! Why would alienate people even more than we already do?
>> 
>> L.
>
>
> ----------------------------
> {
> 	"name": "myproject",
> 	"description": "A little web service of mine.",
> 	"authors": ["Peter Parker", "Joe Contrib"],
>
> 	"dependencies": {
> 		"vibe-d": ">=0.7.11",
> 		"mylib:component1": "~master",
> 		"mylib:component2": "~master"
> 	},
>
> 	"subPackages": [
> 		{
> 			"name": "component1",
> 			"targetType": "library",
> 			"sourcePaths": ["source/component1"]
> 		},
> 		{
> 			"name": "component2",
> 			"targetType": "library",
> 			"sourcePaths": ["source/component2"]
> 		}
> 	]
> }
> ----------------------------
>
> vs:
>
> ----------------------------
> name "myproject",
> description "A little web service of mine."
> authors "Peter Parker" "Joe Contrib"
>
> dependencies {
> 	vibe-d ">=0.7.11"
> 	mylib:component1 "~master"
> 	mylib:component2 "~master"
> }
>
> subPackage {
> 	name "component1"
> 	targetType "library"
> 	sourcePaths "source/component1"
> }
>
> subPackage {
> 	name "component2"
> 	targetType "library"
> 	sourcePaths "source/component2"
> }
> ----------------------------
>
> That's why.
>
> Besides, the JSON form wouldn't be going away anyway, it'd still be
> kept.

Why not YAML? It's cleaner than JSON and is very widely known.

On 2013-09-11 13:30, Sönke Ludwig wrote:

> Right now it is a pure development tool. It would be very nice to have
> end user installs somehow supported (either by directly installing
> application packages or by generating OS specific packages such as DEB
> or RPM). But since this enters a highly operating specific area and goes
> into direct competition with the OS package manager, I think it needs a
> lot of thought and caution to be generally useful and not possibly do
> more harm than good in the end. But yes, it should be a primary goal in
> my opinion, too.

I'm thinking this type of package manager should be a development tool as well. But there are a lot of development tools that are executables and not just libraries. Think of your documentation generator. Without having looked at it I would assume it's an executable. I have myself a tool, DStep, which translate C headers to D modules. This is an executable as well.

> Why not _make_ a tag? But uploading zipped packages (or better
> specifying an external link) could be added as an alternative without
> much effort.

I'm not referring to uploading zip packages. Say I have ten tags but I only want dub to know about five of them. I want to manually say "make the tag v0.0.1 available".

> This is something that may still need some adjustments, but after all
> it's just a default when there is a "source/app.d" or
> "source/<packname>.d" file - an explicit "targetType": "xxx" will fix it
> when it's off.

Yes, but I still don't like the default. To avoid misunderstandings, will the dub, by default _both_ build an executable _and_ a library. Or will it build an executable _or_ a library?

> "views" is maybe not general enough, but for what it's worth I'm also
> using it on other kinds of projects (e.g. to specify GUI templates).
> "res" might be a bit *too* general, but this could for sure be improved
> somehow. "public" is indeed specific for web stuff, so it may be the
> best to drop it by default.

It's possible to import images, video, audio and many other type of resources as well using string imports.

> I've also thought about offering a set of template projects usable for
> "dub init", so that there could e.g. be a true vibe.d skeleton, but by
> default it would be a minimal project with only a "source" folder.

Yes, I was thinking the same.

> Note that there is "dub describe" to extract the information relevant
> for actual building plus there is "dub generate xxx", so nothing is in
> the way of using a separate tool for building. But having integrated
> building IMO is highly convenient and keeping the build description in a
> standardized format is one of the key selling points.

Hmm ok. I would go with two separate tools that are well integrated with each other.

BTW, neither "dub generate rdmd" or "dub generate build" seems to be working.

> You can put an additional "bar": "==0.0.1" dependency in the main
> package to lock it to a certain version.

I don't want to care about indirect dependencies. That's the job of the package manager. If I need to keep track of indirect dependencies then the package manager isn't helping much.

> Or, of course, just use "==" in the bar package in the first place.

I might only control the "myproject" package.

> This could also be implemented as a (semi-)automatic function along
> the lines of "dub lock-versions" and  "dub lock-versions --upgrade".

I think this should be the default. I don't want my package to break at random just because there's a newer version available of an indirect dependency.

I think that a key feature of a package manager should be that any time in time you're installing a package of a given version should result in the _exact_ same packages, including indirect packages.

> On the other hand it's difficult to make a general statement that this
> is always the best way as this may for example prevent important
> security fixes to get incorporated unless the main package maintainer
> releases an explicit update...

Any update, may it only be security fixes, may break a build. No, semantic versions doesn't help here. Locking down by default is the safest.

If the package maintainer doesn't release an update how will you get security updates?

It's better if there's an explicit way to update an indirect dependency.

I would rather add "bar": "==0.0.2" on one or two packages to force an update of an indirect dependency rather then set "bar:" "==0.0.1" on _all_ indirect dependency. This can be quite a large tree if there are many dependencies.

At work we have a Ruby on Rails project. It currently uses 160 packages, including Rails and all its sub components. It was pure hell before "bundler" was released which locks down all package dependencies as I've described here. Counting all indirect dependencies as well I get over 413 packages. Would you like to specify the exact version of all these?

Saying that "bundler" was a salvation for the Ruby community is an understatement.

When we get a new employee or a new computer we want, of course, to have the _exact_ same packages we use on all the other development machines and the ones in production. Otherwise we cannot ensure that everything is working the same way on our development machines as on the production machines.

Hey, just adding a new production machine we could end up with different packages before. Pure nightmare.

-- 
/Jacob Carlborg

On Wednesday, 11 September 2013 at 15:01:37 UTC, Jacob Carlborg wrote:
> I'm thinking this type of package manager should be a development tool as well. But there are a lot of development tools that are executables and not just libraries. Think of your documentation generator. Without having looked at it I would assume it's an executable. I have myself a tool, DStep, which translate C headers to D modules. This is an executable as well.

Those should be provided as sources and built by dub too. Distributing binary packages requires both package signing and reasonable web of trust - something that is not easy to "just implement" from scratch. Otherwise any single malicious package may ruin reputation of the whole system.

On Wednesday, 11 September 2013 at 06:12:41 UTC, Sönke Ludwig wrote:
> Am 10.09.2013 23:04, schrieb Nick Sabalausky:
>> On Tue, 10 Sep 2013 23:01:12 +0200
>> "Brad Anderson" <eco@gnuk.net> wrote:
>>>
>>> I vote yes but only if Sönke feels it is ready. I suspect he has
>>> a few things he'll probably want done before this happens (the
>>> potential switch from JSON to SDL comes to mind).
>>
>> I assume that would be a backwards-compatible change. Make SDL the
>> preferred, but keep JSON in service.
>>
>
> Exactly. Given enough interest, we could also make a more formal review process for a future SDL based format to ensure a maximum chance of a solid, forward compatible format.
>
> Of my former list mentioned in the VisualD thread [1], only package signing is really still missing, but that's probably not mission critical for now. The command line build process also needs to be improved one way or another at some point (mostly caching pre-compiled dependencies), but that also isn't really a strong argument anymore.
>
> All in all I'd say that the things that are in the package format [2] by now form a pretty solid basis to move forward without worrying too much about future breakage.
>
> [1]: http://forum.rejectedsoftware.com/groups/rejectedsoftware.vibed/post/79
> [2]: http://code.dlang.org/package-format

Perhaps there should be a version number in the package format so it can be changed when needed without a lot of headache and breakage.

Am 11.09.2013 17:01, schrieb Jacob Carlborg:
>> Why not _make_ a tag? But uploading zipped packages (or better
>> specifying an external link) could be added as an alternative without
>> much effort.
>
> I'm not referring to uploading zip packages. Say I have ten tags but I
> only want dub to know about five of them. I want to manually say "make
> the tag v0.0.1 available".

It will only look at version tags of the form vA.B.C(postfix) any reason to hide one of those? It could be added as a feature to the registry, but is there a compelling use case to warrant the costs?

>
>> This is something that may still need some adjustments, but after all
>> it's just a default when there is a "source/app.d" or
>> "source/<packname>.d" file - an explicit "targetType": "xxx" will fix it
>> when it's off.
>
> Yes, but I still don't like the default. To avoid misunderstandings,
> will the dub, by default _both_ build an executable _and_ a library. Or
> will it build an executable _or_ a library?

"or", you can choose which using "dub --config=library" or "dub --config=application" (the default).

>
>> Note that there is "dub describe" to extract the information relevant
>> for actual building plus there is "dub generate xxx", so nothing is in
>> the way of using a separate tool for building. But having integrated
>> building IMO is highly convenient and keeping the build description in a
>> standardized format is one of the key selling points.
>
> Hmm ok. I would go with two separate tools that are well integrated with
> each other.
>
> BTW, neither "dub generate rdmd" or "dub generate build" seems to be
> working.

Works for me, but master was broken for some hours. Maybe you caught a bad version?


>> This could also be implemented as a (semi-)automatic function along
>> the lines of "dub lock-versions" and  "dub lock-versions --upgrade".
>
> I think this should be the default. I don't want my package to break at
> random just because there's a newer version available of an indirect
> dependency.
>
> I think that a key feature of a package manager should be that any time
> in time you're installing a package of a given version should result in
> the _exact_ same packages, including indirect packages.

I just don't know if that can be generalized to everyone or at least the majority of users. Maybe this is a good topic for starting a wider discussion/poll.

>
>> On the other hand it's difficult to make a general statement that this
>> is always the best way as this may for example prevent important
>> security fixes to get incorporated unless the main package maintainer
>> releases an explicit update...
>
> Any update, may it only be security fixes, may break a build. No,
> semantic versions doesn't help here. Locking down by default is the safest.

If used correctly, by definition, it _does_ help. Reality, especially in the fast moving D environment, may be different, though. However it seems to work quite well in the C world.

>
> If the package maintainer doesn't release an update how will you get
> security updates?

By upgrading and getting updates of indirect dependencies, even if the main package wasn't updated.

>
> It's better if there's an explicit way to update an indirect dependency.
>
> I would rather add "bar": "==0.0.2" on one or two packages to force an
> update of an indirect dependency rather then set "bar:" "==0.0.1" on
> _all_ indirect dependency. This can be quite a large tree if there are
> many dependencies.
>
> At work we have a Ruby on Rails project. It currently uses 160 packages,
> including Rails and all its sub components. It was pure hell before
> "bundler" was released which locks down all package dependencies as I've
> described here. Counting all indirect dependencies as well I get over
> 413 packages. Would you like to specify the exact version of all these?
>
> Saying that "bundler" was a salvation for the Ruby community is an
> understatement.
>
> When we get a new employee or a new computer we want, of course, to have
> the _exact_ same packages we use on all the other development machines
> and the ones in production. Otherwise we cannot ensure that everything
> is working the same way on our development machines as on the production
> machines.
>
> Hey, just adding a new production machine we could end up with different
> packages before. Pure nightmare.
>

Note that I definitely don't oppose to the idea of integrating such a mechanism -- I do see the value. The question is if it should be the default or not (Bundler is opt-in after all), taking into account for example how well it interacts with branches and different use cases. I'm quite open there, but I first have to play that through with all of my stuff before I can voice an opinion.

On Wednesday, 11 September 2013 at 14:11:11 UTC, John Colvin wrote:
> Why not YAML? It's cleaner than JSON and is very widely known.

YAML is nice but can be surprisingly tricky to write by hand sometimes (especially for people not used to significant whitespace).

Here's the discussion about JSON vs. YAML vs. SDL on the dub forum:

http://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/2/

On Wednesday, 11 September 2013 at 15:01:37 UTC, Jacob Carlborg wrote:
> On 2013-09-11 13:30, Sönke Ludwig wrote:
>
>> Right now it is a pure development tool. It would be very nice to have
>> end user installs somehow supported (either by directly installing
>> application packages or by generating OS specific packages such as DEB
>> or RPM). But since this enters a highly operating specific area and goes
>> into direct competition with the OS package manager, I think it needs a
>> lot of thought and caution to be generally useful and not possibly do
>> more harm than good in the end. But yes, it should be a primary goal in
>> my opinion, too.
>
> I'm thinking this type of package manager should be a development tool as well. But there are a lot of development tools that are executables and not just libraries. Think of your documentation generator. Without having looked at it I would assume it's an executable. I have myself a tool, DStep, which translate C headers to D modules. This is an executable as well.


I have to completely disagree with you here.  Where would it end?  Would it install vim for me?  Install the Java VM so it could run some Java tool?  The level of effort needed to add this functionality—which would duplicate dozens of existing package management systems that already do this job well—is not worth it (and that's before you even consider the amount of effort that would be needed to maintain such a system).

dub handles source libraries and handles them well (for such a young project).  There is no need to make it do everything and I think trying to would be detrimental.

On 10/09/13 22:48, Andrei Alexandrescu wrote:
> We've been experimenting with http://code.dlang.org for a while and things are
> going well. In particular Sönke has been very active about maintaining and
> improving it, which brings further confidence in the future of the project.
>
> We're considering making dub the official package manager for D. What do you all
> think?

I have no direct experience of using dub, so take this as the point of view of someone coming to it fresh and with a "What's this all about?" mindset.

Anyway, these are the general things I feel:

   * Having an official package manager (and hence package repo) is a great way
     to get round the current state of confusion about D libraries, with
     orphaned projects in places like dsource.org etc.

     In fact I think having a well-maintained one-stop shop to find actively
     maintained/developed D libraries is _more_ important than having a nice
     tool to install them -- but a good tool is the icing on the cake, because
     it makes it easy to just grab stuff.

   * It doesn't matter that much if the tool isn't perfect right now.  Obviously
     if Sönke has objections that matters, but otherwise, moving forward is a
     good motivation for everyone to join in and fix any flaws.

   * Currently I think the documentation for dub is inadequate, at least that
     which was trivial for me to find.  I guess that extra stuff might be
     available post-install via --help or man-pages, but that material needs
     to be online to read _before_ downloading and installing.  Essentially I
     wasn't able to get any meaningful impression of what the tool would
     actually be like to use.

   * Specific things I'd like to have from a docs point of view: more detailed
     descriptions of how to use dub as a consumer of packages, how to handle
     both user-local and system-wide install of dub packages, a clear
     description of _where_ it puts user- and system-wide installed packages
     (and how to customize that), how to use packages once installed (and how
     to do so with different D compilers), how to update and uninstall packages.

     If anyone wants to enlighten me in this thread, I'd really appreciate it
     (actually some of these questions have already been answered in other
     threads), but I also think that info needs to be on the dub website. :-)

At the moment I'm a bit committed to other things but I might have a go at turning Dgraph into a dub package just to see how it goes.  Should be a nice usability test. :-)

Best wishes,

    -- Joe

On Wednesday, September 11, 2013 16:11:10 John Colvin wrote:
> Why not YAML? It's cleaner than JSON and is very widely known.

YAML is just plain evil. It doesn't ignore whitespace.

- Jonathan M Davis