On 2011-03-18 18:04, Chris Manning wrote:
> On 17/03/2011 22:49, Jason E. Aten wrote:
>> Somewhat tongue in cheek, we could call it dabal.
>>
>> As in, "get on dabal!" :-)
>
> If D gets accepted for Google Summer of Code, I think this would be a
> great idea for a project and I would be interested in implementing it as
> a student. Although, it does seem overly ambitious so maybe only some of
> this could be for the gsoc (and if I do this It'd be great to carry on
> working on it anyway).
>
> What does everybody think about this? Should I draw up a proposal of
> some kind?
>
> Chris

I've been thinking for quite some time to build a package management system for D, lets call it dpac as an example. This is the ideas that I have:

Basically copy how RubyGems works.
Use Ruby as a DSL for dpacsepc files which is used to create to create the dpac file. This is an example for how a file used to build a package could look like:

name "Foo Bar"
summary "This is the Foo Bar package"
version "1.0.0"
type :lib
author "Jacob Carlborg"
files ["lib.d"] # list of the files in the package
build :make # other options could be :dsss :cmake and so on
dversion 2 # D1 or D2

Build a dpac package out of the dpacspec file:

dpac foobar.dpacspec

Publish the package:

$ dpac publish foobar

Install the package:

$ dpac install foobar

A dpac package would just be a zip file (or some other type of archive) containing all the necessary files to build the package and a file with meta data.

All packages would be manged on a basic RESTful web server. Using GET to download a package and POST to publish a package.

I'm working on a build system for D that I was thinking about to integrate with the package management system. Then the build system could track the files needed to build the package, making the "files" attribute optional.

I also has a tool called DVM, https://bitbucket.org/doob/dvm , used for installing and managing different versions of D compilers. I was thinking about integrating DVM with the package management system to be able to install different packages for different compilers.

-- 
/Jacob Carlborg

On 19/03/2011 14:36, Jacob Carlborg wrote:
> On 2011-03-18 18:04, Chris Manning wrote:
>> On 17/03/2011 22:49, Jason E. Aten wrote:
>>> Somewhat tongue in cheek, we could call it dabal.
>>>
>>> As in, "get on dabal!" :-)
>>
>> If D gets accepted for Google Summer of Code, I think this would be a
>> great idea for a project and I would be interested in implementing it as
>> a student. Although, it does seem overly ambitious so maybe only some of
>> this could be for the gsoc (and if I do this It'd be great to carry on
>> working on it anyway).
>>
>> What does everybody think about this? Should I draw up a proposal of
>> some kind?
>>
>> Chris
>
> I've been thinking for quite some time to build a package management
> system for D, lets call it dpac as an example. This is the ideas that I
> have:
>
> Basically copy how RubyGems works.
> Use Ruby as a DSL for dpacsepc files which is used to create to create
> the dpac file. This is an example for how a file used to build a package
> could look like:
>
> name "Foo Bar"
> summary "This is the Foo Bar package"
> version "1.0.0"
> type :lib
> author "Jacob Carlborg"
> files ["lib.d"] # list of the files in the package
> build :make # other options could be :dsss :cmake and so on
> dversion 2 # D1 or D2
>
> Build a dpac package out of the dpacspec file:
>
> dpac foobar.dpacspec
>
> Publish the package:
>
> $ dpac publish foobar
>
> Install the package:
>
> $ dpac install foobar
>
> A dpac package would just be a zip file (or some other type of archive)
> containing all the necessary files to build the package and a file with
> meta data.
>
> All packages would be manged on a basic RESTful web server. Using GET to
> download a package and POST to publish a package.
>
> I'm working on a build system for D that I was thinking about to
> integrate with the package management system. Then the build system
> could track the files needed to build the package, making the "files"
> attribute optional.
>
> I also has a tool called DVM, https://bitbucket.org/doob/dvm , used for
> installing and managing different versions of D compilers. I was
> thinking about integrating DVM with the package management system to be
> able to install different packages for different compilers.
>
I was thinking about something more similar to portage's ebuild system or arch's AUR. This would mean that the sources could be stored anywhere and just the info to build a package would be stored in a centralised location.

For publishing these build "scripts", again, AUR's system comes to mind, although perhaps something more controll(ed/able). Also, gentoo's sunrise overlay.

Of course, this comes with the downside of the user having to compile the packages on their end.

Chris

On 2011-03-19 17:05, Chris Manning wrote:
> On 19/03/2011 14:36, Jacob Carlborg wrote:
>> On 2011-03-18 18:04, Chris Manning wrote:
>>> On 17/03/2011 22:49, Jason E. Aten wrote:
>>>> Somewhat tongue in cheek, we could call it dabal.
>>>>
>>>> As in, "get on dabal!" :-)
>>>
>>> If D gets accepted for Google Summer of Code, I think this would be a
>>> great idea for a project and I would be interested in implementing it as
>>> a student. Although, it does seem overly ambitious so maybe only some of
>>> this could be for the gsoc (and if I do this It'd be great to carry on
>>> working on it anyway).
>>>
>>> What does everybody think about this? Should I draw up a proposal of
>>> some kind?
>>>
>>> Chris
>>
>> I've been thinking for quite some time to build a package management
>> system for D, lets call it dpac as an example. This is the ideas that I
>> have:
>>
>> Basically copy how RubyGems works.
>> Use Ruby as a DSL for dpacsepc files which is used to create to create
>> the dpac file. This is an example for how a file used to build a package
>> could look like:
>>
>> name "Foo Bar"
>> summary "This is the Foo Bar package"
>> version "1.0.0"
>> type :lib
>> author "Jacob Carlborg"
>> files ["lib.d"] # list of the files in the package
>> build :make # other options could be :dsss :cmake and so on
>> dversion 2 # D1 or D2
>>
>> Build a dpac package out of the dpacspec file:
>>
>> dpac foobar.dpacspec
>>
>> Publish the package:
>>
>> $ dpac publish foobar
>>
>> Install the package:
>>
>> $ dpac install foobar
>>
>> A dpac package would just be a zip file (or some other type of archive)
>> containing all the necessary files to build the package and a file with
>> meta data.
>>
>> All packages would be manged on a basic RESTful web server. Using GET to
>> download a package and POST to publish a package.
>>
>> I'm working on a build system for D that I was thinking about to
>> integrate with the package management system. Then the build system
>> could track the files needed to build the package, making the "files"
>> attribute optional.
>>
>> I also has a tool called DVM, https://bitbucket.org/doob/dvm , used for
>> installing and managing different versions of D compilers. I was
>> thinking about integrating DVM with the package management system to be
>> able to install different packages for different compilers.
>>
> I was thinking about something more similar to portage's ebuild system
> or arch's AUR. This would mean that the sources could be stored anywhere
> and just the info to build a package would be stored in a centralised
> location.
>
> For publishing these build "scripts", again, AUR's system comes to mind,
> although perhaps something more controll(ed/able). Also, gentoo's
> sunrise overlay.
>
> Of course, this comes with the downside of the user having to compile
> the packages on their end.
>
> Chris

Oh, I forgot a part. Just as with RubyGems you would be able to install a package from a repository (git, mercurial or something else) as long as it contains the file containing the package specification file (dpacspec).

Of course you could manually downloading a package and installing it.

-- 
/Jacob Carlborg

> On Sat, 19 Mar 2011 12:19:58 +0100, Jacob Carlborg wrote:
>> What are people's experiences with the various options for build systems with D?
> 
> It's not very easy to make an incremental build system for D because of several reasons. Some are due to how the language works and some are due to how DMD works:
> 
> * DMD doesn't output all data in all the object files - This can perhaps be solved by compiling with the -lib switch
> 
> * When you change one D file you need to recompile ALL files that depend on the changed file. To compare with C/C++ which has source and header files you only need to recompile the source file if you change it
> 
> * DMD doesn't keep the fully qualified module name when naming object files resulting in foo.bar will conflict with bar.bar. Issue 3541.

[The above is from the packaging system discussion in the "a cabal for D?" thread; here I am branching this to a new topic because I'd like anyone interested in incremental build processes to notice and contribute if they have input.]

That is an interesting observation, Jacob. Thank you for pointing that out.

Is there anything else (open question anyone) that would prevent D projects from doing incremental builds?  Lack of support for incremental builds is a show stopper.  Or in this case, the show would never get funded to begin with.

Or to ask it another way, what would it take to get incremental builds?

Even AutoHotkey has a system of finding and installing libraries. It can run example code without installing (well, it is a scripting language), show the source, documentation and some other things.

Snapshot: http://i.imgur.com/Uv1Gr.jpg

On 17/03/11 21.44, Jason E. Aten wrote:
> Please correct me if I'm wrong, but I observe that there doesn't appear
> to be a package management system / standard repository for D libraries.
> Or is there?
>
> I'm talking about something as easy to use as R's CRAN,
>> install.packages("rforest")
>
> or cpan for perl, ctan for latex, dpgk/apt for debian, cabal for Haskell/
> Hackage, etc.
>
> If there's not a commonly utilized one currently, perhaps we could
> "borrow" cabal, with a trivial port.  cabal is Haskell's package manager.
>
> Not only does having a standard package install system facilitate
> adoption, it greatly facilitates code sharing and library maturation.

It seems quite a few people are interested in such a tool. I suggest that those who are interested in the subject form some kind of SIG and creates a page on http://prowiki.org/wiki4d with ideas, insights, requirements that the rest of the community can follow. I think discussions still need to be done in this newsgroup though.

This means that someone will actually need to implement something or modify an existing solution at some point for it to make sense. But initially we need to settle on something that "everybody" would be happy with and would support as the standard tool if possible.

I'm willing to throw in some time implementing it when the curl stuff that I'm currently working on is done. Hopefully others can put some work in this tool as well to get things rolling.

Anyone think this is a good idea?

Should I create a SIG page?

/Jonas

On 2011-03-22 12:03, Jonas Drewsen wrote:
> On 17/03/11 21.44, Jason E. Aten wrote:
>> Please correct me if I'm wrong, but I observe that there doesn't appear
>> to be a package management system / standard repository for D libraries.
>> Or is there?
>>
>> I'm talking about something as easy to use as R's CRAN,
>>> install.packages("rforest")
>>
>> or cpan for perl, ctan for latex, dpgk/apt for debian, cabal for Haskell/
>> Hackage, etc.
>>
>> If there's not a commonly utilized one currently, perhaps we could
>> "borrow" cabal, with a trivial port. cabal is Haskell's package manager.
>>
>> Not only does having a standard package install system facilitate
>> adoption, it greatly facilitates code sharing and library maturation.
>
> It seems quite a few people are interested in such a tool. I suggest
> that those who are interested in the subject form some kind of SIG and
> creates a page on http://prowiki.org/wiki4d with ideas, insights,
> requirements that the rest of the community can follow. I think
> discussions still need to be done in this newsgroup though.
>
> This means that someone will actually need to implement something or
> modify an existing solution at some point for it to make sense. But
> initially we need to settle on something that "everybody" would be happy
> with and would support as the standard tool if possible.
>
> I'm willing to throw in some time implementing it when the curl stuff
> that I'm currently working on is done. Hopefully others can put some
> work in this tool as well to get things rolling.
>
> Anyone think this is a good idea?
>
> Should I create a SIG page?
>
> /Jonas

I think it's a good idea. I'm planning to start on a tool like this when I finished the std.net.isemail module.

-- 
/Jacob Carlborg

Hi Russel. (BTW, I was the guy a few weeks ago at the London Java Community Meetup pub meet that Barry Cranford mentioned in an SMS. Sorry you couldn't make it, maybe next time. :p )

On 18/03/2011 08:52, Russel Winder wrote:
>
> Note that every language-specific package manager conflicts directly
> with every operating system package manager.  Thus RubyGems, CPAN,
> Cabal, Maven, Go, etc. conflicts with the package management of Debian,
> Fedora, SUSE, FreeBSD, MacPorts, etc.

Which is the way it should be. (Note that "conflicts" is likely not the best word, rather "ignores" or "does not make use or interact with" is a better term.) I'm well convinced of this because these two - OS and language package manager - /usually/ serve very distinct purposes. The OS PMs are usually focused on installing applications, not building them (yes, there are exceptions, like Gentoo). In particular the OS PMs most likely won't support functionality you would want from a language PM. (say for example that you would want to specify that your library has a dependency on any other library that exports a D package named org.foo.bar?)
But we don't even need to go there, just the fact that OS PMs are not cross-platform is reason enough for them to be unsuitable to be used (on their own) as language PM. There needs to be a language PM abstracted from OS/platform.
Perhaps some additional (and optional) integration with the OS PM could be available in the language PM. But the language PM would still be "on top". And even so I'm not convinced this integrations makes much sense. Maybe for dynamic and interpreted languages like Ruby and Python, where built application and sources are often the same... here perhaps the term "conflict" as mentioned above is more appropriate, the distinction between end-user and developer-user is less clear. But none of this applies to D, being a compiled language.

-- 
Bruno Medeiros - Software Engineer

On 18/03/2011 18:42, Russel Winder wrote:
> On Fri, 2011-03-18 at 10:41 -0400, Jesse Phillips wrote:
> [ . . . ]
>> Otherwise I would be for using the native packaging system, but it
>> isn't easy for the contributor, the maintainer of the server, or the
>> one building a system that works with all of them.
>
> The problem is that the system administrator wants to use just the OS
> packaging system to make things easy for them to maintain the system.
> The user wants to be able to install stuff easily and may not have
> sufficient permissions to actually use the OS packaging system.  Sys
> admins don't want to have to learn N packages to deal with N languages,
> the use probably only works in a couple of languages and so doesn't care
> if they have to learn two different packagin systems so as to get stuff
> done.
>

I agree sys admins should not have to learn language specific package managers just because the user wants to use one. Rather what should happen is that language PMs should be able to be "installed" on any user-specified location, and installed multiple times. Just like a compiler should. They should not be designed to work only as an OS "singleton", so to speak.

> This is not an easy issue.  I just find the knee-jerk reaction of "we
> have this new language therefore we must have a brand new (build system|
> packaging system|shell|implementation of every comms protocol|new user
> interface library|operating system)" leads to too many distractions from
> getting stuff done using the good tools that are already available.
>

What I think makes sense for a D package manager is to integrate with existing *build* systems (make, Cmake, Ant, Maven, SCons, etc., etc., whatever might be appropriate). But definitely not with OS package managers.

-- 
Bruno Medeiros - Software Engineer

On 19/03/2011 14:36, Jacob Carlborg wrote:
> On 2011-03-18 18:04, Chris Manning wrote:
>> On 17/03/2011 22:49, Jason E. Aten wrote:
>>> Somewhat tongue in cheek, we could call it dabal.
>>>
>>> As in, "get on dabal!" :-)
>>
>> If D gets accepted for Google Summer of Code, I think this would be a
>> great idea for a project and I would be interested in implementing it as
>> a student. Although, it does seem overly ambitious so maybe only some of
>> this could be for the gsoc (and if I do this It'd be great to carry on
>> working on it anyway).
>>
>> What does everybody think about this? Should I draw up a proposal of
>> some kind?
>>
>> Chris
>
> I've been thinking for quite some time to build a package management
> system for D, lets call it dpac as an example. This is the ideas that I
> have:
>
> Basically copy how RubyGems works.
> Use Ruby as a DSL for dpacsepc files which is used to create to create
> the dpac file. This is an example for how a file used to build a package
> could look like:
>
> name "Foo Bar"
> summary "This is the Foo Bar package"
> version "1.0.0"
> type :lib
> author "Jacob Carlborg"
> files ["lib.d"] # list of the files in the package
> build :make # other options could be :dsss :cmake and so on
> dversion 2 # D1 or D2
>

Copying Rubygems would be a bad idea, or at least a very incomplete one. And the example code above shows why, doesn't anyone else see it?
Just look at the line:
build :make # other options could be :dsss :cmake and so on
Basicly, this would be a D package manager that requires another package manager to build D artifacts... kinda not very useful, right? If we skip the issue of building D artifacts (executables, dynamic libraries, static libraries, etc.) we skip one essential aspect of a D package/build management system (if not the main aspect, but that is debatable...). This is why I think just copying/cloning a PM from interpreted languages is not helping us much.

-- 
Bruno Medeiros - Software Engineer