While lying in the bed with fever yesterday (so please excuse any careless mistakes), I was pondering a bit about the current discussions regarding Phobos additions, package management, etc. It occurred to me that there is a central unanswered question, which I think deserves to be broadly discussed right now.

But first, let me start out by describing how I see current situation regarding D2. Leaving aside a few minor things like @property enforcement or the recent suggestions about a new alias syntax, the language is fairly stable and critical bugs in DMD 2 are not frequent enough to make it completely unusable for day-to-day development anymore. Of course, there is still a large way to go for the D toolchain (with the ideal result being a rock-solid self-hosting compiler front-end, usable as a library as well), but in a sense, we are more or less at the end of a certain stage of D2 development.

I think most of you would agree with me if I say that the main goal for D2 right now should be to build a vibrant library ecosystem around the language, to foster adoption in real-world applications. There has been a number of related discussions recently, but as mentioned above, I think there is a central question:

Have we reached the critical mass yet where it makes sense to split the effort in a number of smaller library projects, or are we off better with concentrating on a central, comprehensive standard library (Phobos), considering the current community size?

I do not really have an answer to this question, but here are a few thoughts on the topic, which might also help to make clearer what I mean:

I think that adopting a Boost-like review process for Phobos has certainly been a clever and valuable move, for more than one reason. First, together with the move to Git, it has helped to reinforce the point that D2 and Phobos are open to contributions from everyone, given that they meet certain quality standards. Second, it certainly boosts code quality of further standard library additions, which had been a problem for some parts in the past (at least from my point of view, no offense intended). Third, and this overlaps with another point below, I think that the quality improvements will also help to reduce bit rot, which has traditionally been a problem with D libraries.

But however good a fit this model is for the standard library, I think it is no silver bullet either. There are small, one-off style projects, arising from a central need, where the amount of time needed to get the code through the whole review process is prohibitive – even if the code quality was high enough –, but the result is still usable for the wide public. Common examples for this would be low-level wrappers for C libraries, although they don't really qualify for inclusion into Phobos for other reasons (often, another wrapper layer is needed to be usable with common D idioms). Also, people new to the language might be scared away by the mere thought of contributing to a standard library. How to make sure that these libraries are not forgotten? Maybe a central package system with SCM (Git, …) integration can help here?

And, which brings me to the next point, how to fight the unfavorable outcome of having a huge inscrutable pile of half-finished bit-rotten code, a problem that DSource is currently experiencing? A central, well-maintained standard library effort with a wider scope could certainly help to reduce this problem, at least from the (D) user side, but on the other hand, larger amounts of code de facto becoming unmaintained would be a problem for it as well.

Should we build something like a staging area, an incubator for community contributions not taken yet through formal review, but of interest for a wider audience? What about the etc.* package – would it be an option to expand it into such an incubation area? If not, what should it evolve into – a collection of C-level library bindings (see the recent discussion on SQLite bindings started by David Simcha)? Who will take care of the maintenance duties?

Looking forward to a stimulating discussion,
David

David Nadlinger Wrote:

> Should we build something like a staging area, an incubator for community contributions not taken yet through formal review, but of interest for a wider audience? What about the etc.* package – would it be an option to expand it into such an incubation area? If not, what should it evolve into – a collection of C-level library bindings (see the recent discussion on SQLite bindings started by David Simcha)? Who will take care of the maintenance duties?

While growing the standard library is great, or developing a packaging system for D code is all great stuff. We really need a good central search repository. Something that gives maintenance,stability, and is search able by category.

That said, their really isn't a great number of mature, ready to use libraries yet. So why not just use what we already have, and keep it updated with these candidate libraries:

http://www.prowiki.org/wiki4d/wiki.cgi?DevelopmentWithD/Libraries

On Saturday 19 March 2011 07:43:37 David Nadlinger wrote:
> While lying in the bed with fever yesterday (so please excuse any careless mistakes), I was pondering a bit about the current discussions regarding Phobos additions, package management, etc. It occurred to me that there is a central unanswered question, which I think deserves to be broadly discussed right now.
> 
> But first, let me start out by describing how I see current situation regarding D2. Leaving aside a few minor things like @property enforcement or the recent suggestions about a new alias syntax, the language is fairly stable and critical bugs in DMD 2 are not frequent enough to make it completely unusable for day-to-day development anymore. Of course, there is still a large way to go for the D toolchain (with the ideal result being a rock-solid self-hosting compiler front-end, usable as a library as well), but in a sense, we are more or less at the end of a certain stage of D2 development.
> 
> I think most of you would agree with me if I say that the main goal for D2 right now should be to build a vibrant library ecosystem around the language, to foster adoption in real-world applications. There has been a number of related discussions recently, but as mentioned above, I think there is a central question:
> 
> Have we reached the critical mass yet where it makes sense to split the effort in a number of smaller library projects, or are we off better with concentrating on a central, comprehensive standard library (Phobos), considering the current community size?
> 
> I do not really have an answer to this question, but here are a few thoughts on the topic, which might also help to make clearer what I mean:
> 
> I think that adopting a Boost-like review process for Phobos has certainly been a clever and valuable move, for more than one reason. First, together with the move to Git, it has helped to reinforce the point that D2 and Phobos are open to contributions from everyone, given that they meet certain quality standards. Second, it certainly boosts code quality of further standard library additions, which had been a problem for some parts in the past (at least from my point of view, no offense intended). Third, and this overlaps with another point below, I think that the quality improvements will also help to reduce bit rot, which has traditionally been a problem with D libraries.
> 
> But however good a fit this model is for the standard library, I think it is no silver bullet either. There are small, one-off style projects, arising from a central need, where the amount of time needed to get the code through the whole review process is prohibitive – even if the code quality was high enough –, but the result is still usable for the wide public. Common examples for this would be low-level wrappers for C libraries, although they don't really qualify for inclusion into Phobos for other reasons (often, another wrapper layer is needed to be usable with common D idioms). Also, people new to the language might be scared away by the mere thought of contributing to a standard library. How to make sure that these libraries are not forgotten? Maybe a central package system with SCM (Git, …) integration can help here?
> 
> And, which brings me to the next point, how to fight the unfavorable outcome of having a huge inscrutable pile of half-finished bit-rotten code, a problem that DSource is currently experiencing? A central, well-maintained standard library effort with a wider scope could certainly help to reduce this problem, at least from the (D) user side, but on the other hand, larger amounts of code de facto becoming unmaintained would be a problem for it as well.
> 
> Should we build something like a staging area, an incubator for community contributions not taken yet through formal review, but of interest for a wider audience? What about the etc.* package – would it be an option to expand it into such an incubation area? If not, what should it evolve into – a collection of C-level library bindings (see the recent discussion on SQLite bindings started by David Simcha)? Who will take care of the maintenance duties?
> 
> Looking forward to a stimulating discussion,
> David

There has been some discussion in the past of creating an incubator project of sorts where code which may or may not make it into Phobos can be put so that it can develop and evolve with people actually using it - maybe even using it heavily - before it tried to get into Phobos. Once a library was considered mature enough, it could go through the Phobos review process and attempt to get into the standard library. If it succeeded, then, in theory, we'd have a well- used and well-tested library added to Phobos. If it failed, it would still be around for people to use, and it could continue to be used and evolve - either to make a later attempt at inclusion in Phobos or to simply live as a 3rd party library that folks find useful but isn't appropriate for inclusion in the standard library for one reason or another.

Exactly what the best way to do all this would be, I don't know, but it would need to be far better manage than the current situation with dsource. The current activity level of a project would need to clear, as well as how complete and stable it's considered. DSource is probably the best place to host such a project, but obviously dsource would have to be cleaned up first, and it would need someone who was willing to spearhead the effort and manage it. Otherwise, we'll just end up with a situation similar to what dsource is now.

Really, the problem is that someone needs to take the initiative on this. They need to work on setting it up and supporting the ecosystem which would result in a group of such projects. Good ideas tend to be presented around here and then go nowhere, because no one actually takes the initiative to do them. The "wouldn't this be a good idea?" tactic doesn't tend to get very far, even if everyone agrees, simply because someone has to put in the time and effort to do it, and while people may think that it's a good idea, there are only so many people working on Phobos and other D-related stuff, and there's a lot to be done, and everyone has something that they'd like to see done, and _that_ is what they're generally working on.

So, I definitely think that an incubation project is a great idea. It has been proposed before. However, unless someone steps up to the plate and takes on setting it up and organizing it, it's not going to happen.

- Jonathan M Davis

On 03/19/2011 06:12 PM, Jonathan M Davis wrote:
> Really, the problem is that someone needs to take the initiative on this. They
> need to work on setting it up and supporting the ecosystem which would result in
> a group of such projects. Good ideas tend to be presented around here and then
> go nowhere, because no one actually takes the initiative to do them. The
> "wouldn't this be a good idea?" tactic doesn't tend to get very far, even if
> everyone agrees, simply because someone has to put in the time and effort to do
> it, and while people may think that it's a good idea, there are only so many
> people working on Phobos and other D-related stuff, and there's a lot to be done,
> and everyone has something that they'd like to see done, and _that_ is what
> they're generally working on.

Words of the wise.

It's like hearing a woman for the laundry list of stuff she'd like in a man and compare (and contrast) that with the kind of man she does respond to. The "desirable stuff" list is built using a rational response, whereas in reality it's the emotional response that guides decisions. We hackers seem to use a similar process when deciding what to work on :o).

That's in particular why we all know e.g. networking is a necessity, but instead we work on various other stuff. The person who has an emotional positive response for networking code hasn't showed up yet - or just has in the person of Jonas.


Andrei

On Sat, Mar 19, 2011 at 6:12 PM, Jonathan M Davis <jmdavisProg@gmx.com>wrote:

>
> Really, the problem is that someone needs to take the initiative on this.
> They
> need to work on setting it up and supporting the ecosystem which would
> result in
> a group of such projects. Good ideas tend to be presented around here and
> then
> go nowhere, because no one actually takes the initiative to do them. The
> "wouldn't this be a good idea?" tactic doesn't tend to get very far, even
> if
> everyone agrees, simply because someone has to put in the time and effort
> to do
> it, and while people may think that it's a good idea, there are only so
> many
> people working on Phobos and other D-related stuff, and there's a lot to be
> done,
> and everyone has something that they'd like to see done, and _that_ is what
> they're generally working on.
>
>
> - Jonathan M Davis
>

It's not that someone needs to take the initiative, it's just that there aren't that many D developers.  I hope things improve once GDC is officially part of GCC and D becomes available on all GNU/Linux OSs.  Another thing that might need to happen is for the D project to join The Software Freedome Conservancy (SFC) or form its own non-profit 501(c)(3) organization, similar to Python Software Foundation.

But that's just and I could be wrong.

> On Sat, Mar 19, 2011 at 6:12 PM, Jonathan M Davis
<jmdavisProg@gmx.com>wrote:
> > Really, the problem is that someone needs to take the initiative on this.
> > They
> > need to work on setting it up and supporting the ecosystem which would
> > result in
> > a group of such projects. Good ideas tend to be presented around here and
> > then
> > go nowhere, because no one actually takes the initiative to do them. The
> > "wouldn't this be a good idea?" tactic doesn't tend to get very far, even
> > if
> > everyone agrees, simply because someone has to put in the time and effort
> > to do
> > it, and while people may think that it's a good idea, there are only so
> > many
> > people working on Phobos and other D-related stuff, and there's a lot to
> > be done,
> > and everyone has something that they'd like to see done, and _that_ is
> > what they're generally working on.
> > 
> > 
> > - Jonathan M Davis
> 
> It's not that someone needs to take the initiative, it's just that there aren't that many D developers.  I hope things improve once GDC is officially part of GCC and D becomes available on all GNU/Linux OSs. Another thing that might need to happen is for the D project to join The Software Freedome Conservancy (SFC) or form its own non-profit 501(c)(3) organization, similar to Python Software Foundation.
> 
> But that's just and I could be wrong.

It's both. Without any developers, there obviously won't be any code. However, until someone takes the initiative and sets up a proper place and framework for projects to be posted to with the idea that it's an incubator for possible additions to Phobos or for major 3rd party D projects, then there's no place for those developers to post their stuff. Right now, such stuff would either be posted on dsource and be lost in all of the cruft sitting there, or it would be posted on someplace like github where there's no real connection between any of the projects. A proper incubator site/project would be _the_ place to go looking for D projects, and it would be properly managed so that the state of each project was clear and dead/inactive projects weren't in the way (be it because they're removed or put in an area where such projects go and don't get in the way).

So, we need someone to take the initiative to set up a proper incubator site/project for D projects, and then we need developers to actually write projects/libraries and post them there.

- Jonathan M Davis

On Sun, Mar 20, 2011 at 2:41 AM, Jonathan M Davis <jmdavisProg@gmx.com>wrote:
>
> It's both. Without any developers, there obviously won't be any code.
> However,
> until someone takes the initiative and sets up a proper place and framework
> for projects to be posted to with the idea that it's an incubator for
> possible
> additions to Phobos or for major 3rd party D projects, then there's no
> place
> for those developers to post their stuff. Right now, such stuff would
> either
> be posted on dsource and be lost in all of the cruft sitting there, or it
> would be posted on someplace like github where there's no real connection
> between any of the projects. A proper incubator site/project would be _the_
> place to go looking for D projects, and it would be properly managed so
> that
> the state of each project was clear and dead/inactive projects weren't in
> the
> way (be it because they're removed or put in an area where such projects go
> and don't get in the way).
>
> So, we need someone to take the initiative to set up a proper incubator site/project for D projects, and then we need developers to actually write projects/libraries and post them there.
>
> - Jonathan M Davis
>

I'm still not sure what people mean by "incubator".  If you are talking about dsource2, then no, it's a bad idea.  People should be able to use whatever site, whatever project management software, and whatever tool they want for their code.  Besides, a lot of momentum is needed to get something like RubyForge to work, and D doesn't have that momentum, yet.

As for D and Phobos, I think what you are trying to describe is a place where interested developers could quickly find out _what_ needs to be done and _how_ to get involved.  That place should be www.digitalmars.com/d but it's not; everything from its 80s design style to lack of important information.  Just compare http://docs.python.org/devguide/ to it.  First impressions count, and I'm not just talking about looks here.

So, yes, "build it and they will come."

> On Sun, Mar 20, 2011 at 2:41 AM, Jonathan M Davis
<jmdavisProg@gmx.com>wrote:
> > It's both. Without any developers, there obviously won't be any code.
> > However,
> > until someone takes the initiative and sets up a proper place and
> > framework for projects to be posted to with the idea that it's an
> > incubator for possible
> > additions to Phobos or for major 3rd party D projects, then there's no
> > place
> > for those developers to post their stuff. Right now, such stuff would
> > either
> > be posted on dsource and be lost in all of the cruft sitting there, or it
> > would be posted on someplace like github where there's no real connection
> > between any of the projects. A proper incubator site/project would be
> > _the_ place to go looking for D projects, and it would be properly
> > managed so that
> > the state of each project was clear and dead/inactive projects weren't in
> > the
> > way (be it because they're removed or put in an area where such projects
> > go and don't get in the way).
> > 
> > So, we need someone to take the initiative to set up a proper incubator site/project for D projects, and then we need developers to actually write projects/libraries and post them there.
> > 
> > - Jonathan M Davis
> 
> I'm still not sure what people mean by "incubator".  If you are talking about dsource2, then no, it's a bad idea.  People should be able to use whatever site, whatever project management software, and whatever tool they want for their code.  Besides, a lot of momentum is needed to get something like RubyForge to work, and D doesn't have that momentum, yet.

In a way, it _would_ be dsource2. In a way it would be for Phobos what Boost is for the STL. What exactly it is would depend on how it was set up. But the idea is to have a place to post libraries or modules which the community at large might find useful and which might be desirable in the standard library. It would be a place where such stuff could be posted and potentially worked on by the community at large. Such libaries/modules would therefore have the opportunity to used by the community at large and evolve according to that use. Once a library/module within in the incubator project reached the point where it was relatively solid and mature, it might try and get into Phobos. If it didn't make sense to put it in Phobos (for whatever reason), then it could stay in the incubator as a useful 3rd party library.

It's not necessarily the case that the source would be hosted on whatever site was for the incubator project. The main thing is to have a place for such D projects to be posted, publicly visible, and make it easy to search through. Projects which were unmaintained would be removed/archived, and the state of the current projects would be clear. So, you'd have a group of 3rd party libraries which were reasonably-well maintained and which could develop into something which would go into Phobos or which would just be a solid 3rd party library (there's plenty which would be generally useful but too narrow in scope to belong in Phobos).

How that compares to RubyForge, I don't know, since I know nothing about RubyForge. It wouldn't be quite like dsource either, since dsource is just a bunch of poorly grouped projects whose state is totally unclear and often unmaintained. It would be something similar to what Boost is to C++'s STL, but I'm not sure that we want something _that_ strict, since stuff has to be thoroughly reviewed before it gets into Boost, and part of the point of the incubator project is to get stuff in there which could be developed prior to reaching the point that it would need to be thoroughly reviewed (such as when it would try and get into Phobos).

Regardless, no one has taken the time to put together such an incubator project, and exactly what it would be and how it would work would be highly dependent on whoever put it together. It's been proposed before and generally found to be a good idea, but until someone actually does it, it won't exist.

> As for D and Phobos, I think what you are trying to describe is a place where interested developers could quickly find out _what_ needs to be done and _how_ to get involved.  That place should be www.digitalmars.com/d but it's not; everything from its 80s design style to lack of important information.  Just compare http://docs.python.org/devguide/ to it.  First impressions count, and I'm not just talking about looks here.

That would be a separate - albeit related - issue. But it _is_ something that could definitely be improved.

- Jonathan M Davis

On 03/20/2011 08:41 AM, Jonathan M Davis wrote:
> A proper incubator site/project would be _the_
> place to go looking for D projects, and it would be properly managed so that
> the state of each project was clear and dead/inactive projects weren't in the
> way (be it because they're removed or put in an area where such projects go
> and don't get in the way).

I'm doubting about the idea of dead projects, because many seem to equal it to "no recent edit". But most highly used libs of most PLs are "dead" for a long while, according to this criterion, aren't they? They do the job (well or not) and rarely need further edition.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

Am 20.03.2011 11:22, schrieb spir:
> On 03/20/2011 08:41 AM, Jonathan M Davis wrote:
>> A proper incubator site/project would be _the_
>> place to go looking for D projects, and it would be properly managed so that
>> the state of each project was clear and dead/inactive projects weren't in the
>> way (be it because they're removed or put in an area where such projects go
>> and don't get in the way).
> 
> I'm doubting about the idea of dead projects, because many seem to equal it to "no recent edit". But most highly used libs of most PLs are "dead" for a long while, according to this criterion, aren't they? They do the job (well or not) and rarely need further edition.
> 
> Denis

Depends. Most libs at least have updates fixing bugs and security issues.

But for D2 there's another thing: Neither the language nor the standard lib are stable, so a lib that hasn't been maintained for some time probably doesn't even compile anymore - and this won't change in the near future (especially for the standard lib).

Cheers,
- Daniel