Adding new (or replacement) phobos modules without wider testing is not a scalable approach for D.
New modules go from unofficial to official in a single step and are therefore inadequately battle-tested before becoming part of the somewhat ossified environment of a standard library.

Assertions:

1) In the current situation, modules go from independent projects to becoming official parts of phobos at a single point. It's a binary switch.

2) New modules, in the form proposed for submission, are fully read through by a small number of people.

3) New modules, in the form proposed for submission, have often been seriously used by only an even smaller number of people.

4) Breaking changes to phobos are currently undesirable.


Argument:
Due to the combination of 1,2 and 3, we unwittingly introduce bugs and poor design decisions in to the standard library, with all the inflexibility that entails. Due to 4 we then either cannot or do not fix these problems in an optimal fashion, if at all. We can do better.


Necessary Solution:
People need to really use these new modules before they are pulled in to phobos. In particular, the API must be stress tested in a variety of situations.*


Implementation:
In the current situation all the code is public, so people are totally free to try out the proposed modules and test them. They don't and they won't.

I propose that the current review process be redirected to a new target package in the phobos repo, stdx, which would then have a separate review process for inclusion in std.

I would imagine a compulsory waiting period of the order of months, combined with a requirement of evidence that the module has been effective in the real world and that any outstanding problems have been resolved appropriately**. Stdx would make no explicit promises about API stability, but modules would be considered as semi-finished products; Revisions that effectively amount to a significant rewrite (in particular to the API) would require re-submission for initial review.

Stdx would occupy a space between phobos and unaffiliated packages, allowing for subtle (i.e. missed during initial review) but critical problems requiring breakage to be identified BEFORE inclusion in the standard library proper. I believe providing this official stepping stone for new modules will result in significantly wider usage and testing, benefiting phobos, D and it's community.


*API design is *hard*. Knowing what is a good decision ahead of time is a matter of experience.
** The appropriate action may be to not fix it. There are always trade-offs.

P.S.
I am aware that this is not a new idea at all. I thought it was worth presenting with a clean slate.

I liked the proposal to add special `dub` category for those (`dub` needs some categorization anyway). Only drawback is that it actually needs to be implemented instead of simply making a decision :)

On 04/10/13 12:02, John Colvin wrote:
> I propose that the current review process be redirected to a new target package
> in the phobos repo, stdx, which would then have a separate review process for
> inclusion in std.

As you say, this has been proposed before and I think the objection was along the lines that in practice, so many people would just use stdx (because they wanted those features NOW, NOW, NOW!) that you'd have the same backwards compatibility issues arising there -- you'd wind up with the same fear of breaking change and deprecation as we currently have in the standard library, and it would also cause breaking change if stdx.somemodule was approved and moved to std.somemodule.

I remember someone pointed to a similar "experimental" module namespace in Java, which effectively became a standard namespace in this way.

> Stdx would occupy a space between phobos and unaffiliated packages, allowing for
> subtle (i.e. missed during initial review) but critical problems requiring
> breakage to be identified BEFORE inclusion in the standard library proper. I
> believe providing this official stepping stone for new modules will result in
> significantly wider usage and testing, benefiting phobos, D and it's community.

There is an alternative risk -- that people will be more inclined to just throw stuff over the wall into stdx because, hey, it's an experimental/testing area, the whole idea is for people to try out imperfect code and work out what's wrong with it ...

... but then people are kind of locked into that imperfect code, because even if it's imperfect it still provides something that no other library/module does, and they and their code suffer from the breaking changes all the same.

> I am aware that this is not a new idea at all. I thought it was worth presenting
> with a clean slate.

I've presented Devil's Advocate positions above, but of course it was worth re-raising the idea.  Anything that does in practice raise the bar of Phobos quality is very welcome.

On Friday, 4 October 2013 at 10:30:56 UTC, Joseph Rushton Wakeling wrote:
> On 04/10/13 12:02, John Colvin wrote:
>> I propose that the current review process be redirected to a new target package
>> in the phobos repo, stdx, which would then have a separate review process for
>> inclusion in std.
>
> As you say, this has been proposed before and I think the objection was along the lines that in practice, so many people would just use stdx (because they wanted those features NOW, NOW, NOW!) that you'd have the same backwards compatibility issues arising there -- you'd wind up with the same fear of breaking change and deprecation as we currently have in the standard library

The lack of promise of API stability in stdx would have to be very explicit and obvious. The maintainers of phobos would have to remain stalwart in preventing any alteration of this, whether de facto or deliberate.

> and it would also cause breaking change if stdx.somemodule was approved and moved to std.somemodule.

That is the least bad type of breaking change.

> I remember someone pointed to a similar "experimental" module namespace in Java, which effectively became a standard namespace in this way.
>
>> Stdx would occupy a space between phobos and unaffiliated packages, allowing for
>> subtle (i.e. missed during initial review) but critical problems requiring
>> breakage to be identified BEFORE inclusion in the standard library proper. I
>> believe providing this official stepping stone for new modules will result in
>> significantly wider usage and testing, benefiting phobos, D and it's community.
>
> There is an alternative risk -- that people will be more inclined to just throw stuff over the wall into stdx because, hey, it's an experimental/testing area, the whole idea is for people to try out imperfect code and work out what's wrong with it ...

The bar for passing initial review should remain exactly where it is now. Nothing should be allowed in to stdx unless it appears for all intents and purposes to be a finished, high quality module that would - to the best of the evidence available - be fine to add straight to phobos. The purpose of stdx is to gather more evidence before making the commitment.

>> I am aware that this is not a new idea at all. I thought it was worth presenting
>> with a clean slate.
>
> I've presented Devil's Advocate positions above, but of course it was worth re-raising the idea.  Anything that does in practice raise the bar of Phobos quality is very welcome.

On Friday, 4 October 2013 at 10:30:56 UTC, Joseph Rushton Wakeling wrote:
> There is an alternative risk -- that people will be more inclined to just throw stuff over the wall into stdx because, hey, it's an experimental/testing area, the whole idea is for people to try out imperfect code and work out what's wrong with it ...

And to be 100% clear - I meant this dub category to be moderated and only filled after formal review process. We simply need some place for modules which got conceptually approved but can't yet meet API stability and/or implementation quality required for inclusion into Phobos as-is.

The thing is, code is never perfect and as a language keeps evolving your latest tricks will soon become obsolete or deprecated. It happens all the time (Java, Cocoa etc. "use this instead"). If I look back at the code I wrote years ago, it still works, but it's no longer up to current standards (my own as well as the language's). I think including things in the std. library really makes people (like me) use and thereby _test_ them, asking (not so) stupid questions on D.learn etc. If I have learned anything it's that my most ignorant (innocent) questions trigger responses from people who are more experienced, pointing out this or that flaw, telling you tricks and workarounds. D is a language based on practical experience, not on ideology or anything (that's why I like it). So stdx, while it seems to be a good idea, would be a limbo for code, eternally in a state of "not bad, but not quite there yet". And all code is like that. We'll never quite get there.

On 2013-10-04 12:17, Dicebot wrote:
> I liked the proposal to add special `dub` category for those (`dub`
> needs some categorization anyway). Only drawback is that it actually
> needs to be implemented instead of simply making a decision :)

For that, dub needs to be bundled with DMD.

-- 
/Jacob Carlborg

On Friday, 4 October 2013 at 13:30:39 UTC, Jacob Carlborg wrote:
> On 2013-10-04 12:17, Dicebot wrote:
>> I liked the proposal to add special `dub` category for those (`dub`
>> needs some categorization anyway). Only drawback is that it actually
>> needs to be implemented instead of simply making a decision :)
>
> For that, dub needs to be bundled with DMD.

I think it will at some moment. Adding such categorization capabilities is just another step towards official endorsement.

On Friday, 4 October 2013 at 10:02:21 UTC, John Colvin wrote:
> I would imagine a compulsory waiting period of the order of months, combined with a requirement of evidence that the module has been effective in the real world and that any outstanding problems have been resolved appropriately**.

I think this requirement is unobtainable. You're basically saying, "Hey go use this in your real world applications. We've got a mandated break of that application, just don't know when that is. But totally use this like you can rely on it."

I know it is really nice to have these test libraries released with the compiler, but people really just need to go out and use these libraries before the inclusion. I'm guilty too. I had use for std.uuid, but didn't test it against my real code. It went through review and even after inclusion I was still using the basic generator I created from an RFC doc. I've since made the switch, but I didn't do it because a review needed my help.

I think a stdx could be beneficial in our current state but I think it should be temporary. One maybe two years. The timing for move should be clear, accepted => stdx => next release => std.

On Saturday, 5 October 2013 at 01:53:13 UTC, Jesse Phillips wrote:
> On Friday, 4 October 2013 at 10:02:21 UTC, John Colvin wrote:
>> I would imagine a compulsory waiting period of the order of months, combined with a requirement of evidence that the module has been effective in the real world and that any outstanding problems have been resolved appropriately**.
>
> I think this requirement is unobtainable. You're basically saying, "Hey go use this in your real world applications. We've got a mandated break of that application, just don't know when that is. But totally use this like you can rely on it."
>
> I know it is really nice to have these test libraries released with the compiler, but people really just need to go out and use these libraries before the inclusion. I'm guilty too. I had use for std.uuid, but didn't test it against my real code. It went through review and even after inclusion I was still using the basic generator I created from an RFC doc. I've since made the switch, but I didn't do it because a review needed my help.
>
> I think a stdx could be beneficial in our current state but I think it should be temporary. One maybe two years. The timing for move should be clear, accepted => stdx => next release => std.

I quite like the idea of a stdx and imagine it to be similar to what we have in boost for C++.

A set of high quality, peer-reviewed libraries that are worthy of Phobos but haven't yet had enough real-world testing to be accepted into Phobos proper. Hopefully the peer-review process would include core Phobos developers to ensure the D-essence and quality is maintained throughout.

> I think a stdx could be beneficial in our current state but I think it should be temporary. One maybe two years. The timing for move should be clear, accepted => stdx => next release => std.

I agree partially with this. Once a library is flagged for Phobos from stdx it has to have a consistent and set migration path. But I don't think all stdx libraries need to be Phobos targets. For example, I don't think Phobos needs a CSV reader module, nor std.net.curl, even though both are very useful. Opt-in stdx modules via the official dub package manager would be a good option, allowing Phobos proper to remain small and light as possible.

G.