Hi all,

Normal Phobos submission procedure is usually like that:

1. write entire module from scratch by oneself
2. submit for voting
3. rewrite wrong parts, if there are none then add it to Phobos
4. otherwise goto 2

It is hard for one person to write entire module in such way it satisfies everyone, especially for very new or complex additions, such as database handling or cryptography.

Here, I propose that we add experimental "exp" hierarchy to Phobos for such projects. I know etc hierarchy may me used for that but IMHO separate "exp" would be more appropriate. In this namespace "beta" modules will slowly evolve into official std modules.

Some (obvious) advantages are:

* developers may receive feedback very early in the process, saving them from mass rewrites when in the opinion of community they made some wrong coding decission.
* developers may receive coding help, i.e. many of them may collaborate on one big module.
* users may test experimental modules early. This is important, because usability issues may be catched earlier, not after submission when API usually becomes frozen and it is too late.

"Exp" code may be shipped with each release just like "etc" code. Users using experimental code should be aware of breaking changes that may be introduced with each release or even with each commit.

Thoughts?

Piotr Szturmaj Wrote:

> Hi all,
> 
> Normal Phobos submission procedure is usually like that:
> 
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
> 
> It is hard for one person to write entire module in such way it satisfies everyone, especially for very new or complex additions, such as database handling or cryptography.
> 
> Here, I propose that we add experimental "exp" hierarchy to Phobos for such projects. I know etc hierarchy may me used for that but IMHO separate "exp" would be more appropriate. In this namespace "beta" modules will slowly evolve into official std modules.
> 
> Some (obvious) advantages are:
> 
> * developers may receive feedback very early in the process, saving them
> from mass rewrites when in the opinion of community they made some wrong
> coding decission.
> * developers may receive coding help, i.e. many of them may collaborate
> on one big module.
> * users may test experimental modules early. This is important, because
> usability issues may be catched earlier, not after submission when API
> usually becomes frozen and it is too late.
> 
> "Exp" code may be shipped with each release just like "etc" code. Users using experimental code should be aware of breaking changes that may be introduced with each release or even with each commit.
> 
> Thoughts?

I like it. More people will keep an eye on new versions for new modules that are coming in. And if one wants to write new module for his needs, it can easily check exp to see is he reinventing wheel. (exmple. my suggestion here few days back to upgrade std.process only to find there is upgraded module - if I didn't ask on this NG I would never know there is that module). So exp sounds good.

On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
> Hi all,
>
> Normal Phobos submission procedure is usually like that:
>
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
>
> It is hard for one person to write entire module in such way it
> satisfies everyone, especially for very new or complex additions, such
> as database handling or cryptography.
>
> Here, I propose that we add experimental "exp" hierarchy to Phobos for
> such projects. I know etc hierarchy may me used for that but IMHO
> separate "exp" would be more appropriate. In this namespace "beta"
> modules will slowly evolve into official std modules.
>
> Some (obvious) advantages are:
>
> * developers may receive feedback very early in the process, saving them
> from mass rewrites when in the opinion of community they made some wrong
> coding decission.
> * developers may receive coding help, i.e. many of them may collaborate
> on one big module.
> * users may test experimental modules early. This is important, because
> usability issues may be catched earlier, not after submission when API
> usually becomes frozen and it is too late.
>
> "Exp" code may be shipped with each release just like "etc" code. Users
> using experimental code should be aware of breaking changes that may be
> introduced with each release or even with each commit.
>
> Thoughts?

Isn't this just reinventing git branches?

If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc.  That's the whole point of a branch.

Peter Alexander wrote:
> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>> "Exp" code may be shipped with each release just like "etc" code. Users
>> using experimental code should be aware of breaking changes that may be
>> introduced with each release or even with each commit.
>>
>> Thoughts?
>
> Isn't this just reinventing git branches?
>
> If people have their work-in-progress branches on GitHub then people can
> already try them out, submit pull requests etc. That's the whole point
> of a branch.

Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.

On 12/18/2011 10:00 AM, Piotr Szturmaj wrote:
> Peter Alexander wrote:
>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>> "Exp" code may be shipped with each release just like "etc" code. Users using experimental code should be aware of breaking changes that may be introduced with each release or even with each commit.
>>>
>>> Thoughts?
>>
>> Isn't this just reinventing git branches?
>>
>> If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
> 
> Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.

That would be because there are no branches for it.

cd $DPATH/src/phobos
git branch
<crickets>

This proposal seems good, but I would do it in the form of an "exp" branch rather than an "exp" directory.

If this is desired for releases, then perhaps the release scripts could clone both the master branch and the exp branch so that it's all out in the open.

On Sunday, 18 December 2011 at 14:18:34 UTC, Piotr Szturmaj wrote:
> Hi all,
>
> Normal Phobos submission procedure is usually like that:
>
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
>
> It is hard for one person to write entire module in such way it satisfies everyone, especially for very new or complex additions, such as database handling or cryptography.
>
> Here, I propose that we add experimental "exp" hierarchy to Phobos for such projects. I know etc hierarchy may me used for that but IMHO separate "exp" would be more appropriate. In this namespace "beta" modules will slowly evolve into official std modules.
>
> Some (obvious) advantages are:
>
> * developers may receive feedback very early in the process, saving them from mass rewrites when in the opinion of community they made some wrong coding decission.
> * developers may receive coding help, i.e. many of them may collaborate on one big module.
> * users may test experimental modules early. This is important, because usability issues may be catched earlier, not after submission when API usually becomes frozen and it is too late.
>
> "Exp" code may be shipped with each release just like "etc" code. Users using experimental code should be aware of breaking changes that may be introduced with each release or even with each commit.
>
> Thoughts?

I think that people should just their WIP + links to github/docs on the "review queue" wiki page linked from the dlang.org left menu.

/Jonas

The idea reminds me of how extensions are managed in OpenGL: http://www.opengl.org/resources/features/OGLextensions/
Often hardware vendors like S3, nVidia or ATi invented cool stuff, like texture compression and were free to add a prefixed function name to their drivers (S3_…, NV_…, ATI_…). Sometimes other vendors would implement those functions as well and they are eventually renamed to EXT_…. Then there is the architecture review board (ARB) that picks up these extensions and evaluates how they could fit into OpenGL. Often changes are necessary for the function to become ARB_…. I think this is all still independent of the specification release cycle (OpenGL is just a specification). When a new specification is planned, ARB_… functions make it into GL_… core functionality.
I believe this concept helps innovation. But who would decide which module is a good fit for Phobos and which one is better fetched through an external package manager? For example, what if I came up a complete containers package that has different approaches from std.containers or a 2D drawing/image API or a D compiler front-end?
In OpenGL terminology, I think it would be an ARB_… already, if it is in a distribution of DMD. It would have a specification that some review board has accepted for inclusion into Phobos. Anything with a clear specification on paper or as an implementation could be accepted, but on the other hand, just saying that "I start writing an image handling module" should not be enough to be in .ext in Phobos. Otherwise some .ext modules will disappear suddenly or get stuck there forever, because as they mature the review board realizes, that the API is not going to be what they like to see in Phobos. That's frustrating for both the developer and users.

So in short: yes, but review what ends up in .ext

On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj <bncrbme@jadamspam.pl> wrote:

> Peter Alexander wrote:
>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>> using experimental code should be aware of breaking changes that may be
>>> introduced with each release or even with each commit.
>>>
>>> Thoughts?
>>
>> Isn't this just reinventing git branches?
>>
>> If people have their work-in-progress branches on GitHub then people can
>> already try them out, submit pull requests etc. That's the whole point
>> of a branch.
>
> Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.

I think it would be enough to post something on D.announce and then whoever is interested can watch your branch.  Maybe even a wiki page could categorize the branches.

I agree with others that github's forking and branching mechanism works quite well for developing multi-user projects.  For example, both Lars Kyllingstad and I are working on a new std.process module, he's doing the main design and unixen implementations, I'm doing the windows implementation.

-Steve

Steven Schveighoffer wrote:
> On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
> <bncrbme@jadamspam.pl> wrote:
>
>> Peter Alexander wrote:
>>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>>> using experimental code should be aware of breaking changes that may be
>>>> introduced with each release or even with each commit.
>>>>
>>>> Thoughts?
>>>
>>> Isn't this just reinventing git branches?
>>>
>>> If people have their work-in-progress branches on GitHub then people can
>>> already try them out, submit pull requests etc. That's the whole point
>>> of a branch.
>>
>> Yes, but what I propose is the centralized repository for that
>> branches (eventual candidates to std). Currently no one knows all
>> modules that are being worked on.
>
> I think it would be enough to post something on D.announce and then
> whoever is interested can watch your branch. Maybe even a wiki page
> could categorize the branches.
>
> I agree with others that github's forking and branching mechanism works
> quite well for developing multi-user projects. For example, both Lars
> Kyllingstad and I are working on a new std.process module, he's doing
> the main design and unixen implementations, I'm doing the windows
> implementation.

Okay, I agree too. The core problem is that modules after inclusion become official/public instantly and API must be frozen. Eventual API issues discovered after the inclusion are often impossible to fix because nobody "likes" deprecations and breaking changes.

So, instead of making modules public from the start, I would rather give some additional "beta" time for them. Interested users may check out these modules in the "real world" during that time. Also, beta code may introduce breaking changes without worrying.

I think review is a requirement of course but it is often not enough. Some things may not be catched during review. Code must be extensively _used_ to prove its quality.

If you agree with me, there is only one additional rule to add to Phobos contribution process. After a review and voting process, add module to etc (or exp, whatever) for one month. Then move it to std if no/little changes were made during that trial. If changes are big the trial period should be extended.

I think this idea needs further consideration.

Summarizing the earlier discussion, there were four schools of thought:

1. This is a good idea.
2. This is a good idea, but let's use github branches.
3. This is a good idea, but let's use newsgroup postings.
4. This is a good idea, but let's use the review queue page on the wiki.

I'd hate to see this good idea not get implemented because we can't agree on the mechanism. The suggestions above are not mutually exclusive.

It seems to me that there are two primary requirements for this to be effective:

1. There should be ONE place where people can go to see what's under development.

2. That ONE place needs to be easy to find, as in easy for someone visiting the D programming language website for the very first time.

Here's my proposal: (Pay attention, there's a quiz at the end.)

Add another menu item under the "Community" menu to the left here. Call it "Under Development" or "Work in Progress" or "Experimental Library modules" or redefine the "Projects & Libraries" link. Whatever. The link would lead to a list of modules under development. These could be github branches, dsource projects, review queue items, etc. It doesn't matter how the project is sourced or maintained.

Anyone who has a module under development could add a URL to their project page at that ONE place.
 These could be in any stage of development -- just a proposal or maybe just a good idea, a formal specification, a beta version, a complete library, or even a wish list...)

There may be other, better ways to implement this, but I would find it very useful to have that ONE place.

For example, I spent quite a bit of time, spread over a couple of weeks, looking for a D language statistics library. It turns out there is at least one.

Quiz: 1) Where is it? 2) Why isn't it being considered for Phobos?


On Monday, 19 December 2011 at 16:46:46 UTC, Piotr Szturmaj wrote:
> Steven Schveighoffer wrote:
>> On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
>> <bncrbme@jadamspam.pl> wrote:
>>
>>> Peter Alexander wrote:
>>>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>>>> using experimental code should be aware of breaking changes that may be
>>>>> introduced with each release or even with each commit.
>>>>>
>>>>> Thoughts?
>>>>
>>>> Isn't this just reinventing git branches?
>>>>
>>>> If people have their work-in-progress branches on GitHub then people can
>>>> already try them out, submit pull requests etc. That's the whole point
>>>> of a branch.
>>>
>>> Yes, but what I propose is the centralized repository for that
>>> branches (eventual candidates to std). Currently no one knows all
>>> modules that are being worked on.
>>
>> I think it would be enough to post something on D.announce and then
>> whoever is interested can watch your branch. Maybe even a wiki page
>> could categorize the branches.
>>
>> I agree with others that github's forking and branching mechanism works
>> quite well for developing multi-user projects. For example, both Lars
>> Kyllingstad and I are working on a new std.process module, he's doing
>> the main design and unixen implementations, I'm doing the windows
>> implementation.
>
> Okay, I agree too. The core problem is that modules after inclusion become official/public instantly and API must be frozen. Eventual API issues discovered after the inclusion are often impossible to fix because nobody "likes" deprecations and breaking changes.
>
> So, instead of making modules public from the start, I would rather give some additional "beta" time for them. Interested users may check out these modules in the "real world" during that time. Also, beta code may introduce breaking changes without worrying.
>
> I think review is a requirement of course but it is often not enough. Some things may not be catched during review. Code must be extensively _used_ to prove its quality.
>
> If you agree with me, there is only one additional rule to add to Phobos contribution process. After a review and voting process, add module to etc (or exp, whatever) for one month. Then move it to std if no/little changes were made during that trial. If changes are big the trial period should be extended.