View mode: basic / threaded / horizontal-split · Log in · Help
December 18, 2011
Incubated modules for Phobos
Hi all,

Normal Phobos submission procedure is usually like that:

1. write entire module from scratch by oneself
2. submit for voting
3. rewrite wrong parts, if there are none then add it to Phobos
4. otherwise goto 2

It is hard for one person to write entire module in such way it 
satisfies everyone, especially for very new or complex additions, such 
as database handling or cryptography.

Here, I propose that we add experimental "exp" hierarchy to Phobos for 
such projects. I know etc hierarchy may me used for that but IMHO 
separate "exp" would be more appropriate. In this namespace "beta" 
modules will slowly evolve into official std modules.

Some (obvious) advantages are:

* developers may receive feedback very early in the process, saving them 
from mass rewrites when in the opinion of community they made some wrong 
coding decission.
* developers may receive coding help, i.e. many of them may collaborate 
on one big module.
* users may test experimental modules early. This is important, because 
usability issues may be catched earlier, not after submission when API 
usually becomes frozen and it is too late.

"Exp" code may be shipped with each release just like "etc" code. Users 
using experimental code should be aware of breaking changes that may be 
introduced with each release or even with each commit.

Thoughts?
December 18, 2011
Re: Incubated modules for Phobos
Piotr Szturmaj Wrote:

> Hi all,
> 
> Normal Phobos submission procedure is usually like that:
> 
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
> 
> It is hard for one person to write entire module in such way it 
> satisfies everyone, especially for very new or complex additions, such 
> as database handling or cryptography.
> 
> Here, I propose that we add experimental "exp" hierarchy to Phobos for 
> such projects. I know etc hierarchy may me used for that but IMHO 
> separate "exp" would be more appropriate. In this namespace "beta" 
> modules will slowly evolve into official std modules.
> 
> Some (obvious) advantages are:
> 
> * developers may receive feedback very early in the process, saving them 
> from mass rewrites when in the opinion of community they made some wrong 
> coding decission.
> * developers may receive coding help, i.e. many of them may collaborate 
> on one big module.
> * users may test experimental modules early. This is important, because 
> usability issues may be catched earlier, not after submission when API 
> usually becomes frozen and it is too late.
> 
> "Exp" code may be shipped with each release just like "etc" code. Users 
> using experimental code should be aware of breaking changes that may be 
> introduced with each release or even with each commit.
> 
> Thoughts?

I like it. More people will keep an eye on new versions for new modules that are coming in. And if one wants to write new module for his needs, it can easily check exp to see is he reinventing wheel. (exmple. my suggestion here few days back to upgrade std.process only to find there is upgraded module - if I didn't ask on this NG I would never know there is that module). So exp sounds good.
December 18, 2011
Re: Incubated modules for Phobos
On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
> Hi all,
>
> Normal Phobos submission procedure is usually like that:
>
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
>
> It is hard for one person to write entire module in such way it
> satisfies everyone, especially for very new or complex additions, such
> as database handling or cryptography.
>
> Here, I propose that we add experimental "exp" hierarchy to Phobos for
> such projects. I know etc hierarchy may me used for that but IMHO
> separate "exp" would be more appropriate. In this namespace "beta"
> modules will slowly evolve into official std modules.
>
> Some (obvious) advantages are:
>
> * developers may receive feedback very early in the process, saving them
> from mass rewrites when in the opinion of community they made some wrong
> coding decission.
> * developers may receive coding help, i.e. many of them may collaborate
> on one big module.
> * users may test experimental modules early. This is important, because
> usability issues may be catched earlier, not after submission when API
> usually becomes frozen and it is too late.
>
> "Exp" code may be shipped with each release just like "etc" code. Users
> using experimental code should be aware of breaking changes that may be
> introduced with each release or even with each commit.
>
> Thoughts?

Isn't this just reinventing git branches?

If people have their work-in-progress branches on GitHub then people can 
already try them out, submit pull requests etc.  That's the whole point 
of a branch.
December 18, 2011
Re: Incubated modules for Phobos
Peter Alexander wrote:
> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>> "Exp" code may be shipped with each release just like "etc" code. Users
>> using experimental code should be aware of breaking changes that may be
>> introduced with each release or even with each commit.
>>
>> Thoughts?
>
> Isn't this just reinventing git branches?
>
> If people have their work-in-progress branches on GitHub then people can
> already try them out, submit pull requests etc. That's the whole point
> of a branch.

Yes, but what I propose is the centralized repository for that branches 
(eventual candidates to std). Currently no one knows all modules that 
are being worked on.
December 18, 2011
Re: Incubated modules for Phobos
On 12/18/2011 10:00 AM, Piotr Szturmaj wrote:
> Peter Alexander wrote:
>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>> using experimental code should be aware of breaking changes that may be
>>> introduced with each release or even with each commit.
>>>
>>> Thoughts?
>>
>> Isn't this just reinventing git branches?
>>
>> If people have their work-in-progress branches on GitHub then people can
>> already try them out, submit pull requests etc. That's the whole point
>> of a branch.
> 
> Yes, but what I propose is the centralized repository for that branches
> (eventual candidates to std). Currently no one knows all modules that
> are being worked on.

That would be because there are no branches for it.

cd $DPATH/src/phobos
git branch
<crickets>

This proposal seems good, but I would do it in the form of an "exp"
branch rather than an "exp" directory.

If this is desired for releases, then perhaps the release scripts could
clone both the master branch and the exp branch so that it's all out in
the open.
December 18, 2011
Re: Incubated modules for Phobos
On Sunday, 18 December 2011 at 14:18:34 UTC, Piotr Szturmaj wrote:
> Hi all,
>
> Normal Phobos submission procedure is usually like that:
>
> 1. write entire module from scratch by oneself
> 2. submit for voting
> 3. rewrite wrong parts, if there are none then add it to Phobos
> 4. otherwise goto 2
>
> It is hard for one person to write entire module in such way it 
> satisfies everyone, especially for very new or complex 
> additions, such as database handling or cryptography.
>
> Here, I propose that we add experimental "exp" hierarchy to 
> Phobos for such projects. I know etc hierarchy may me used for 
> that but IMHO separate "exp" would be more appropriate. In this 
> namespace "beta" modules will slowly evolve into official std 
> modules.
>
> Some (obvious) advantages are:
>
> * developers may receive feedback very early in the process, 
> saving them from mass rewrites when in the opinion of community 
> they made some wrong coding decission.
> * developers may receive coding help, i.e. many of them may 
> collaborate on one big module.
> * users may test experimental modules early. This is important, 
> because usability issues may be catched earlier, not after 
> submission when API usually becomes frozen and it is too late.
>
> "Exp" code may be shipped with each release just like "etc" 
> code. Users using experimental code should be aware of breaking 
> changes that may be introduced with each release or even with 
> each commit.
>
> Thoughts?

I think that people should just their WIP + links to github/docs 
on the "review queue" wiki page linked from the dlang.org left 
menu.

/Jonas
December 19, 2011
Re: Incubated modules for Phobos
The idea reminds me of how extensions are managed in OpenGL:  
http://www.opengl.org/resources/features/OGLextensions/
Often hardware vendors like S3, nVidia or ATi invented cool stuff, like  
texture compression and were free to add a prefixed function name to their  
drivers (S3_…, NV_…, ATI_…). Sometimes other vendors would implement those  
functions as well and they are eventually renamed to EXT_…. Then there is  
the architecture review board (ARB) that picks up these extensions and  
evaluates how they could fit into OpenGL. Often changes are necessary for  
the function to become ARB_…. I think this is all still independent of the  
specification release cycle (OpenGL is just a specification). When a new  
specification is planned, ARB_… functions make it into GL_… core  
functionality.
I believe this concept helps innovation. But who would decide which module  
is a good fit for Phobos and which one is better fetched through an  
external package manager? For example, what if I came up a complete  
containers package that has different approaches from std.containers or a  
2D drawing/image API or a D compiler front-end?
In OpenGL terminology, I think it would be an ARB_… already, if it is in a  
distribution of DMD. It would have a specification that some review board  
has accepted for inclusion into Phobos. Anything with a clear  
specification on paper or as an implementation could be accepted, but on  
the other hand, just saying that "I start writing an image handling  
module" should not be enough to be in .ext in Phobos. Otherwise some .ext  
modules will disappear suddenly or get stuck there forever, because as  
they mature the review board realizes, that the API is not going to be  
what they like to see in Phobos. That's frustrating for both the developer  
and users.

So in short: yes, but review what ends up in .ext
December 19, 2011
Re: Incubated modules for Phobos
On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj <bncrbme@jadamspam.pl>  
wrote:

> Peter Alexander wrote:
>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>> using experimental code should be aware of breaking changes that may be
>>> introduced with each release or even with each commit.
>>>
>>> Thoughts?
>>
>> Isn't this just reinventing git branches?
>>
>> If people have their work-in-progress branches on GitHub then people can
>> already try them out, submit pull requests etc. That's the whole point
>> of a branch.
>
> Yes, but what I propose is the centralized repository for that branches  
> (eventual candidates to std). Currently no one knows all modules that  
> are being worked on.

I think it would be enough to post something on D.announce and then  
whoever is interested can watch your branch.  Maybe even a wiki page could  
categorize the branches.

I agree with others that github's forking and branching mechanism works  
quite well for developing multi-user projects.  For example, both Lars  
Kyllingstad and I are working on a new std.process module, he's doing the  
main design and unixen implementations, I'm doing the windows  
implementation.

-Steve
December 19, 2011
Re: Incubated modules for Phobos
Steven Schveighoffer wrote:
> On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
> <bncrbme@jadamspam.pl> wrote:
>
>> Peter Alexander wrote:
>>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>>> "Exp" code may be shipped with each release just like "etc" code. Users
>>>> using experimental code should be aware of breaking changes that may be
>>>> introduced with each release or even with each commit.
>>>>
>>>> Thoughts?
>>>
>>> Isn't this just reinventing git branches?
>>>
>>> If people have their work-in-progress branches on GitHub then people can
>>> already try them out, submit pull requests etc. That's the whole point
>>> of a branch.
>>
>> Yes, but what I propose is the centralized repository for that
>> branches (eventual candidates to std). Currently no one knows all
>> modules that are being worked on.
>
> I think it would be enough to post something on D.announce and then
> whoever is interested can watch your branch. Maybe even a wiki page
> could categorize the branches.
>
> I agree with others that github's forking and branching mechanism works
> quite well for developing multi-user projects. For example, both Lars
> Kyllingstad and I are working on a new std.process module, he's doing
> the main design and unixen implementations, I'm doing the windows
> implementation.

Okay, I agree too. The core problem is that modules after inclusion 
become official/public instantly and API must be frozen. Eventual API 
issues discovered after the inclusion are often impossible to fix 
because nobody "likes" deprecations and breaking changes.

So, instead of making modules public from the start, I would rather give 
some additional "beta" time for them. Interested users may check out 
these modules in the "real world" during that time. Also, beta code may 
introduce breaking changes without worrying.

I think review is a requirement of course but it is often not enough. 
Some things may not be catched during review. Code must be extensively 
_used_ to prove its quality.

If you agree with me, there is only one additional rule to add to Phobos 
contribution process. After a review and voting process, add module to 
etc (or exp, whatever) for one month. Then move it to std if no/little 
changes were made during that trial. If changes are big the trial period 
should be extended.
January 04, 2012
Re: Incubated modules for Phobos
I think this idea needs further consideration.

Summarizing the earlier discussion, there were four schools of 
thought:

1. This is a good idea.
2. This is a good idea, but let's use github branches.
3. This is a good idea, but let's use newsgroup postings.
4. This is a good idea, but let's use the review queue page on 
the wiki.

I'd hate to see this good idea not get implemented because we 
can't agree on the mechanism. The suggestions above are not 
mutually exclusive.

It seems to me that there are two primary requirements for this 
to be effective:

1. There should be ONE place where people can go to see what's 
under development.

2. That ONE place needs to be easy to find, as in easy for 
someone visiting the D programming language website for the very 
first time.

Here's my proposal: (Pay attention, there's a quiz at the end.)

Add another menu item under the "Community" menu to the left 
here. Call it "Under Development" or "Work in Progress" or 
"Experimental Library modules" or redefine the "Projects & 
Libraries" link. Whatever. The link would lead to a list of 
modules under development. These could be github branches, 
dsource projects, review queue items, etc. It doesn't matter how 
the project is sourced or maintained.

Anyone who has a module under development could add a URL to 
their project page at that ONE place.
 These could be in any stage of development -- just a proposal or 
maybe just a good idea, a formal specification, a beta version, a 
complete library, or even a wish list...)

There may be other, better ways to implement this, but I would 
find it very useful to have that ONE place.

For example, I spent quite a bit of time, spread over a couple of 
weeks, looking for a D language statistics library. It turns out 
there is at least one.

Quiz: 1) Where is it? 2) Why isn't it being considered for Phobos?


On Monday, 19 December 2011 at 16:46:46 UTC, Piotr Szturmaj wrote:
> Steven Schveighoffer wrote:
>> On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
>> <bncrbme@jadamspam.pl> wrote:
>>
>>> Peter Alexander wrote:
>>>> On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
>>>>> "Exp" code may be shipped with each release just like "etc" 
>>>>> code. Users
>>>>> using experimental code should be aware of breaking changes 
>>>>> that may be
>>>>> introduced with each release or even with each commit.
>>>>>
>>>>> Thoughts?
>>>>
>>>> Isn't this just reinventing git branches?
>>>>
>>>> If people have their work-in-progress branches on GitHub 
>>>> then people can
>>>> already try them out, submit pull requests etc. That's the 
>>>> whole point
>>>> of a branch.
>>>
>>> Yes, but what I propose is the centralized repository for that
>>> branches (eventual candidates to std). Currently no one knows 
>>> all
>>> modules that are being worked on.
>>
>> I think it would be enough to post something on D.announce and 
>> then
>> whoever is interested can watch your branch. Maybe even a wiki 
>> page
>> could categorize the branches.
>>
>> I agree with others that github's forking and branching 
>> mechanism works
>> quite well for developing multi-user projects. For example, 
>> both Lars
>> Kyllingstad and I are working on a new std.process module, 
>> he's doing
>> the main design and unixen implementations, I'm doing the 
>> windows
>> implementation.
>
> Okay, I agree too. The core problem is that modules after 
> inclusion become official/public instantly and API must be 
> frozen. Eventual API issues discovered after the inclusion are 
> often impossible to fix because nobody "likes" deprecations and 
> breaking changes.
>
> So, instead of making modules public from the start, I would 
> rather give some additional "beta" time for them. Interested 
> users may check out these modules in the "real world" during 
> that time. Also, beta code may introduce breaking changes 
> without worrying.
>
> I think review is a requirement of course but it is often not 
> enough. Some things may not be catched during review. Code must 
> be extensively _used_ to prove its quality.
>
> If you agree with me, there is only one additional rule to add 
> to Phobos contribution process. After a review and voting 
> process, add module to etc (or exp, whatever) for one month. 
> Then move it to std if no/little changes were made during that 
> trial. If changes are big the trial period should be extended.
Top | Discussion index | About this forum | D home