January 31, 2013
On 1/31/13 10:38 AM, Don wrote:
> The basic problem is that there are hundreds of potential numeric
> algorithms and data structures of equal importance to these ones. In
> fact, the total number of mathematical algorithms is probably a
> substantial fraction of the total algorithms in computer science!
>
> Even a module which contained only FFT, could be quite large, once it
> included all the important related transforms.

A possible solution would be to make std.numeric a knot for several others, or convert it to a package.

> I'm not sure that we can solve this without addressing the high-level
> question: What is the scope of Phobos?
>
> How big will it eventually get? Twice its current size? Ten times? A
> hundred times?

I think the current trend is to go for large libraries.

> Both SmallPhobos and LargePhobos are reasonable, but we do have to pick
> one. Currently we have aspects of both approaches, but they aren't
> compatible.

LargePhobos. Agreed that we need to package-ize some of the current modules.


Andrei


January 31, 2013
On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:
> I'm not sure that we can solve this without addressing the high-level question: What is the scope of Phobos?
>
> How big will it eventually get? Twice its current size? Ten times? A hundred times?

Purely by chance I read an article on the blog of Jarrett Billingsley about this topic:

http://www.jfbillingsley.com/blog/?p=206
January 31, 2013
On Thu, Jan 31, 2013 at 07:53:58PM +0100, Zach the Mystic wrote:
> On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:
> >I'm not sure that we can solve this without addressing the high-level question: What is the scope of Phobos?
> >
> >How big will it eventually get? Twice its current size? Ten times? A hundred times?
> 
> Purely by chance I read an article on the blog of Jarrett Billingsley about this topic:
> 
> http://www.jfbillingsley.com/blog/?p=206

Interesting. I think Phobos can be easily divided into two parts, with core stuff like std.algorithm, std.range, std.stdio, etc., in one, and less common but still widely applicable stuff like numeric algorithms in a "2nd party" library (or libraries).

Even if we don't actually split Phobos up in that way, this may help us reorganize Phobos in a better way. The current flat std hierarchy is slowly approaching the point where it's starting to tear at the seams. We should consider starting a 3rd level of hierarchy sooner rather than later.


T

-- 
Help a man when he is in trouble and he will remember you when he is in trouble again.
February 01, 2013
I agree that we need a distinct package for numerics, both numbers and algorithms.

I proposed a birds-of-a-feather discussion on this topic for DConf2013, but haven't heard back yet. If not officially, we can still discuss this there, perhaps?

Paul D. Anderson
February 01, 2013
On Thursday, 31 January 2013 at 17:05:50 UTC, Andrei Alexandrescu wrote:
> On 1/31/13 10:38 AM, Don wrote:
>> The basic problem is that there are hundreds of potential numeric
>> algorithms and data structures of equal importance to these ones. In
>> fact, the total number of mathematical algorithms is probably a
>> substantial fraction of the total algorithms in computer science!
>>
>> Even a module which contained only FFT, could be quite large, once it
>> included all the important related transforms.
>
> A possible solution would be to make std.numeric a knot for several others, or convert it to a package.
>
>> I'm not sure that we can solve this without addressing the high-level
>> question: What is the scope of Phobos?
>>
>> How big will it eventually get? Twice its current size? Ten times? A
>> hundred times?
>
> I think the current trend is to go for large libraries.
>
>> Both SmallPhobos and LargePhobos are reasonable, but we do have to pick
>> one. Currently we have aspects of both approaches, but they aren't
>> compatible.
>
> LargePhobos. Agreed that we need to package-ize some of the current modules.
>
>
> Andrei

Another consideration is the process for a package to become "standard". Current practice is to implement a package and go through a review in this forum. That's a good thing, but I'd like to see a more interactive approach, where the community was involved sooner:

1. A package is proposed for future inclusion in the library. Presumably this is done by someone with an interest in the subject, ideally this is someone who wants to spearhead the effort and may already have a draft implementation.

2. If there's enough interest to move forward, community members could bicker about all the bikeshed stuff: structures, methods, implementations, API, etc. until there's enough agreement about what the result should look like.

3. Interested parties could collaborate on the implementation and release.

4. There would have to be time limits. Many (most?) D projects start with a bang but then fizzle out before they're complete. There would need to be (monthly?) progress reviews. If the discussion or the implementation has dwindled, a reconsideration of the project could occur. If unsuccessful, the project would be moved off the active list. If there is renewed interest or if somebody else wants to take a crack at it the previous work can be a resource, but the project should probably go back to step 1.

You all see the problem with this: the very short attention span of the D community. But we've got a good start between the DIP process and the review queue. The only thing really different would be the progress reviews, which would hopefully prevent projects falling of the end of the earth.

This is just one possibility and there are probably better ways to do this. But I think some sort of maintenance of active projects is needed.

Paul
February 01, 2013
On 2013-02-01 20:15, Paul D. Anderson wrote:

> Another consideration is the process for a package to become "standard".
> Current practice is to implement a package and go through a review in
> this forum. That's a good thing, but I'd like to see a more interactive
> approach, where the community was involved sooner:
>
> 1. A package is proposed for future inclusion in the library. Presumably
> this is done by someone with an interest in the subject, ideally this is
> someone who wants to spearhead the effort and may already have a draft
> implementation.
>
> 2. If there's enough interest to move forward, community members could
> bicker about all the bikeshed stuff: structures, methods,
> implementations, API, etc. until there's enough agreement about what the
> result should look like.
>
> 3. Interested parties could collaborate on the implementation and release.
>
> 4. There would have to be time limits. Many (most?) D projects start
> with a bang but then fizzle out before they're complete. There would
> need to be (monthly?) progress reviews. If the discussion or the
> implementation has dwindled, a reconsideration of the project could
> occur. If unsuccessful, the project would be moved off the active list.
> If there is renewed interest or if somebody else wants to take a crack
> at it the previous work can be a resource, but the project should
> probably go back to step 1.
>
> You all see the problem with this: the very short attention span of the
> D community. But we've got a good start between the DIP process and the
> review queue. The only thing really different would be the progress
> reviews, which would hopefully prevent projects falling of the end of
> the earth.
>
> This is just one possibility and there are probably better ways to do
> this. But I think some sort of maintenance of active projects is needed.

Isn't this what the whole community is about and what we're already doing. Just not very formal or official.

-- 
/Jacob Carlborg
February 01, 2013
On Friday, 1 February 2013 at 20:04:47 UTC, Jacob Carlborg wrote:
> Isn't this what the whole community is about and what we're already doing. Just not very formal or official.

You're right, but what I was looking for was something that actually was more formal.

I think the only thing that's missing is someone to keep track of and check up on development efforts. So I'll revise my proposal: I'll volunteer to try to gather information on the projects that are in-work and see what I can do to find their status and gauge their progress. If doing this turns out to be too difficult to manage or of little value I'll announce that and shut up. But if it works, I thik it will add value to our efforts.

Paul
February 02, 2013
On Thursday, 31 January 2013 at 19:04:05 UTC, H. S. Teoh wrote:
> On Thu, Jan 31, 2013 at 07:53:58PM +0100, Zach the Mystic wrote:
>> On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:
>> >I'm not sure that we can solve this without addressing the
>> >high-level question: What is the scope of Phobos?
>> >
>> >How big will it eventually get? Twice its current size? Ten times?
>> >A hundred times?
>> 
>> Purely by chance I read an article on the blog of Jarrett
>> Billingsley about this topic:
>> 
>> http://www.jfbillingsley.com/blog/?p=206
>
> Interesting. I think Phobos can be easily divided into two parts, with
> core stuff like std.algorithm, std.range, std.stdio, etc., in one, and
> less common but still widely applicable stuff like numeric algorithms in
> a "2nd party" library (or libraries).
>
> Even if we don't actually split Phobos up in that way, this may help us
> reorganize Phobos in a better way. The current flat std hierarchy is
> slowly approaching the point where it's starting to tear at the seams.
> We should consider starting a 3rd level of hierarchy sooner rather than
> later.
>
>
> T

I don't know if this is relevant, but I find Jarret's writings to be quite brilliant. He designed a Lua-like language called miniD, based partly on D, but renamed Croc after he left the D community. I feel like I've learned a lot about programming just by reading the spec for his language.
February 02, 2013
On Thursday, 31 January 2013 at 19:04:05 UTC, H. S. Teoh wrote:
> Interesting. I think Phobos can be easily divided into two parts, with
> core stuff like std.algorithm, std.range, std.stdio, etc., in one, and
> less common but still widely applicable stuff like numeric algorithms in
> a "2nd party" library (or libraries).
>
> Even if we don't actually split Phobos up in that way, this may help us
> reorganize Phobos in a better way. The current flat std hierarchy is
> slowly approaching the point where it's starting to tear at the seams.
> We should consider starting a 3rd level of hierarchy sooner rather than
> later.
>
>
> T

Whoops. Forgot to actually give the site:
http://www.jfbillingsley.com/croc/
February 02, 2013
On 01/31/2013 08:02 PM, H. S. Teoh wrote:
> Interesting. I think Phobos can be easily divided into two parts, with
> core stuff like std.algorithm, std.range, std.stdio, etc., in one, and
> less common but still widely applicable stuff like numeric algorithms in
> a "2nd party" library (or libraries).

Speaking as a researcher, I've on occasion had cause to call on stuff from such "2nd party" libraries, and it's not necessarily a happy experience.

A while ago I had cause to call on a package in CRAN, the contributed package archive for the R statistical programming language.  It wasn't one of the modules that's packaged for Debian et al., so I had to pull things in directly from R's own package management and build and install locally.

In and of itself, R makes this very easy, but what I wasn't prepared for was that in installing the one package I was interested in, it would pull in and install dozens of other CRAN packages.  And then, when I looked inside the code, actually finding out what it was doing was nightmarish, because the module of interest was built on top of several other contributions by the same authors. Understanding the code involved a massive wild goose chase through all those other contributed modules to find what functions were being called and what they did.

I don't know if this is typical of CRAN packages, because I tend to use R infrequently, but my strong impression was of code that had been built, thrown over the wall and then built on top of without any attention to design, integration or performance (when I re-implemented the algorithms using Octave/MATLAB they were much faster, and I doubt this was down to the superiority of the language or interpreter).

And this code wasn't built by stupid people -- they were very good statisticians.  In their defence, I suspect the reason they built higgledy-piggledy as they did was because they knew their earlier modules _worked_ and reliability was the most important thing for them.

That's the cost of 2nd-party libraries -- they are very hit-and-miss in terms of design, sustainability and hence, reliability.

Now, that said, I think a "2nd-party" repository for D could be a great project, but what I _wouldn't_ like to see was that repository being considered an adequate replacement for carefully designed standard library functionality.  One of the things I love about D is precisely the breadth of Phobos' support, and it feels like the solution to Phobos' problems is a better design and review process rather than ringfencing a too-small set of core functionality.

Of course, a 2nd-party repository could be part of the prototyping and experimentation behind new standard-library work, just as Boost is for C++.