On Friday, 2 June 2017 at 18:43:32 UTC, Andrei Alexandrescu wrote:

> A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.

Indeed, the idea is an inviting one. Fixing mistakes of the past may be tedious, but it is a good way of moving forward. I don't think such a project actually needs a large amount of full-time participants: if not any other reason, it is impractical. We can't reasonably hope to have experts on everything in an OSS project like this, at least not on a day-to-day basis. A small core group inviting PRs and perhaps using a voting system for accepting/rejecting features should suffice. The biggest challenge is to dilute incoming features into most basic forms, simple, efficient and reusable components, which may occasionally fall out of the collective expertise.
However, it might be prudent to adopt a waiting stance on this, and let several key language features to mature (i.e. the current DIP queue). The big problem of Phobos is that it grew together with the language, often lagging behind in terms of use of language features, while still growing the code base, depending on obsolete functionality, and making future fixes harder and harder. Repeating this will be a huge mistake, and inevitably will lead to talks about Phobos 3 in a few years.

On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
> A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.

Haha, I happened to be reading Spolsky's blog, and came across this:

	https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/

Especially the bit about throwing away years of accumulated programming work.


T

-- 
Ignorance is bliss... until you suffer the consequences!

On Friday, June 02, 2017 13:37:05 H. S. Teoh via Digitalmars-d wrote:
> On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
>
> > A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.
>
> Haha, I happened to be reading Spolsky's blog, and came across this:
>
>   https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part
> -i/
>
> Especially the bit about throwing away years of accumulated programming work.

I don't entirely agree with his sentiments on that one, but he does have a point. Throwing the baby out with the bath water is obviously something to avoid, but sometimes redoing code from scratch does really fix problems - especially when the original has serious design problems that can't be fixed without redoing it. However, thinking about what he's saying there, it starts looking like the most valuable part of your code base is your tests (assuming that they're thorough), because then if you do completely rewrite something, you can verify that it's correct.

As for "Phobos 2" specifically though, I have very mixed feelings about such an idea. Being able to rearrange things how we'd like and fix some of the mistakes that we've been stuck with would be great, but you're basically planning on breaking every single D program, even if it's not immediately and folks could switch over to std2 (or whatever the package was) at a point that made sense for them. And especially if you're talking about range-related stuff, you'd be essentially splitting the D community and its code in two, because you'd almost certainly be dealing with incompatible libraries - especially if you're actually fixing some of the range-related problems. And that sort of thing just seems like it would be too destructive to be worth the risk. It's not quite a language split like you got with something like python 2 and python 3 or D1 and D2, but if the old and new standard libraries were not compatible, it would be close to that - it also seems too much like the whole Phobos vs Tango mess, even if the idea would be to be transitioning from one to the other rather than really having competing libraries (but as we've seen with python 2 vs 3 and D1 vs D2, major transitions like this tend to result in the old and the new competing even if that's not the intent).

And honestly, for the most part, I don't think that redesigning the standard library would actually buy us much. Aside from range-related issues, everything I can think of is essentially a minor annoyance or wort. Sure, it would be nice if we could fix it, but it's not worth a massive upheaval of the D ecosystem. The only stuff that I can think of right now that would be a huge gain if we could change it is range-related stuff. I can think of several items there that would be nice to fix or at least look at changing:

1. Get rid of auto-decoding ranges and the concept of narrow strings. string and wstring would be ranges of char and wchar, everything operating on characters would then be operating at the code unit, code point, or grapheme level (depending on what was appropriate for the algorithm), and generic stuff would just operate on the elements regardless of whether they were code units, code points, or graphemes, leaving it to the programmer to handle it correctly.

2. Get rid of save and disallow classes as ranges. Postblit constructors then do what save does now. If you want a class to be a range, then wrap it in a struct, kind of like a smart pointer but for ranges.

3. Examine how to better deal with input ranges vs forward ranges and look at requiring that input ranges have reference semantics whereas forward ranges must have value semantics (as it stands, an input range _without_ reference semantics really should be a forward range, but forward ranges can still have reference semantics - hence the need for save).

4. Investigate adjusting the range hierarchy so that we have random-access ranges which aren't input ranges. I don't know for sure that that makes sense, but as I understand it, Mir does something like that, because it has a definite performance benefit under some circumstances.

5. Look at whether a car/cdr approach would be better for input ranges (and maybe any range that isn't random-access) than front/popFront. Andrei wants to do something with that for immutable ranges, and it could help resolve the issue of const ranges if that's what we did in general (but without something like Phobos 2, I personally think that such a change is too drastic even if Andrei is currently thinking about it for immutable containers).

6. Look at improving output ranges so that you actually know whether something is going to fit in them - possibly by having a type of output range that has a length or some other member indicating the space remaining.

7. Look at formalizing what std.digest did with output ranges and have a way to indicate that you're done putting stuff in them so that they can do something and "close" or "complete" the range.

And there are probably more things that we could look at with ranges, which would be nice to fix. So, being able to go back to the drawing board on some level with ranges would be _fantastic_, but to do so would break most everything, and aside from ranges, I really don't see much gain in a "Phobos 2". Aside from nitpicks with names and whatnot, it should be possible to make most changes without doing a full, new iteration of the standard library.

And as Daniel Murphy pointed out several times at dconf, if you really step back and just use D as it is, it's pretty darn fantastic, and there's too much going with folks trying to make it perfect. Yes, D and Phobos have warts. Yes, it would be nice to make some improvements which would not be backwards compatible. But what we have is already pretty amazing, and when fixing something requires pulling the rug out from _everyone_ using D, you have to really consider that it's better having something amazing but imperfect than to try and fix it. Yes, sometimes, drastic changes are worth the breakage - especially when most of the required changes actually results in bugs being fixed - but redoing the standard library? That really seems like too big a change.

- Jonathan M Davis

On Friday, June 02, 2017 14:43:32 Andrei Alexandrescu via Digitalmars-d wrote:
> On 06/02/2017 01:43 PM, H. S. Teoh via Digitalmars-d wrote:
> > On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
> > [...]
> >
> >> D has a painful history with two competing standard libraries. If you seriously propose this path, I hope Andrei and Walter will publicly and vehemently oppose it. Otherwise that ghost from the past becomes a PR disaster for D.
> >
> > [...]
> >
> > I think this is overreacting.  "Phobos 2" is not supposed to be a *competing* library, but, as I see it, more like an alpha version of the next "major iteration" of Phobos. The "next major version" with a brand new paradigm, if you will, as opposed to incremental changes to the "current major version".
>
> A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.
>
> As an example, eliminating phobos' and druntime's reliance on "static this" would improve conviviality of D in mixed-language projects, but we have yet to find folks to do the work. To the best of my knowledge only David, Stanislav, and myself have been involved so far.

A related point is what's happened with modules like std.json and std.xml. We've said for years that they need to be replaced. But have they been? No. Some work has been done with the idea that it would replace them, but either it's never been done, or there was too much arguing over the replacement and what it should do for it to get through the review process. We can't replace these two modules, and we're now talking about the possibility of redoing the entire standard library? When you think of it that way, it seems insane, even if it might be desirable.

Sure, if we were going to redo Phobos, most of it would probably just be transferred to "Phobos 2" but rearranged as appropriate rather than completely redone, but stuff like revamping ranges would affect almost everything in Phobos, making that a _major_ task even without considering how it would affect all of the D code that isn't in Phobos but uses it. I think that we've pretty much proven that we can't do that with the current level of manpower.

And of course, doing stuff like adjusting Phobos to mitigate the effect of auto-decoding or to revamp function signatures such that most overloads are done internally with static ifs where possible - or the static constructor issue - all are smaller things that we have trouble getting done even when we mostly agree that they should be done. We do get some work done, to be sure, but we need more eyes on pull requests so that they get merged in a timely manner and more pull requests to fix these smaller issues. If we can't even do those right, how could we possibly revamp all of the standard library?

I can certainly empathize with the sentiment that it would be nice to go back to the drawing board on some things, but I think that doing that with the whole standard library is just too much - both from the standpoint of manpower and with regards to how much code it would ultimately break.

- Jonathan M Davis

On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:
> Especially the bit about throwing away years of accumulated programming work.

Our IRC discussion was more about simplifying the Phobos interface through breaking changes than actually just dropping and rewriting. Most the "phobos 2" would just be today's phobos with mistakes changed, autodecoding deprecated then removed, eager functions same deal (the discussion that started it was split vs splitter), static array interfaces removed (you'd have to explicitly slice at the call site instead of Phobos trying to two-layer ref them), etc.

So step one of this would be to copy existing code. Then cut the mistakes out.

The modified modules are just given a new package name (or something) to ease the transition.

On Friday, 2 June 2017 at 21:59:42 UTC, Jonathan M Davis wrote:
> On Friday, June 02, 2017 13:37:05 H. S. Teoh via Digitalmars-d wrote:
>> On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
>>
>> > A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.
>>
>> Haha, I happened to be reading Spolsky's blog, and came across this:
>>
>>   https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part
>> -i/
>>
>> Especially the bit about throwing away years of accumulated programming work.
>
> I don't entirely agree with his sentiments on that one, but he does have a point. Throwing the baby out with the bath water is obviously something to avoid, but sometimes redoing code from scratch does really fix problems - especially when the original has serious design problems that can't be fixed without redoing it. However, thinking about what he's saying there, it starts looking like the most valuable part of your code base is your tests (assuming that they're thorough), because then if you do completely rewrite something, you can verify that it's correct.
>
> As for "Phobos 2" specifically though, I have very mixed feelings about such an idea. Being able to rearrange things how we'd like and fix some of the mistakes that we've been stuck with would be great, but you're basically planning on breaking every single D program, even if it's not immediately and folks could switch over to std2 (or whatever the package was) at a point that made sense for them. And especially if you're talking about range-related stuff, you'd be essentially splitting the D community and its code in two, because you'd almost certainly be dealing with incompatible libraries - especially if you're actually fixing some of the range-related problems. And that sort of thing just seems like it would be too destructive to be worth the risk. It's not quite a language split like you got with something like python 2 and python 3 or D1 and D2, but if the old and new standard libraries were not compatible, it would be close to that - it also seems too much like the whole Phobos vs Tango mess, even if the idea would be to be transitioning from one to the other rather than really having competing libraries (but as we've seen with python 2 vs 3 and D1 vs D2, major transitions like this tend to result in the old and the new competing even if that's not the intent).
>
> And honestly, for the most part, I don't think that redesigning the standard library would actually buy us much. Aside from range-related issues, everything I can think of is essentially a minor annoyance or wort. Sure, it would be nice if we could fix it, but it's not worth a massive upheaval of the D ecosystem. The only stuff that I can think of right now that would be a huge gain if we could change it is range-related stuff. I can think of several items there that would be nice to fix or at least look at changing:
>
> 1. Get rid of auto-decoding ranges and the concept of narrow strings. string and wstring would be ranges of char and wchar, everything operating on characters would then be operating at the code unit, code point, or grapheme level (depending on what was appropriate for the algorithm), and generic stuff would just operate on the elements regardless of whether they were code units, code points, or graphemes, leaving it to the programmer to handle it correctly.
>
> 2. Get rid of save and disallow classes as ranges. Postblit constructors then do what save does now. If you want a class to be a range, then wrap it in a struct, kind of like a smart pointer but for ranges.
>
> 3. Examine how to better deal with input ranges vs forward ranges and look at requiring that input ranges have reference semantics whereas forward ranges must have value semantics (as it stands, an input range _without_ reference semantics really should be a forward range, but forward ranges can still have reference semantics - hence the need for save).
>
> 4. Investigate adjusting the range hierarchy so that we have random-access ranges which aren't input ranges. I don't know for sure that that makes sense, but as I understand it, Mir does something like that, because it has a definite performance benefit under some circumstances.

Mir random access ranges (ndslices) are input ranges. The performance benefit comes from the way how they are implemented, constructed, and iterated.

Also, ndslice can be created in top of a range that have only one primitive opIndex. A user can define his own ndslices with all possible range primitives very fast - define a struct with opIndex method and use sliceField to construct ndslice on top of it.

Ilya

On Sat, Jun 03, 2017 at 02:16:16AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
> On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:
> > Especially the bit about throwing away years of accumulated programming work.
> 
> Our IRC discussion was more about simplifying the Phobos interface through breaking changes than actually just dropping and rewriting. Most the "phobos 2" would just be today's phobos with mistakes changed, autodecoding deprecated then removed, eager functions same deal (the discussion that started it was split vs splitter), static array interfaces removed (you'd have to explicitly slice at the call site instead of Phobos trying to two-layer ref them), etc.
> 
> So step one of this would be to copy existing code. Then cut the mistakes out.
> 
> The modified modules are just given a new package name (or something)
> to ease the transition.

Now that you put it this way, this suddenly seems a lot less disruptive and intimidating than it first sounded.

And it sounds more possible to actually use the deprecation cycle to gradually pull this off, as opposed to throwing out the baby with the bath water and restarting from ground zero.

And by putting the modified modules in a new temporary namespace, we can ease the transition enough that perhaps we might actually see this happen, one day. (Haha, so I hope.)


T

-- 
Some days you win; most days you lose.

On Saturday, 3 June 2017 at 06:37:27 UTC, H. S. Teoh wrote:
> And by putting the modified modules in a new temporary namespace, we can ease the transition enough that perhaps we might actually see this happen, one day. (Haha, so I hope.)

Javax ?

On Saturday, 3 June 2017 at 06:37:27 UTC, H. S. Teoh wrote:
> On Sat, Jun 03, 2017 at 02:16:16AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
>> On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:
>> > Especially the bit about throwing away years of accumulated programming work.
>> 
>> Our IRC discussion was more about simplifying the Phobos interface through breaking changes than actually just dropping and rewriting. Most the "phobos 2" would just be today's phobos with mistakes changed, autodecoding deprecated then removed, eager functions same deal (the discussion that started it was split vs splitter), static array interfaces removed (you'd have to explicitly slice at the call site instead of Phobos trying to two-layer ref them), etc.
>> 
>> So step one of this would be to copy existing code. Then cut the mistakes out.
>> 
>> The modified modules are just given a new package name (or something)
>> to ease the transition.
>
> Now that you put it this way, this suddenly seems a lot less disruptive and intimidating than it first sounded.
>
> And it sounds more possible to actually use the deprecation cycle to gradually pull this off, as opposed to throwing out the baby with the bath water and restarting from ground zero.
>
> And by putting the modified modules in a new temporary namespace, we can ease the transition enough that perhaps we might actually see this happen, one day. (Haha, so I hope.)
>
>
> T

Same will there come to be a need to replace Phobos 2 (when it happens) with 3 all because we prefer things to be renamed and relocated... when we've not fully discovered what phobos currently offers or improvements to be made.

stdx.data.json was not well documented last time I checked. Couldn't see any major benefit over std.json besides the performance claim. Besides, phobos is an MVP, there are more faster dub packages and nothing stops u from using them.

Does it really matter if it belongs in the std namespace?