A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.

We could do away with auto-decoding. Design it around using Andrei's allocators throughout. Make the GC optional from the start. Fix a few important naming conflicts and discrepancies.

There are problems, of course. Rampant code duplication is probably the biggest (though maybe public imports of identical code would help a lot).

I don't really expect this to go anywhere but I am curious to hear what changes you'd all like to see made to Phobos that can't happen because of backward compatibility. Also, how would you approach doing this? An on disk copy of Phobos with changes would not be an acceptable approach, I think.

On 2017-06-01 20:40, Brad Anderson wrote:

> I am curious to hear what changes you'd all like to see made to Phobos that can't happen because
> of backward compatibility.

Perhaps completely asynchronous IO throughout.

-- 
/Jacob Carlborg

On Thu, Jun 01, 2017 at 06:40:05PM +0000, Brad Anderson via Digitalmars-d wrote:
> A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.

+1. I'm for this.  There's just too much code in the existing Phobos that certain (wrong, in hindsight) design decisions are essentially impossible to fix.


> We could do away with auto-decoding.

+1000. ;-)


> Design it around using Andrei's allocators throughout. Make the GC optional from the start.

+1.


> Fix a few important naming conflicts and discrepancies.

As well as certain less-than-ideal names that we were forced to use because the good names have been taken up by not-so-nice implementations dating from before we had the hindsight to know what the best design was.


> There are problems, of course. Rampant code duplication is probably the biggest (though maybe public imports of identical code would help a lot).
> 
> I don't really expect this to go anywhere but I am curious to hear what changes you'd all like to see made to Phobos that can't happen because of backward compatibility.

1) A big one, not necessarily caused by backward compatibility but more by the sheer amount of code that would need to be fixed / rewritten, is the nasty hairball of a dependency graph among the current Phobos modules.

One particularly egregious example is std.format.  Due to the way it was originally implemented, it depended on a lot of other code (IIRC including std.bigint), so even if all you ever did was writefln("%d"), for example, your program would end up depending on std.bigint, std.complex, and all sorts of other stuff that never actually gets used. This in itself is not necessarily *that* bad, but what really makes it hurt is that many other modules depend on std.format. So you could be importing a seemingly completely-unrelated module, and it would pull in std.format, which in turn pulls in other stuff, and by the end you've basically imported about half of Phobos, with the associated executable bloat, even though you really only needed to call 1 function.

And the icing on the cake is that some of these complicated dependencies are circular, which every now and then cause mysterious linker errors and/or surprising dependence on declaration order, that mysteriously vanish when you move seemingly-unrelated code around.

If we were to go the route of Phobos2, I'd like the principle of pay-as-you-go to be baked into its design from the very beginning. If all you need is to format an int, it shouldn't also pull in std.complex and std.bigint. This can be done by compile-time format strings (which we now have, though it's currently just a frontend to the existing hairball code).  If function X and function Y are essentially independent of each other, they really ought to go into their own (sub)modules, now that we have package.d support in the compiler.

Of course, certain things like std.range.primitives and std.allocator will be depended on by 90% of the other modules, which is unavoidable, but which also means that we need to think VERY carefully about what we put into these foundational modules. If at all possible they should not depend on anything else.


2) A lesser item, perhaps would be to change the definitions of input / forward ranges, so that forward ranges are just by-value ranges, and input ranges are by-reference ranges, instead of forward ranges needing explicit calling of .save to save the iteration state. See:

http://lists.cubik.org/pipermail/digitalmars-d/2015-November/241712.html


3) An even lesser item, but now that we have a chance: clean up the mess
that is the current isSomeString, isNarrowString, etc.. Well, some of
them will be useless since we wouldn't have autodecoding, but still, we
need to sit down and properly design these traits so that they are (1)
useful, (2) orthogonal, and (3) have sensible names.


> Also, how would you approach doing this? An on disk copy of Phobos with changes would not be an acceptable approach, I think.

I'd say, start in a different namespace than std, so that the two can coexist until Phobos2 is ready to take over prime time. (And even then, Phobos1 should probably stick around for a while until people have migrated over.)  This new namespace can either be completely outside std, like std2 (bad name, I know), or it can be a dedicated sub-namespace, like std.new.

Preferably it should be something unique and easily greppable, so that when Phobos1 is finally gone, we can replace the std namespace with a simple search-and-replace.  But perhaps it would be best to just choose a new namespace with a nice name instead, like phobos.*, so that std.* will eventually just go the way of the dinosaur.

Once the namespace is sorted out, how we distribute the code should be pretty easy to decide.  It could start off as an independent github repo, perhaps, or a dub package or whatever, then when it becomes mature enough we could graft it into the current Phobos repo, and thereby make it available for all D users, and thus begin the gradual transition from std.*.


T

-- 
Not all rumours are as misleading as this one.

On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:
> A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.
>
> [...]
Maybe we do not have to replace, just expand on the line. Now the standard library is a lot of things are common, do not rely on GC。
I write a lib to expand the Phobos : https://github.com/dushibaiyu/yu

On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:
> A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.

Mir libraries are alternative to Phobos
https://github.com/libmir

> We could do away with auto-decoding. Design it around using

Primitives that do not perform auto-decoding.
http://docs.algorithm.dlang.io/latest/mir_array_primitives.html

> Andrei's allocators throughout. Make the GC optional from the start. Fix a few important naming conflicts and discrepancies.

It is a huge work.

Allocators are part of the Phobos, GC is part of the runtime.
So, allocators should be moved to runtime.

> There are problems, of course. Rampant code duplication is probably the biggest (though maybe public imports of identical code would help a lot).
>
> I don't really expect this to go anywhere but I am curious to hear what changes you'd all like to see made to Phobos that can't happen because of backward compatibility. Also, how would you approach doing this? An on disk copy of Phobos with changes would not be an acceptable approach, I think.

Mir uses separate dub packages for algorithmic and system parts. We are looking for new contributors.

Best,
Ilya

On Thursday, 1 June 2017 at 20:58:52 UTC, H. S. Teoh wrote:
> On Thu, Jun 01, 2017 at 06:40:05PM +0000, Brad Anderson via Digitalmars-d wrote:
>> A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.
>
> +1. I'm for this.  There's just too much code in the existing Phobos that certain (wrong, in hindsight) design decisions are essentially impossible to fix.
>
>
>> We could do away with auto-decoding.
>
> +1000. ;-)
>
>
>> Design it around using Andrei's allocators throughout. Make the GC optional from the start.
>
> +1.
>
>
>> Fix a few important naming conflicts and discrepancies.
>
> As well as certain less-than-ideal names that we were forced to use because the good names have been taken up by not-so-nice implementations dating from before we had the hindsight to know what the best design was.
>
>
>> There are problems, of course. Rampant code duplication is probably the biggest (though maybe public imports of identical code would help a lot).
>> 
>> I don't really expect this to go anywhere but I am curious to hear what changes you'd all like to see made to Phobos that can't happen because of backward compatibility.
>
> 1) A big one, not necessarily caused by backward compatibility but more by the sheer amount of code that would need to be fixed / rewritten, is the nasty hairball of a dependency graph among the current Phobos modules.
>
> One particularly egregious example is std.format.  Due to the way it was originally implemented, it depended on a lot of other code (IIRC including std.bigint), so even if all you ever did was writefln("%d"), for example, your program would end up depending on std.bigint, std.complex, and all sorts of other stuff that never actually gets used. This in itself is not necessarily *that* bad, but what really makes it hurt is that many other modules depend on std.format. So you could be importing a seemingly completely-unrelated module, and it would pull in std.format, which in turn pulls in other stuff, and by the end you've basically imported about half of Phobos, with the associated executable bloat, even though you really only needed to call 1 function.
>
> And the icing on the cake is that some of these complicated dependencies are circular, which every now and then cause mysterious linker errors and/or surprising dependence on declaration order, that mysteriously vanish when you move seemingly-unrelated code around.
>
> If we were to go the route of Phobos2, I'd like the principle of pay-as-you-go to be baked into its design from the very beginning. If all you need is to format an int, it shouldn't also pull in std.complex and std.bigint. This can be done by compile-time format strings (which we now have, though it's currently just a frontend to the existing hairball code).  If function X and function Y are essentially independent of each other, they really ought to go into their own (sub)modules, now that we have package.d support in the compiler.
>
> Of course, certain things like std.range.primitives and std.allocator will be depended on by 90% of the other modules, which is unavoidable, but which also means that we need to think VERY carefully about what we put into these foundational modules. If at all possible they should not depend on anything else.
>
>
> 2) A lesser item, perhaps would be to change the definitions of input / forward ranges, so that forward ranges are just by-value ranges, and input ranges are by-reference ranges, instead of forward ranges needing explicit calling of .save to save the iteration state. See:
>
> http://lists.cubik.org/pipermail/digitalmars-d/2015-November/241712.html
>
>
> 3) An even lesser item, but now that we have a chance: clean up the mess
> that is the current isSomeString, isNarrowString, etc.. Well, some of
> them will be useless since we wouldn't have autodecoding, but still, we
> need to sit down and properly design these traits so that they are (1)
> useful, (2) orthogonal, and (3) have sensible names.
>
>
>> Also, how would you approach doing this? An on disk copy of Phobos with changes would not be an acceptable approach, I think.
>
> I'd say, start in a different namespace than std, so that the two can coexist until Phobos2 is ready to take over prime time. (And even then, Phobos1 should probably stick around for a while until people have migrated over.)  This new namespace can either be completely outside std, like std2 (bad name, I know), or it can be a dedicated sub-namespace, like std.new.
>
> Preferably it should be something unique and easily greppable, so that when Phobos1 is finally gone, we can replace the std namespace with a simple search-and-replace.  But perhaps it would be best to just choose a new namespace with a nice name instead, like phobos.*, so that std.* will eventually just go the way of the dinosaur.
>
> Once the namespace is sorted out, how we distribute the code should be pretty easy to decide.  It could start off as an independent github repo, perhaps, or a dub package or whatever, then when it becomes mature enough we could graft it into the current Phobos repo, and thereby make it available for all D users, and thus begin the gradual transition from std.*.
>
>
> T

This could be done if a clear plan and layout was created before any real work was to start. Phobos should have the look and feel of .net IMO(or better, but not worse).


.Net is extremely well organized, logical, and useful.

On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:
> A (surely controversial) idea popped into my head while talking in #d on Freenode. The C++ guys are making an STL2 (the highlight of it being that it is range based). What about taking all the lessons learned from Phobos and creating a Phobos 2? It wouldn't replace the current version. You could import either in one program. It also wouldn't be a radical redesign. Most of Phobos could be used as is. What it would do is allow fixing some hard or impossible problems without losing backward compatibility.
>
> We could do away with auto-decoding. Design it around using Andrei's allocators throughout. Make the GC optional from the start. Fix a few important naming conflicts and discrepancies.
>
> There are problems, of course. Rampant code duplication is probably the biggest (though maybe public imports of identical code would help a lot).
>
> I don't really expect this to go anywhere but I am curious to hear what changes you'd all like to see made to Phobos that can't happen because of backward compatibility. Also, how would you approach doing this? An on disk copy of Phobos with changes would not be an acceptable approach, I think.

Frankly, I do not see the need for Phobos2. If you want to build alternative packages, just go ahead and publish them via dub like Mir, for example. You can even make a meta package, if you find yourself using the same group of packages all the time. Still, why would you call that meta package "Phobos2"? It only confuses people.

If you want to rewrite parts of the standard library, build the alternatives first and then we can adopt them piecewise.

Nevertheless, I would love to read a detailed analysis of Phobos and what should be improved. Please, write a blog post somewhere. However, do not mention "Phobos2".

D has a painful history with two competing standard libraries. If you seriously propose this path, I hope Andrei and Walter will publicly and vehemently oppose it. Otherwise that ghost from the past becomes a PR disaster for D.

On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
> Frankly, I do not see the need for Phobos2. If you want to build alternative packages, just go ahead and publish them via dub like Mir, for example. You can even make a meta package, if you find yourself using the same group of packages all the time. Still, why would you call that meta package "Phobos2"? It only confuses people.
>
> If you want to rewrite parts of the standard library, build the alternatives first and then we can adopt them piecewise.

Makes sense. I've said in the past that the standard library should be smaller now that we have Dub. The standard library should serve as a common framework that lets libraries interoperate easier.

If an effort was undertaken it should probably not be called Phobos2 until such time (if ever) that it becomes Phobos2 by community vote and Walter and Andrei approval.

C++'s Boost has a problem where libraries people are working on get called Boost Whatever in anticipation of being proposed for Boost and many of them never even make it to the review stage so it becomes confusing about what is and isn't in Boost. It's also confusing because competing libraries will be proposed and a non-Boost library is already taking Boost ObviousName so they have to come up with some weird name instead.

> Nevertheless, I would love to read a detailed analysis of Phobos and what should be improved. Please, write a blog post somewhere. However, do not mention "Phobos2".

This is the heart of why I made the post. I want to read what people want myself.

> D has a painful history with two competing standard libraries. If you seriously propose this path, I hope Andrei and Walter will publicly and vehemently oppose it. Otherwise that ghost from the past becomes a PR disaster for D.

Not very seriously. I'm more interested in the discussion than anything. Like I said, I don't think this will go anywhere.

I do want to say that I don't think the Phobos/Tango situation is relevant. Those were two competing, mutually exclusive standard libraries. That problem has long since been resolved with people using Tango and Phobos together in their D2 programs just fine. I understand why anyone would be weary at the idea though.

On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
[...]
> D has a painful history with two competing standard libraries. If you seriously propose this path, I hope Andrei and Walter will publicly and vehemently oppose it. Otherwise that ghost from the past becomes a PR disaster for D.
[...]

I think this is overreacting.  "Phobos 2" is not supposed to be a *competing* library, but, as I see it, more like an alpha version of the next "major iteration" of Phobos. The "next major version" with a brand new paradigm, if you will, as opposed to incremental changes to the "current major version".


T

-- 
Give me some fresh salted fish, please.

On 06/02/2017 01:43 PM, H. S. Teoh via Digitalmars-d wrote:
> On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
> [...]
>> D has a painful history with two competing standard libraries. If you
>> seriously propose this path, I hope Andrei and Walter will publicly
>> and vehemently oppose it. Otherwise that ghost from the past becomes a
>> PR disaster for D.
> [...]
> 
> I think this is overreacting.  "Phobos 2" is not supposed to be a
> *competing* library, but, as I see it, more like an alpha version of the
> next "major iteration" of Phobos. The "next major version" with a brand
> new paradigm, if you will, as opposed to incremental changes to the
> "current major version".

A clean slate is alluring, and there are several things that can be done differently in Phobos, as there are in any project that's been around for a while. It may, however, be difficult to find enough people able and willing to take such a large project off the ground. There are plenty of great things that can be done with the standard library, ranging from the trivial - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to the most creative.

As an example, eliminating phobos' and druntime's reliance on "static this" would improve conviviality of D in mixed-language projects, but we have yet to find folks to do the work. To the best of my knowledge only David, Stanislav, and myself have been involved so far.


Andrei