On Wednesday, 19 August 2020 at 12:40:59 UTC, Adam D. Ruppe wrote:

> A copy of everything is an acceptable cost.

That's a rather unique position.
The only way that would work properly is to deprecate the non v2021 namespace.
And not develop it anymore.

Otherwise it'll diverge more and more.

On 8/19/20 8:40 AM, Adam D. Ruppe wrote:
> On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:
>> On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:
>>> Why not do it in std.v2021.*?
>>>
>>
>> What do you mean? Have a complete alternative phobos library?
>>
>> autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.
> 
> Andrei's suggestion is a good idea to allow forward massive changes without breaking any existing code.

I think it's an OK idea. But I don't want to wait forever for this to gel. And these PRs are not going to break code, it shouldn't affect anything, since I'm not disabling autodecoding at all except in a separate test.

When we get to the point where all of Phobos can be built without autodecoding, then we have to make the decision how to proceed with the breakage.

> The only regret is we didn't think to version it like this from the start.
> 
> A copy of everything is an acceptable cost.

The copy of everything is not acceptable to me. I had to change something like 53 lines of std.utf. That's about 1.5% of the 4k+ line file.

If we have to maintain 2 copies of std.utf, most of which are EXACTLY the same, it would be a nightmare. And that's one module.

The only way this works is if we split the files at the point of release (when no-autodecoding is completely ready), and only backport bug fixes to the prior version.

In this sense, it's better IMO to do this based on semantic versioning and git branches rather than copies of files.

-Steve

On 8/19/20 8:40 AM, Adam D. Ruppe wrote:
> On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:
>> On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:
>>> Why not do it in std.v2021.*?
>>>
>>
>> What do you mean? Have a complete alternative phobos library?
>>
>> autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.
> 
> Andrei's suggestion is a good idea to allow forward massive changes without breaking any existing code.
> 
> The only regret is we didn't think to version it like this from the start.
> 
> A copy of everything is an acceptable cost.

Don't copy everything, alias everything that doesn't need change and only add the declarations that should be changed.

The only problem I see with this is documentation. We need to have an easy way to define and navigate documentation for different versions of the stdlib.

That article by Yegge once more convinced me - we need to stop (thinking of) changing and start adding. We've been paralyzed by the fear of change while the opportunity (made even better by a nice module system) was always there.

On Wednesday, 19 August 2020 at 12:56:33 UTC, Stefan Koch wrote:
> That's a rather unique position.

It is a major version bump to indicate a split in backward compatibility. You'd treat it just like that in any other context; the module name just includes the x in semver's x.y.z.

This beats using a git branch because both can exist simultaneously. Suppose you import library old that uses the autodecoding version but you want to use library new that doesn't. No problem, both co-exist now.

Managing it can be tricky though, perhaps you would use a git branch internally to distribute and cherry-pick patches but then the release includes all the supported versions for simultaneous import. Though I suspect if we don't make a new one until the old one is frozen it won't be that much work anyway (at least if the compiler stops breaking code).

That's why it is v2021, indicating it is the year 2021 release. It gets frozen at some point in the year and then other breaking changes are put in v2022 which carries on the tradition.

On Wednesday, 19 August 2020 at 13:10:29 UTC, Andrei Alexandrescu wrote:
> Don't copy everything, alias everything that doesn't need change and only add the declarations that should be changed.

Yes, that too.

> The only problem I see with this is documentation. We need to have an easy way to define and navigate documentation for different versions of the stdlib.

My dpldocs.info can already pull any specific version you want from git:

http://phobos.dpldocs.info/v2.076.0/std.algorithm.html

just a fun fact. I've kinda wanted it to show automatic change logs in the past but haven't gotten around to that yet...

But anyway, with aliases, they'd be hyperlinked automatically (at least by my doc gen) and with different major version package namespaces, all would be available naturally.

I'm convinced we could make it work, if it doesn't already, it shouldn't be insurmountable tweaks to the doc generators.

> That article by Yegge once more convinced me - we need to stop (thinking of) changing and start adding. We've been paralyzed by the fear of change while the opportunity (made even better by a nice module system) was always there.

Glad to see others finally seeing this! Back before dub came out I argued we should use the module system for this and found no traction. I even gave up on doing it myself (but instead I simply don't break my code - the arsd library has had a total of 8 breaking changes going back a decade across all 70+ of its modules and most of them are trivial).

But it still remains on my mind as a viable thing to explore in more detail someday.

On 8/19/20 9:10 AM, Andrei Alexandrescu wrote:
> On 8/19/20 8:40 AM, Adam D. Ruppe wrote:
>> On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:
>>> On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:
>>>> Why not do it in std.v2021.*?
>>>>
>>>
>>> What do you mean? Have a complete alternative phobos library?
>>>
>>> autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.
>>
>> Andrei's suggestion is a good idea to allow forward massive changes without breaking any existing code.
>>
>> The only regret is we didn't think to version it like this from the start.
>>
>> A copy of everything is an acceptable cost.
> 
> Don't copy everything, alias everything that doesn't need change and only add the declarations that should be changed.

I've done exactly this for another library: converting mysql-native to @safe. See the result: https://github.com/mysql-d/mysql-native/pull/214. It's not as simple as you say. I had to split out the common parts into an "impl" package, and then create "safe" and "unsafe" packages. Not only that, but the unsafe ones are going to suffer from performance because everything underneath has to be safe. And this is a pretty unchanging library, not like Phobos.

Consider std.algorithm. Nearly all the functions are going to have to import a separate std.range, to get the no-autodecoding functionality (i.e. anything that depends on hasLength, hasIndexing, etc.). But the functions themselves will be identical. So it will be mostly a copy, with a few aliases.

I don't think this idea works without copying a large majority of code, whereas versioning it means changing very little.

> The only problem I see with this is documentation. We need to have an easy way to define and navigate documentation for different versions of the stdlib.

That is also a problem that I faced in mysql-native. I don't have a good solution.

> 
> That article by Yegge once more convinced me - we need to stop (thinking of) changing and start adding. We've been paralyzed by the fear of change while the opportunity (made even better by a nice module system) was always there.

The context of that article is somewhat similar, but not completely. With google services there is no using a previous version of their cloud service if they take it away. With the D compiler, you can alaways compile with previous versions until you can handle the new version. We aren't taking away the downloads of previous versions of the compiler. In fact, we can maintain older versions if we want to!

I think the better answer is to move to semantic versioning of the language, and have a scheduled "breaking" upgrade once a year (or longer), and commit to maintaining bug fixes for the previous major revision for a period of time. Give people the option of using older code if they want to keep some old package compiling and running. If they are actively maintaining it, then they likely need to upgrade.

-Steve

On 8/19/20 8:23 AM, Steven Schveighoffer wrote:
> On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:
>> On 8/16/20 1:00 PM, Steven Schveighoffer wrote:
>>> Hi all,
>>>
>>> TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595
>>
>> Why not do it in std.v2021.*?
>>
> 
> What do you mean? Have a complete alternative phobos library?

We discussed this here a couple of times before - just add a new package v2021 in std. Inside it, alias all declarations that you don't plan to change to the corresponding symbols in std, then add declarations for the things you do want to change.

It's like patching - you only add the deltas. It's simple and easy to set up.

> autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.

I wouldn't be so sure. I think there are large swaths of the stdlib that don't do anything with autodecoding. Like, threading stuff, parallel stuff, file stuff, numeric stuff, conversion stuff (sans strings), most of datetime stuff, and probably a ton more. I know there's likely to find some autodecoding parts in the above but the point is they are not about strings or autodecoding.

Even those that do can be arranged so you only write deltas. Let me pick one at random:

// Current code
auto cmp(R1, R2)(R1 r1, R2 r2)
if (isInputRange!R1 && isInputRange!R2)
{
    ... somewhere inside ...
    static if (isDynamicArray!R1 && isDynamicArray!R2
        && __traits(isUnsigned, E1) && __traits(isUnsigned, E2)
        && E1.sizeof == 1 && E2.sizeof == 1
        // Both or neither must auto-decode.
        && (is(immutable E1 == immutable char) == is(immutable E2 == immutable char)))
    {
        ... special case arrays of ubyte/char ...
    }
    else static if (!(isSomeString!R1 && isSomeString!R2))
    {
        ... not influenced by autodecoding ...
    }
    else
    {
        ... autodecode stuff ...
    }
}


// New code in std.v2021
auto cmp(R1, R2)(R1 r1, R2 r2)
if (isInputRange!R1 && isInputRange!R2)
{
    enum bool influencedByAutodecode = ...;
    static if (!influencedByAutodecode)
        return std.cmp(r1, r2);
    else
    {
        ... patch ...
    }
}

Even the patch can reuse std.cmp if you pass std.cmp a range of code units so you undo its penchant to autodecode.

* * *

Lastly, I should add the following. You wrote, and I quote it again because it is so important:

> autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.

This is a very powerful argument AGAINST your current approach to CHANGE things. You are basically saying the semantics of the standard library will be deeply affected by your work. Put another way, the rework of the standard library will be very incompatible with its current version. So you are killing all compatibility.

We don't want another Python 3 vs. Python 2. It almost killed Python, and they could afford to lose a lot more than we do. And let's face it - nobody is singing an ode to Python 3 now that they have Unicode built in. They solved no real problem and in the best of cases they made it marginally easy to do work that was very well done by a couple of specialized libraries.

Changing the stdlib to remove autodecoding would kill the D programming language.

You don't want to kill the D programming language, do you?

Stop changing.

Start adding.

On 8/19/20 9:21 AM, Adam D. Ruppe wrote:
> On Wednesday, 19 August 2020 at 13:10:29 UTC, Andrei Alexandrescu wrote:
>> Don't copy everything, alias everything that doesn't need change and only add the declarations that should be changed.
> 
> Yes, that too.
> 
>> The only problem I see with this is documentation. We need to have an easy way to define and navigate documentation for different versions of the stdlib.
> 
> My dpldocs.info can already pull any specific version you want from git:
> 
> http://phobos.dpldocs.info/v2.076.0/std.algorithm.html
> 
> just a fun fact. I've kinda wanted it to show automatic change logs in the past but haven't gotten around to that yet...
> 
> But anyway, with aliases, they'd be hyperlinked automatically (at least by my doc gen) and with different major version package namespaces, all would be available naturally.
> 
> I'm convinced we could make it work, if it doesn't already, it shouldn't be insurmountable tweaks to the doc generators.
> 
>> That article by Yegge once more convinced me - we need to stop (thinking of) changing and start adding. We've been paralyzed by the fear of change while the opportunity (made even better by a nice module system) was always there.
> 
> Glad to see others finally seeing this! Back before dub came out I argued we should use the module system for this and found no traction. I even gave up on doing it myself (but instead I simply don't break my code - the arsd library has had a total of 8 breaking changes going back a decade across all 70+ of its modules and most of them are trivial).
> 
> But it still remains on my mind as a viable thing to explore in more detail someday.

I thought of adding: "Cue plug of adrdox" to my message, but I knew it was unnecessary :o).

Versioning stdlib might require significant changes to ddoc or an opportunity for adrdox to differentiate itself.

On 8/19/20 9:06 AM, Steven Schveighoffer wrote:
> When we get to the point where all of Phobos can be built without autodecoding, then we have to make the decision how to proceed with the breakage.

That doesn't seem like a good gameplan to me because it's only THE ABSO!@#$#@!$!@#$LUTELY MOST IMPORTANT PART OF THE ENTIRE DEAL.

If you cut phobos open, do surgery after surgery on it, and only then think of how you're going to close the patient, you'll get some Frankenstein monster as a result.

STOP CHANGING.

START ADDING.

On Wednesday, 19 August 2020 at 12:47:46 UTC, Steven Schveighoffer wrote:
> Having a straightforward strategy as to how to migrate to non-autodecoding phobos would be key to getting this finalized.

Could warnings and/or deprecation messages for `.front` on arrays come to the rescue here? In order to adjust projects in advance for at least a year?