no-autodecode -- game plan

Hi all,

TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595


So what is the plan here? I have added hooks for dlang's CI so that any work we do on making phobos work without autodecoding does not go back to being broken.

As expected, I'm finding bugs. But I have probably the most relevant module passing Phobos unittests without autodecode: std.utf.

My plan is to pick a module or group of modules each week and spend some time getting it to pass.

Help is appreciated! So what it the process, you ask? I hope this is simple enough for everyone to follow:

1. set up your phobos dev environment (for now, I'm doing everything in the posix makefile, so it cannot be done via Windows -- if you want to fix this, please do!)
2. build phobos with no autodecoding:
NO_AUTODECODE=1 make -f posix.mak
3a. If you want to test a specific module:

NO_AUTODECODE=1 make -f posix.mak std/somemodule.test

3b. If you want to test all modules (and find one that breaks you can work on):

NO_AUTODECODE=1 make -f posix.mak unittest

4. Fix it so it passes. It should pass both with and without autodecoding. Try not to disable specific tests to get it to pass, we want no-autodecode phobos to be usable! Instead, change them to either not depend on autodecoding [1]  (preferred) or add a static if that tests both modes [2].

5. Add the modules that now pass to the posix.mak variable named NO_AUTODECODE_MODULES. Once this is merged, any changes to that module should have to pass without autodecoding.

The autodecode test is run in buildkite. PLEASE don't merge any PRs for Phobos if this test is failing! instead, ping me.

If you want to work on this with me, please let's coordinate either on slack or you can post here. We shouldn't be duplicating efforts (as much as possible), and Phobos is pretty big.

-Steve

[1] examples of places where I changed the test to be autodecode agnostic:
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L2401-R2402
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L3375-R3379

[2] examples of places where I configured tests to test both autodecode and non-autodecode:
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L3619-R3639

On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer wrote: > Hi all, > > TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595 > > [...] Awesome!

On 8/16/20 1:00 PM, Steven Schveighoffer wrote: > Hi all, > > TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595 Why not do it in std.v2021.*?

On 8/18/20 6:21 PM, Andrei Alexandrescu wrote: > On 8/16/20 1:00 PM, Steven Schveighoffer wrote: >> Hi all, >> >> TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595 > > Why not do it in std.v2021.*? > What do you mean? Have a complete alternative phobos library? autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything.

On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote: > On 8/18/20 6:21 PM, Andrei Alexandrescu wrote: >> On 8/16/20 1:00 PM, Steven Schveighoffer wrote: >>> Hi all, >>> >>> TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595 >> >> Why not do it in std.v2021.*? >> > > What do you mean? Have a complete alternative phobos library? > > autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything. I do believe he is joking .... but maybe he's not?

On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote: > On 8/18/20 6:21 PM, Andrei Alexandrescu wrote: >> Why not do it in std.v2021.*? >> > > What do you mean? Have a complete alternative phobos library? > > autodecoding infects so many other modules, so it would be really hard to single out only some modules, we'd likely have to make a copy of nearly everything. Andrei's suggestion is a good idea to allow forward massive changes without breaking any existing code. The only regret is we didn't think to version it like this from the start. A copy of everything is an acceptable cost.

On 8/19/20 4:45 AM, Per Nordlöw wrote: > On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer wrote: >> Hi all, >> >> TL;DR: I'm working on removing autodecoding from phobos. I have the first real PR: https://github.com/dlang/phobos/pull/7595 > > Will this in the long run affect user-code anyway? Yes, user code that depends on "xyz".front being dchar will be affected, etc. I expect that at some point soon (as I fix modules), we will get to a point where no-autodecode phobos is cohesive enough that we can start testing other projects. Then we will see the extent of the breakage. I don't really want to commit to saying how much there will be. My gut tells me there will be little breakages everywhere, but that are easily fixed. Having a straightforward strategy as to how to migrate to non-autodecoding phobos would be key to getting this finalized. > > And how will it affect compiled code? Considering that nearly all "optimizations" that concern autodecode go back to treating narrow strings as arrays, I'd say it's going to be a net positive in terms of performance. -Steve

Forums