On Saturday, 30 March 2019 at 01:10:59 UTC, H. S. Teoh wrote:
> And a big part of this is effective communication of the high-level goals of the project and what's expected of would-be contributions.

I'm hoping we can set a lot of that high level goals at the dconf AGM (along with a lot of other problems). I'll be publishing the (draft, to be added to by others) agenda soon.

On Thursday, March 28, 2019 2:05:41 PM MDT Andrei Alexandrescu via Digitalmars-d wrote:
> On 3/28/19 3:43 PM, ag0aep6g wrote:
> > I've started that endeavor, because the only tangible response I got when I asked what do do was from Jonathan M Davis who said: "it could be argued that generic, range-based functions simply shouldn't ever be assigning one range to another."[1]
>
> This is a sign things have gotten off the rail. Assignment should be accessible and have obvious semantics.

That's really the problem. Some operations on ranges are not well defined, and so you can't rely on them in generic code. The prime example is copying (though assignment then gets tied up in that). If a range is a reference type, then iterating the copy iterates the original; if a range is a value type, then iterating the copy doesn't affect the original; and if a range is a pseudo-reference type, then exactly what happens depends on how it's implemented. The end result is that you basically can't use a range after it's been copied (which includes passing it to a function or using it with foreach), because what happens depends on how the range was implemented.

I've thought for a while now that how things should really work would be to require that basic input ranges be reference types and that forward ranges be value types (ditching save, requiring copy construction instead, which then requires wrapping classes in structs to be ranges), though that potentially means that calling the same function on basic input ranges and forward ranges gets a bit sticky. The biggest problem though of course is compatibility, but if we're seriously looking at doing a v2 for major stuff in Phobos or Phobos as a whole, then we definitely should be sitting down and working through what we did wrong with ranges and fix this sort of thing.

- Jonathan M Davis

On Thursday, March 28, 2019 11:04:58 AM MDT Andrei Alexandrescu via Digitalmars-d wrote:
> On 3/28/19 9:16 AM, ag0aep6g wrote:
> > On 28.03.19 14:05, Andrei Alexandrescu wrote:
> >> Then some ranges are not meant to be assignable.
> >
> > Should Phobos be compatible with those ranges?
>
> A variety of algorithm and data structures in Phobos are relying on assignment. Presumably a good part of them can be converted with ease to use single assignment, but not all.
>
> The overall message is we got bogged down on the "wrong" side of generality - cross-cutting and nonscalable code additions to support unprincipled and low-impact corner cases.
>
> Part of that is we've been cagey about defining copy and assignment semantics of ranges in a simple and comprehensive manner. It seems to me going with these is the right thing:
>
> * Input ranges are copyable and assignable, and have pointer semantics (all copies refer to the same underlying position, and advancing one advances all others).
>
> * Forward ranges are copyable and assignable, but distinct copies refer to distinct positions in the range such that advancing one does not advance the others.
>
> * We don't support other semantics.

I've thought for a while now that this is what we should do if we could start over. The problem has been that it breaks a ton of code, and a migration path is difficult. Doing a v2 version of Phobos would really help with making fixes like that - though then we have to deal with stuff like whether a library is written to work with v1 or v2, and with stuff like ranges effectively working via duck typing, that could get a bit interesting. If we can do it without splitting the community though, it would allow us to fix some of the big mistakes we made with our core stuff (auto-decoding being another).

- Jonathan M Davis

On Thursday, March 28, 2019 11:42:50 AM MDT H. S. Teoh via Digitalmars-d wrote:
> On Thu, Mar 28, 2019 at 01:04:58PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
>
> > Part of that is we've been cagey about defining copy and assignment semantics of ranges in a simple and comprehensive manner. It seems to me going with these is the right thing:
> >
> > * Input ranges are copyable and assignable, and have pointer semantics (all copies refer to the same underlying position, and advancing one advances all others).
> >
> > * Forward ranges are copyable and assignable, but distinct copies refer to distinct positions in the range such that advancing one does not advance the others.
> >
> > * We don't support other semantics.
>
> What about classes that wrap ranges? E.g., std.ranges.interfaces.  Since classes are reference types, this would mean it's impossible to use those interfaces with forward ranges or above, which is a show-stopping limitation (e.g., if you need to return two different range types selected at runtime -- the current solution is to wrap them in the appropriate class using std.ranges.interfaces).

It's not a problem for basic input ranges, because they'd then have reference semantics. The problem would just be forward ranges, and the solution to that is to use wrapper ranges which are structs. Then they can define a copy constructor which gives value semantics in place of save.

The reality of the matter though is that code should be _really_ wary of using classes for forward ranges, because it requires calling new a lot, and it's really inefficient. So, while we probably should support it for those use cases that really need it, if code doesn't really need it, it shouldn't do it. For instance, I tested dxml with a whole range of range types (including ranges that were classes) when benchmarking, and performance tanked when using ranges that were classes because of all of the memory allocation that was happening.

- Jonathan M Davis

On Thursday, March 28, 2019 3:13:17 PM MDT ag0aep6g via Digitalmars-d wrote:
> On 28.03.19 21:05, Andrei Alexandrescu wrote:
> > We must excise the cancerous tissue.
>
> We can cut out the bad parts, but leave the useful core of RefRange intact, can't we?
>
> The useful core is: a range that wraps a pointer to another range in order to achieve reference semantics. Often, a pointer to a range is already a range, but not always. E.g., `int[]*` is not a range.

The problem is that reference semantics and forward ranges do not go well together at all. As things stand, range-based code is supposed to work with forward ranges that are reference types, and we have save to try and deal with the fact that some ranges are value types, some are reference types, and some are pseudo-reference types, but it really doesn't work very well. And algorithms that require save fundamentally need to copy the range at some point, pretty much destroying the usefulness of something like RefRange.

Basic input ranges are fundamentally reference types or pseudo-reference types, because if they could have value semantics, then they could be forward ranges. Forward ranges fundamentally have to be copyable with value semantics to even work properly. We've just got this messed up situation where we're using save to do the copy instead of postblit or copy constuctors, because we're trying to handle types that aren't value types as if they were.

RefRange is trying to force reference semantics, which then fundamentally doesn't fit with forward ranges. It sort of half fits right now, because of save, but as with forward ranges that are reference types, it causes problems, and code frequently isn't going to work with it without extra work.

As such, I really don't think that RefRange makes sense fundamentally. It _can_ work in corner cases, and years ago, I had a piece of code where it was really useful, which is why I wrote it. But it was a mistake. It's trying to fight against how forward ranges work, and while it might occasionally be useful, in general, it's going to cause problems.

And if we actually go forward with requiring that basic input ranges have reference semantics and that forward ranges have value semantics like Andrei is proposing, then RefRange _really_ doesn't work.

- Jonathan M Davis

On Thursday, March 28, 2019 12:47:49 PM MDT Andrei Alexandrescu via Digitalmars-d wrote:
> On 3/28/19 2:41 PM, Andrei Alexandrescu wrote:
> > On 3/28/19 2:30 PM, H. S. Teoh wrote:
> >> The only way I can see this happening is if we start a new iteration of Phobos, like std.v2.*
> >
> > There'd be carefully carved subsets per capability, such as std.v2.betterc.*, std.v2.nogc.*, std.v2.noexcept.*, std.v2.safe.
> >
> > The capability sets can be combined in alphabetic ordering. E.g. std.v2.noexcept.nogc.* has the subset that uses no gc and no exceptions, but std.v2.nogc.noexcept does not exist.
> >
> > Through the magic of public imports and aliasing there would be no code duplication. Introspection could help, too, a la "define public aliases for all @safe definitions in that module". Some improvements to ddoc might be needed to "see" through aliases.
>
> Oh, and druntime must go.
>
> The whole distinction between the runtime library and the standard library is clowny and has more to do with poorly ported tradition from other languages, than with anything else.
>
> We need one standard library that is entirely pay-as-you-go (e.g. empty
> main() means no library file is ever needed for linking) and that offers
> an opt-in continuum.

LOL. That's a big thing to drop in as an afterthought. There are some useful organizational aspects of having druntime, but it has become a bigger and bigger problem over time as we've tried to do stuff that's required to be in druntime but which effectively requires stuff that's in Phobos - a long standing example of that being UTF encoding and decoding. AFAIK, there are still a number of hurdles to truly making it pay-as-you-go, but if we can do it, it would be great.

This would also be a great opportunity to fix some of the issues with shared in druntime (in particular with the mutex and conditional variable stuff as well as Thread). Too much of it doesn't use shared properly (if at all), and in the case of Thread, it has @safe functions that end up passing thread-local objects to a new thread, which would be much easier to do with something like v2 or moving all of that to std (certainly, there are things that should be fixed there that we can't fix in-place).

All in all though, I think that we're going to have to plan carefully with how we go forward with major changes like this so that we minimize the risk of splitting the community. With D's module system, we have a lot of flexibility, but when we start replacing core code and concepts with new, incompatible versions in separate modules, we do risk compatibility problems. However, full-sale replacement of some modules would definitely fix some of the issues that replacing stuff in pieces within a module has.

- Jonathan M Davis

On Thursday, March 21, 2019 10:11:04 AM MDT Joseph Rushton Wakeling via Digitalmars-d wrote:
> On Wednesday, 20 March 2019 at 19:45:13 UTC, Steven Schveighoffer
>
> wrote:
> > On 3/20/19 1:41 PM, Kagamin wrote:
> >> struct S
> >> {
> >>
> >>     size_t length() const { return 0; }
> >>
> >> }
> >>
> >> auto ref property(T)(auto ref T a){ return a; }
> >>
> >> void f()
> >> {
> >>
> >>     S s;
> >>     assert(s.length == 0); //OK
> >>     static assert(is(typeof(property(s.length))==size_t));
> >>
> >> }
> >>
> >> Can this work?
> >
> > Yep, that's a good idea actually. Make it inout, to cut down on template bloat.
> >
> > I actually use something less nice in iopipe[1], so I may switch to this.
>
> Rather than define a new `property` symbol, why not rewrite the template constraint for `empty` as:
>
>
> @property bool empty (T) (auto ref scope T a)
>      if(is(ReturnType!((T t) => t.length) : size_t))
> {
>      return !a.length;
> }
>
> This matches what is done in the `isInputRange` check for the existence of `empty` (I suspect it's done that way because it's quite common for `SomeRange.empty` to be an enum rather than a method or property).

Honestly, this is another example of over-generality causing problems. length shouldn't ever be anything other than size_t. Anything else causes problems.

- Jonathan M Davis

On 30.03.19 19:32, Jonathan M Davis wrote:
> RefRange is trying to force reference semantics, which then fundamentally
> doesn't fit with forward ranges. It sort of half fits right now, because of
> save, but as with forward ranges that are reference types, it causes
> problems, and code frequently isn't going to work with it without extra
> work.
> 
> As such, I really don't think that RefRange makes sense fundamentally.

You haven't spelled it out, but I suppose you also think that classes don't make sense as forward ranges. And Andrei has stated something to the effect that they wouldn't be supported in his redesign.

That is fine for a range redesign. But with the definitions we have right now, classes can be forward ranges, and Phobos should work with them. And if Phobos works with classes, it can work with RefRange (if we cut out opAssign).

If RefRange really is fundamentally at odds with current ranges, I'd be interested in an example where it causes trouble that isn't due to opAssign (or opSlice() which also shouldn't be there). Maybe I just fail to see the problem.

One issue I'm aware of is that RefRange does a GC allocation on `save`. That isn't pretty, but I wouldn't call it fatal.

I've said before that "we can cut out the bad parts" of RefRange, referring to opAssign, but I think I was wrong. Deprecating assignment would be so limiting that we might as well deprecate the whole thing.

Then again, Walter has made (limited) efforts to support ranges that are non-assignable due to const fields. Nobody acknowledges that those exact changes also work to accommodate RefRange, but they do.

On 3/30/2019 12:38 PM, ag0aep6g wrote:
> Then again, Walter has made (limited) efforts to support ranges that are non-assignable due to const fields. Nobody acknowledges that those exact changes also work to accommodate RefRange, but they do.

I'm fine with adjusting `save()` implementations to use construction rather than assignment semantics, because that's the way `save()` is defined to behave (although it wasn't clearly spelled out).

On 3/30/19 8:05 PM, Walter Bright wrote:
> On 3/30/2019 12:38 PM, ag0aep6g wrote:
>> Then again, Walter has made (limited) efforts to support ranges that are non-assignable due to const fields. Nobody acknowledges that those exact changes also work to accommodate RefRange, but they do.
> 
> I'm fine with adjusting `save()` implementations to use construction rather than assignment semantics, because that's the way `save()` is defined to behave (although it wasn't clearly spelled out).

Great, so ag0aep6g you're good to go with your PR. I'd approved it yesterday.