On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis wrote:
> Considering that a random access range is essentially an abstraction for a dynamic array and that ranges were designed with that in mind, I don't know how you can argue that dynamic arrays shouldn't be treated as ranges.
>
> The auto-decoding is certainly an issue, but that only affects ranges of char and wchar. The vast majority of dynamic arrays are perfectly normal ranges.
>
> - Jonathan M Davis

Phobos' range facilities vomit when you try to deal with static arrays, and they have to be coerced into dynamic arrays before they can be dealt with. This is silly.

It gets under my skin that length and opIndex and opSlice produce different results in phobos' ranges depending on one's current position in the range. This doesn't make sense to me, and the only reason I can conceive of it having become how ranges work throughout phobos is because that's how dynamic arrays work if you force them to act as though they were ranges.

In every single other language I've used, the concept of an Iterable and an Iterator are distinct and very separate. An Iterator is something that can be iterated over; an Iterable is something which can produce an Iterator for iterating over its contents. In D, arrays are Iterables, and phobos endeavors to force them to be Iterators as well. It defies years of basic design wisdom regarding how to differentiate a collection and the means by which one enumerates the items in that collection.

Arrays are Iterables which should be able to produce an Iterator, in D's case a range. They should not themselves be Iterators.

"Poisoned m&ms are certainly an issue, but that only affects people who unwittingly eat them. You see, the vast majority of m&ms are perfectly innocuous."

On Sunday, September 25, 2016 10:58:24 pineapple via Digitalmars-d wrote:
> On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis
>
> wrote:
> > Considering that a random access range is essentially an abstraction for a dynamic array and that ranges were designed with that in mind, I don't know how you can argue that dynamic arrays shouldn't be treated as ranges.
> >
> > The auto-decoding is certainly an issue, but that only affects ranges of char and wchar. The vast majority of dynamic arrays are perfectly normal ranges.
> >
> > - Jonathan M Davis
>
> Phobos' range facilities vomit when you try to deal with static arrays, and they have to be coerced into dynamic arrays before they can be dealt with. This is silly.

Except that static arrays are bona fide containers, and their length can't be mutated. They don't make any sense as ranges at all.

> It gets under my skin that length and opIndex and opSlice produce different results in phobos' ranges depending on one's current position in the range. This doesn't make sense to me, and the only reason I can conceive of it having become how ranges work throughout phobos is because that's how dynamic arrays work if you force them to act as though they were ranges.

It's because ranges are effectively a sliding window over whatever they're iterating over. The only way that it would make sense for slicing or indexing a range to refer to anything other than what the range currently refers to is if there were some sort of container that they were a slice of, and the vast majority of ranges aren't that way. They're not designed to refer to anything other than what they currently refer to, and there is no original for the indices to refer back to. A range is simply a list of values. It may or may not be randomly accessible, and it may or may not refer to anything else like a container. It could be purely generative (e.g. a range over the fibonacci sequence or an infinite list of random numbers). It really makes no sense for indexing or slicing ranges to do anything but have the indices treat the current front of the range as index 0. I can understand that that might be confusing at first if you're thinking of them as specifically refering to elements of a container, but many ranges have nothing to do with a container at all.

> In every single other language I've used, the concept of an Iterable and an Iterator are distinct and very separate. An Iterator is something that can be iterated over; an Iterable is something which can produce an Iterator for iterating over its contents. In D, arrays are Iterables, and phobos endeavors to force them to be Iterators as well. It defies years of basic design wisdom regarding how to differentiate a collection and the means by which one enumerates the items in that collection.
>
> Arrays are Iterables which should be able to produce an Iterator, in D's case a range. They should not themselves be Iterators.

Dynamic arrays aren't really containers in D. Unlike containers, they don't own or manage their own memory. They're only slices of that memory. The fact that you can append to them, and the GC will then increase the amount of memory that the dynamic array refers to (possibly allocating a new block of memory and changing which block of memory the dynamic array is a slice of) certainly muddies things, but it's still not the case the dynamic array owns or manages its memory. What owns or manages the memory depends entirely on what allocated the memory (usually the GC, but not always). Even in the case of concatention or appending, its the GC that manages the memory, not the dynamic array (e.g. a dynamic array which is a slice of malloced memory is going to result in a memory leak if you then append to it, and nothing else was keeping track of that memory; the array itself doesn't know or care who owns or manages its memory). And multiple dynamic arrays can refer to exactly the same block of memory, which is a clear indicator that they don't own or manage their own memory. And in that respect, they're definitely more like iterators than containers.

So, yes, dynamic arrays in D are weird hybrids, but aside from operations related to concatenation or appending, they don't act like containers at all. They act much more like a pair of iterators being passed around together. And both the language and Phobos are very consistent about that.

The one place with dynamic arrays and ranges that is totally wacko is narrow strings, because they're auto-decoded, because then Phobos tries to stop treating them like dynamic arrays. If it didn't do that, AFAIK it would be completely consistent with how it treated dynamic arrays.

I think that you'll have an easier time understanding dynamic arrays in D and ranges in general if you stopped trying to treat dynamic arrays as if they were containers, since they really aren't. And the few operations that make them sort of act like containers are not part of the range API and therefore don't conflict with ranges at all.

- Jonathan M Davis

On Sunday, 25 September 2016 at 10:58:24 UTC, pineapple wrote:

> It gets under my skin that length and opIndex and opSlice produce different results in phobos' ranges depending on one's current position in the range. This doesn't make sense to me, and the only reason I can conceive of it having become how ranges work throughout phobos is because that's how dynamic arrays work if you force them to act as though they were ranges.

But which opIndex and which length? Those of the container, or those of the range? It doesn't make any sense to expect container[].takeExactly(7).length to be different than 7. If range.length returns the length of the container this would break every sensible algorithm out there. How would you even implement binary search (https://rosettacode.org/wiki/Binary_search#D)? It's completely wrong to expect indexing operations to work on the original container. What you do in situations where there is no container, such as generators (e.g. iota https://github.com/dlang/phobos/blob/v2.071.2/std/range/package.d#L4722) and data coming from the network? And how would you get the number of remaining elements in the range?

> Phobos' range facilities vomit when you try to deal with static arrays, and they have to be coerced into dynamic arrays before they can be dealt with. This is silly.

What's the problem here? Just slice them using the [] syntax (i.e. the analog of asrange in your library).

> In every single other language I've used, the concept of an Iterable and an Iterator are distinct and very separate. An Iterator is something that can be iterated over; an Iterable is something which can produce an Iterator for iterating over its contents. In D, arrays are Iterables, and phobos endeavors to force them to be Iterators as well. It defies years of basic design wisdom regarding how to differentiate a collection and the means by which one enumerates the items in that collection.
>
> Arrays are Iterables which should be able to produce an Iterator, in D's case a range. They should not themselves be Iterators.

There's seems to be a misunderstanding what are D's ranges and how they're meant to be used. Containers such as static arrays (i.e T[n]) are Iterable-s whereas slices (i.e. T[]) are Iterator-s by your taxonomy. You get the Iterator from the iterable static array using the opSlice (i.e. []) method. Ranges (and in particular slices) are just positions in an array. The same analogy holds for the containers in std.container: you get a range using the container[], or container.opSlice() syntax. Regardless where the range points to and if it shrinks, the container.length stays the same, assuming you haven't added or removed any elements.

I guess your misunderstanding stems from the fact that you call T[] arrays. This however is wrong. T[] is just slice of an array. If you want to use arrays similar to those in other languages, use std.container.array : Array (http://dlang.org/phobos/std_container_array).

D's built-in dynamic arrays are hidden from you and you only get to interact with them by referring to their elements by using slices. For example, have a look at the following code:

```
import std.algorithm.comparison : equal;

int[] arr = [1, 2, 3, 4];

void append5(int[] s)
{
    s ~= 5; // 1)
}

append5(arr);

assert (arr.equal([1, 2, 3, 4]));
```

As you can see in the example above, since int[] is just a slice/range, appending to it does not modify the underlying container. Instead a new container is created (*) and `s` is made to point to it in 1). If you were to use a "proper" container such as std.container.array, the results would be different:

```
import std.algorithm.comparison : equal;
import std.container.array;

auto arr = Array!int([1, 2, 3, 4]);

void append5(Array!int s)
{
    s ~= 5;
}

append5(arr);

assert (arr[].equal([1, 2, 3, 4, 5]));
```

For more information, I strongly suggest reading Steven's article : http://dlang.org/d-array-article.html (*)

On Sunday, 25 September 2016 at 13:10:42 UTC, ZombineDev wrote:
> But which opIndex and which length? Those of the container, or those of the range? It doesn't make any sense to expect container[].takeExactly(7).length to be different than 7. If range.length returns the length of the container this would break every sensible algorithm out there. How would you even implement binary search (https://rosettacode.org/wiki/Binary_search#D)? It's completely wrong to expect indexing operations to work on the original container. What you do in situations where there is no container, such as generators (e.g. iota https://github.com/dlang/phobos/blob/v2.071.2/std/range/package.d#L4722) and data coming from the network? And how would you get the number of remaining elements in the range?

The argument is not that the length of a range should always evaluate to the length of some container it enumerates, but that the length of the range should not change based on your position in the range. In phobos, popFront effectively reduces the length of the range by one, and offsets all further calls to opIndex and opSlice. I expect length, opIndex, and opSlice to behave the same whether I just built the range, am halfway through the range, or have fully consumed the range.

>> Phobos' range facilities vomit when you try to deal with static arrays, and they have to be coerced into dynamic arrays before they can be dealt with. This is silly.
>
> What's the problem here? Just slice them using the [] syntax (i.e. the analog of asrange in your library).

As Jonathan pointed out there's a lot of muddying of how dynamic arrays are only sorta-kinda genuine arrays. But when using them one should be able to rely on dynamic and static arrays have the same interface, except that one can be resized and appended to and another cannot. Having to slice static arrays to pass them into phobos' HOFs is icky and completely unnecessary. And though you might say that you could just append `[]` to every array you pass in, but while it will work for static and dynamic arrays it won't work for everything that can be passed to a function accepting a range.

Consistency of syntax for performing the same operation upon many different types is not to be undervalued.

On Sunday, 25 September 2016 at 11:48:38 UTC, Jonathan M Davis wrote:
> It's because ranges are effectively a sliding window over whatever they're iterating over.

I think this is the difference in perception - ranges do not _have_ to be sliding windows, they can just as well be windows that don't move. I find the latter approach to be more workable and intuitive that the former, and in almost every case the approach results in more concise and performant code for performing those operations.

When would I ever want `range[0]` to give me the same thing as `range.front`? I want it to give me the first item in the range's view at the time of its creation.

I strongly oppose any change that makes the former approach an inherent part of D. I honestly don't think the latter approach should be an inherent part of D, either, but I'd have a hell of a time trying to use the language if it wasn't an option.

On 9/25/16 12:58 PM, pineapple wrote:
> On Sunday, 25 September 2016 at 04:06:41 UTC, Jonathan M Davis wrote:
>> Considering that a random access range is essentially an abstraction
>> for a dynamic array and that ranges were designed with that in mind, I
>> don't know how you can argue that dynamic arrays shouldn't be treated
>> as ranges.
>>
>> The auto-decoding is certainly an issue, but that only affects ranges
>> of char and wchar. The vast majority of dynamic arrays are perfectly
>> normal ranges.
>>
>> - Jonathan M Davis
>
> Phobos' range facilities vomit when you try to deal with static arrays,
> and they have to be coerced into dynamic arrays before they can be dealt
> with. This is silly.

Yah, it comes as a surprise to many that static arrays are more akin to structs than to arrays. They really are records that happen to have several elements of the same type. Providing indexing for them is a convenience that somewhat adds to the confusion, and converting to dynamic arrays is unsafe because it essentially escapes the innards of the struct. Statically-sized arrays are odd, that's for sure, and that's the case in C and C++ as well.

> It gets under my skin that length and opIndex and opSlice produce
> different results in phobos' ranges depending on one's current position
> in the range. This doesn't make sense to me, and the only reason I can
> conceive of it having become how ranges work throughout phobos is
> because that's how dynamic arrays work if you force them to act as
> though they were ranges.

Better get used to it. Ranges are a generalization of D's slices, not the other way around.

> In every single other language I've used, the concept of an Iterable and
> an Iterator are distinct and very separate. An Iterator is something
> that can be iterated over; an Iterable is something which can produce an
> Iterator for iterating over its contents. In D, arrays are Iterables,
> and phobos endeavors to force them to be Iterators as well. It defies
> years of basic design wisdom regarding how to differentiate a collection
> and the means by which one enumerates the items in that collection.

Ranges don't need necessarily an associated Iterable. This is the case in other languages, too; Lisp/Scheme/Haskell/etc lists are iterables and at the same time their own iterators. But indeed std.container is designed to have containers distinct from their associated ranges, which raises the interesting question what should happen if a range gets "orphaned", i.e. it's still active after its container ceases to exist.

> Arrays are Iterables which should be able to produce an Iterator, in D's
> case a range. They should not themselves be Iterators.

Yah, std.container.Array follows that. It's not an be-all-end-all of design though.


Andrei

On Sunday, September 25, 2016 13:10:42 ZombineDev via Digitalmars-d wrote:
> D's built-in dynamic arrays are hidden from you and you only get to interact with them by referring to their elements by using slices.

That's a common misconception propagated by Steven's otherwise excellent article. As far as the official language spec goes, T[] _is_ the dynamic array. What backs it is irrelevant as far as that goes. It's just that if it's backed by the GC, then when you append to it, the GC might not have to allocate a new memory buffer for the array. All of the operations on a dynamic array work the same regardless of what backs it, and focusing on the memory buffer being the array risks causing you problems when you have to deal with dynamic arrays backed by something else.

But on the whole, what you said was right. It's just the terminology that's off.

- Jonathan M Davis

On Sunday, September 25, 2016 13:40:14 pineapple via Digitalmars-d wrote:
> On Sunday, 25 September 2016 at 11:48:38 UTC, Jonathan M Davis
>
> wrote:
> > It's because ranges are effectively a sliding window over whatever they're iterating over.
>
> I think this is the difference in perception - ranges do not _have_ to be sliding windows, they can just as well be windows that don't move. I find the latter approach to be more workable and intuitive that the former, and in almost every case the approach results in more concise and performant code for performing those operations.
>
> When would I ever want `range[0]` to give me the same thing as `range.front`? I want it to give me the first item in the range's view at the time of its creation.

You're going to be better off if you stop trying to think like there's some sort of original range here. What you have right now is what you have. The first element is index 0, and whether the range was just created or whether it's had a million elements popped off the front is irrelevant. If you want access to elements that aren't currently in the range, then you need to save the range and keep that around so that you can access _that_ range instead of the one that you currently have. Remember that many ranges are generative and aren't backed by any sort of container and that there is no original range that's been iterated over, and index 0 couldn't have anything to refer to except for the current front.

And if you don't like how D's ranges are designed, sorry, but it's never
been like what you're suggesting, and changing it now would break a ton of
code even if we agreed that it were a good idea, and you're not going to get
that agreement. What we have with ranges right now is by no means perfect,
but the whole design is based on the sliding window concept - just like
D's dynamic arrays are - and that has worked fantastically for us.

> I strongly oppose any change that makes the former approach an inherent part of D. I honestly don't think the latter approach should be an inherent part of D, either, but I'd have a hell of a time trying to use the language if it wasn't an option.

The way it works now is how it's always worked with dynamic arrays and ranges in D. If you're trying do anything else, you're just going to run into problems in the long run - particularly when interacting with code written by anyone else. So, while you're obviously free to do whatever you want with your own code, don't expect Phobos or D code in general to change how ranges fundamentally work.

- Jonathan M Davis

On Sunday, 25 September 2016 at 13:45:01 UTC, Andrei Alexandrescu wrote:
> Ranges don't need necessarily an associated Iterable. This is the case in other languages, too; Lisp/Scheme/Haskell/etc lists are iterables and at the same time their own iterators. But indeed std.container is designed to have containers distinct from their associated ranges, which raises the interesting question what should happen if a range gets "orphaned", i.e. it's still active after its container ceases to exist.

And an Iterator doesn't necessarily need an associated Iterable.

I fully recognize how phobos handles ranges, and that it has become "the right way" to do things. But I disagree, and I do not want the core language to force me to take phobos' approach. I greatly appreciate that the only properties of ranges the core language cares about are `empty`, `front`, `back`, `popFront`, `popBack` - it does not itself imply how `opIndex` and `opSlice` are meant to behave for ranges - and I think it should stay that way. I see phobos' arrays-as-ranges functions as poor design, and most certainly not something that should become inextricably tied with the core language.

On Sunday, September 25, 2016 15:45:01 Andrei Alexandrescu via Digitalmars-d wrote:
> Ranges don't need necessarily an associated Iterable. This is the case in other languages, too; Lisp/Scheme/Haskell/etc lists are iterables and at the same time their own iterators. But indeed std.container is designed to have containers distinct from their associated ranges, which raises the interesting question what should happen if a range gets "orphaned", i.e. it's still active after its container ceases to exist.

Another thing to consider here is that given how most range-based functions will create a new range from the container's range when you pass it to them, when operating on ranges over containers, you _very_ quickly end up with ranges that really have nothing to do with the container anymore (hence the fun problems with removing elements from a container by passing a range to it). And that highlights how ranges really don't act like they're backed by containers.

But the safety issue that comes with ranges over containers where the container goes away is definitely a thorny one - as is the fact that removing elements from the container while you have active ranges that refer to it can do funny things to those ranges. I tend to favor C++'s approach to iterators and how they stay valid, but with D's focus on safety, I don't know that that's acceptable (though enforcing safety tends to result in additional overhead which isn't in line with the efficiency goals that go with being a system language). So, I don't know what the best approach is.

- Jonathan M Davis