On Sunday, 29 January 2017 at 23:48:40 UTC, Jordan Wilson wrote:

>> You need to do something like this:
>> auto arrMap = arr.filter!(x => x > 5).map!(x => x^^2).array;
>>
>> It's because arrMap is lazy evaluated.


So does it mean that I cannot assign FilterResult and MapResult to a variable and safely use it later, because underlying array may change meanwhile? Since I am dealing with large arrays, I thought I'd save some memory by not calling .array on MapResult.

On Sunday, 29 January 2017 at 21:41:57 UTC, albert-j wrote:
>     int[] arr;
>     foreach (i; 0..10)
>         arr ~= i;
>
>     writeln("Original array: ",arr);
>     // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] -- OK
>
>     auto arrMap = arr.filter!(x => x > 5).map!(x => x^^2);

arrMap is a range. The filter and map operations only happen when the range is iterated. The array on which arrMap operates is the memory that is referred to by arr. If you change the values in that memory before evaluating arrMap, the result will be affected.

However, `filter` also does something on creation: it skips to the first element that passes the predicate. I.e., it skips to 6 in your code. Or rather, it skips to the memory location where 6 is stored at the time. Since it happens on creation, that skipping will not be re-evaluated when you iterate over arrMap multiple times.

>     writeln("arrMap: ", arrMap);
>     // [36, 49, 64, 81] -- OK

Note that the values are computed while printing to the screen. They're not stored anywhere.

>     int[] toRemove = [1, 2, 9];
>     arr = arr.remove!(x => toRemove.canFind(x)).array;

Removing works by overwriting the array with only the wanted values and discarding the rest.

The case at hand in detail:

* Look at arr[0]. It's 0. That's not in [1, 2, 9], so it's copied to arr[0].
* Look at arr[1]. It's 1. That's in [1, 2, 9], so it's discarded.
* Look at arr[2]. It's 2. That's in [1, 2, 9], so it's discarded.
* Look at arr[3]. It's 3. That's not in [1, 2, 9], so it's copied to arr[1].
* arr[4] through arr[8] are not in [1, 2, 9] either, so they're copied to arr[2] through arr[6].
* arr[9] is in [1, 2, 9], so it's not copied.

Note that arr[7] through arr[9] have not been overwritten. So you've got this in arr: [0, 3, 4, 5, 6, 7, 8, 7, 8, 9]. The last three values are then sliced off, because three values have been removed. Those values are still there in memory, though. The array's new length just stops before them.

>     writeln("Original array after removal: ", arr);
>     // [0, 3, 4, 5, 6, 7, 8] -- OK
>
>     arr ~= arrMap.array;
>
>     writeln("Original array after removal and concatenation: ", arr);
>     // [0, 3, 4, 5, 6, 7, 8, 64, 49, 64, 81] -- what?

Now, you've created arrMap before calling `remove`, so it still uses the old length for its underlying array. But it sees the new values, because `remove` has updated them in place. That means, it sees the values mentioned above: [0, 3, 4, 5, 6, 7, 8, 7, 8, 9]. Except, `filter` has already skipped the first six elements. So arrMap sees this: [8, 7, 8, 9]. Square those values and you get [64, 49, 64, 81], which is what you see.

Generally, I'd say don't alter the underlying range/array between creating and evaluating a range, nor during evaluation. `filter` is most probably not the only range that misbehaves in that situation.

Instead of mixing lazy ranges with the eager, in-place `remove`, you can use filter again:

----
void main()
{
    import std.array: array;
    import std.algorithm: canFind, filter, map;
    import std.range: chain;
    import std.stdio: writeln;

    int[] arr;
    foreach (i; 0..10) arr ~= i;

    immutable int[] toRemove = [1, 2, 9];
    arr = chain(
        arr.filter!(x => !toRemove.canFind(x)),
        arr.filter!(x => x > 5).map!(x => x^^2),
    ).array;

    writeln(arr); /* prints "[0, 3, 4, 5, 6, 7, 8, 36, 49, 64, 81]" */
}
----

On Monday, 30 January 2017 at 00:17:51 UTC, ag0aep6g wrote:
> [...]

Great explanation, thank you!

On Monday, 30 January 2017 at 00:17:51 UTC, ag0aep6g wrote:

> Removing works by overwriting the array with only the wanted values and discarding the rest.

But then why do I get this:

    import std.stdio, std.algorithm, std.array;

    int[] arr;
    foreach (i; 0..10) arr ~= i; // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    writeln("&arr[0]:", &arr[0]); // prints "7F56FAE93000"

    arr = arr.remove!(x => x > 5).array;

    writeln("&arr[0]:", &arr[0]); // prints "7F56FAE92020"

    writeln("arr:",arr); // prints: "[0, 1, 2, 3, 4, 5]"


It looks like arr.remove allocates new memory and copies the data there.

On Monday, 30 January 2017 at 08:50:14 UTC, albert-j wrote:
> On Monday, 30 January 2017 at 00:17:51 UTC, ag0aep6g wrote:
>
>> Removing works by overwriting the array with only the wanted values and discarding the rest.
>
> But then why do I get this:
>
>     import std.stdio, std.algorithm, std.array;
>
>     int[] arr;
>     foreach (i; 0..10) arr ~= i; // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>
>     writeln("&arr[0]:", &arr[0]); // prints "7F56FAE93000"
>
>     arr = arr.remove!(x => x > 5).array;
>
>     writeln("&arr[0]:", &arr[0]); // prints "7F56FAE92020"
>
>     writeln("arr:",arr); // prints: "[0, 1, 2, 3, 4, 5]"
>
>
> It looks like arr.remove allocates new memory and copies the data there.

No, remove works in-place, you are the one specifically asking for a reallocation here: instead of

arr = arr.remove!(x => x>5).array;

write

arr.remove!(x => x>5);


Complete example:

import std.stdio;
import std.format;
import std.algorithm;

void main(string[] args) {
    int [] arr = [0, 1, 2, 3, 4, 5];

    auto before = arr.ptr;
    arr.remove!(x => x>5);
    auto after  = arr.ptr;

    assert(before == after, "%s %s".format(before, after));
}

On Monday, 30 January 2017 at 10:30:22 UTC, cym13 wrote:
> On Monday, 30 January 2017 at 08:50:14 UTC, albert-j wrote:
>> On Monday, 30 January 2017 at 00:17:51 UTC, ag0aep6g wrote:
>>
>>> Removing works by overwriting the array with only the wanted values and discarding the rest.
>>
>> But then why do I get this:
>>
>>     import std.stdio, std.algorithm, std.array;
>>
>>     int[] arr;
>>     foreach (i; 0..10) arr ~= i; // [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>
>>     writeln("&arr[0]:", &arr[0]); // prints "7F56FAE93000"
>>
>>     arr = arr.remove!(x => x > 5).array;
>>
>>     writeln("&arr[0]:", &arr[0]); // prints "7F56FAE92020"
>>
>>     writeln("arr:",arr); // prints: "[0, 1, 2, 3, 4, 5]"
>>
>>
>> It looks like arr.remove allocates new memory and copies the data there.
>
> No, remove works in-place, you are the one specifically asking for a reallocation here: instead of
>
> arr = arr.remove!(x => x>5).array;
>
> write
>
> arr.remove!(x => x>5);
>
>
> Complete example:
>
> import std.stdio;
> import std.format;
> import std.algorithm;
>
> void main(string[] args) {
>     int [] arr = [0, 1, 2, 3, 4, 5];
>
>     auto before = arr.ptr;
>     arr.remove!(x => x>5);
>     auto after  = arr.ptr;
>
>     assert(before == after, "%s %s".format(before, after));
> }

Meh.

Forget that, bad memory. remove isn't working in-place. However slapping ".array" is still asking explicitely for reallocation, so just forget it. Here is a code that works:

import std.conv;
import std.stdio;
import std.format;
import std.algorithm;

void main(string[] args) {
    int [] arr = [8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9];

    auto before = arr.ptr;
    arr = arr.remove!(x => x>5);
    auto after  = arr.ptr;

    assert(arr == [0, 1, 2, 3, 4, 5], arr.to!string);
    assert(before == after, "%s %s".format(before, after));
}

On Monday, 30 January 2017 at 10:45:03 UTC, cym13 wrote:

> Meh.
>
> Forget that, bad memory. remove isn't working in-place. However slapping ".array" is still asking explicitely for reallocation, so just forget it. Here is a code that works:
>
> import std.conv;
> import std.stdio;
> import std.format;
> import std.algorithm;
>
> void main(string[] args) {
>     int [] arr = [8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
>
>     auto before = arr.ptr;
>     arr = arr.remove!(x => x>5);
>     auto after  = arr.ptr;
>
>     assert(arr == [0, 1, 2, 3, 4, 5], arr.to!string);
>     assert(before == after, "%s %s".format(before, after));
> }

OK, got it. Can you do removal without reallocation with std.container.array?

    Array!int arr;
    foreach (i; 0..10) arr ~= i;

On Monday, 30 January 2017 at 12:31:33 UTC, albert-j wrote:

>
> OK, got it. Can you do removal without reallocation with std.container.array?
>
>     Array!int arr;
>     foreach (i; 0..10) arr ~= i;

Sorry, sent too early.

    arr = arr[].remove!(x=> x > 5); //Doesn't work withouth calling .Array!int

On 01/30/2017 01:33 PM, albert-j wrote:
> On Monday, 30 January 2017 at 12:31:33 UTC, albert-j wrote:
>
>>
>> OK, got it. Can you do removal without reallocation with
>> std.container.array?
>>
>>     Array!int arr;
>>     foreach (i; 0..10) arr ~= i;
>
> Sorry, sent too early.
>
>     arr = arr[].remove!(x=> x > 5); //Doesn't work withouth calling
> .Array!int

Looks like it can't be done in that straight-forward manner. But you can do the `remove` and then update just `arr`'s length:

----
void main()
{
    import std.container: Array;
    import std.algorithm: equal, remove;

    Array!int arr;
    foreach (i; 0..10) arr ~= i;
    const oldAddress = &arr[0];

    arr.length = arr[].remove!(x=> x > 5).length;

    assert(equal(arr[], [0, 1, 2, 3, 4, 5])); /* holds */
    assert(&arr[0] is oldAddress); /* holds */
}
----

Maybe std.container.Array can be improved somehow to simplify this. But I'm not sure about the details.