Hi,

Does anybody know if there are any container libraries that make use of scope for iterator/range safety? or is the implementation of scope even ready for that right now?

Thanks

On Saturday, 9 March 2019 at 16:14:34 UTC, bitwise wrote:
> Hi,
>
> Does anybody know if there are any container libraries that make use of scope for iterator/range safety? or is the implementation of scope even ready for that right now?
>
> Thanks

tanya's containers are good candidates for scope/return, since they don't rely on reference counting and GC like Phobos containers. I intend to introduce scope/return ranges soon, because currently using the containers is not very safe.
But honestly I'm not expecting much from these annotations. Last time I tested "return" with dip25 it could catch only some basic cases, but not more complicated cases where I assign ranges to variables and then use them.

On Saturday, 9 March 2019 at 16:34:18 UTC, Eugene Wissner wrote:
> On Saturday, 9 March 2019 at 16:14:34 UTC, bitwise wrote:
>> Hi,
>>
>> Does anybody know if there are any container libraries that make use of scope for iterator/range safety? or is the implementation of scope even ready for that right now?
>>
>> Thanks
>
> tanya's containers are good candidates for scope/return, since they don't rely on reference counting and GC like Phobos containers. I intend to introduce scope/return ranges soon, because currently using the containers is not very safe.
> But honestly I'm not expecting much from these annotations. Last time I tested "return" with dip25 it could catch only some basic cases, but not more complicated cases where I assign ranges to variables and then use them.

Ok thanks, I'll have a look.

All I really need is scoped return values so ranges/iterators and such can't leave the scope of their containers. So even a partial implementation would suffice if it was reliable.

On Saturday, 9 March 2019 at 16:14:34 UTC, bitwise wrote:
> Hi,
>
> Does anybody know if there are any container libraries that make use of scope for iterator/range safety? or is the implementation of scope even ready for that right now?
>
> Thanks

I wrote this:

https://github.com/atilaneves/automem/blob/master/source/automem/vector.d

On Sunday, 10 March 2019 at 12:06:10 UTC, Atila Neves wrote:
>
> I wrote this:
>
> https://github.com/atilaneves/automem/blob/master/source/automem/vector.d

I'm confused by the use of scope. For example, what effect does scope have here?

this(this) scope { ... }

I don't see any ranges/iterators in that class either, which is what's at issue for me.

At present, it seems impossible to create a memory-safe container that supports iterators or ranges in D, unless it's entirely GC.

The best attempt I've seen is std.Array, which uses a ref-counted/pointer-to-pointer payload to make sure any ranges that were given out can see the changes to the payload. Isn't it specified that a struct should be moveable in D though? So as soon as you memcpy an std.Array everything starts exploding/double-deleting. Also, this seems like it would apply to *any* library based ref-counted implementation. So it seems like scope is the last hope for creating an @safe @nogc container that can give out ranges/iterators. Even adding a move-constructor to D seems like a no-go since it's expected that structs can be memcpy'ed.

Am I wrong about this?

Thanks

On Sunday, 10 March 2019 at 15:12:03 UTC, bitwise wrote:
> On Sunday, 10 March 2019 at 12:06:10 UTC, Atila Neves wrote:
>>
>> I wrote this:
>>
>> https://github.com/atilaneves/automem/blob/master/source/automem/vector.d
>
> I'm confused by the use of scope. For example, what effect does scope have here?
>
> this(this) scope { ... }

Like `const`, `scope` on a member function means that the `this` reference is `scope`.

> I don't see any ranges/iterators in that class either,

https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L134
https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L159
https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L145

static assert(isInputRange!(Vector!int));

> At present, it seems impossible to create a memory-safe container that supports iterators or ranges in D, unless it's entirely GC.
>
> The best attempt I've seen is std.Array, which uses a ref-counted/pointer-to-pointer payload to make sure any ranges that were given out can see the changes to the payload. Isn't it specified that a struct should be moveable in D though? So as soon as you memcpy an std.Array everything starts exploding/double-deleting. Also, this seems like it would apply to *any* library based ref-counted implementation. So it seems like scope is the last hope for creating an @safe @nogc container that can give out ranges/iterators. Even adding a move-constructor to D seems like a no-go since it's expected that structs can be memcpy'ed.
>
> Am I wrong about this?

I would think so, especially since I wrote Vector to be @safe with dip1000. Look at this unit test for not being able to escape the payload for example:

https://github.com/atilaneves/automem/blob/master/tests/ut/vector.d#L229

On Sunday, 10 March 2019 at 16:10:10 UTC, Atila Neves wrote:
> On Sunday, 10 March 2019 at 15:12:03 UTC, bitwise wrote:
>> On Sunday, 10 March 2019 at 12:06:10 UTC, Atila Neves wrote:
>>> [...]
>>
>> I'm confused by the use of scope. For example, what effect does scope have here?
>>
>> this(this) scope { ... }
>
> Like `const`, `scope` on a member function means that the `this` reference is `scope`.

Interesting. I'll have to look up some examples.

>> I don't see any ranges/iterators in that class either,
>
> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L134
> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L159
> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L145
>
> static assert(isInputRange!(Vector!int));
>
>> At present, it seems impossible to create a memory-safe container that supports iterators or ranges in D, unless it's entirely GC.
>>
>> The best attempt I've seen is std.Array, which uses a ref-counted/pointer-to-pointer payload to make sure any ranges that were given out can see the changes to the payload. Isn't it specified that a struct should be moveable in D though? So as soon as you memcpy an std.Array everything starts exploding/double-deleting. Also, this seems like it would apply to *any* library based ref-counted implementation. So it seems like scope is the last hope for creating an @safe @nogc container that can give out ranges/iterators. Even adding a move-constructor to D seems like a no-go since it's expected that structs can be memcpy'ed.
>>
>> Am I wrong about this?
>
> I would think so, especially since I wrote Vector to be @safe with dip1000. Look at this unit test for not being able to escape the payload for example:
>
> https://github.com/atilaneves/automem/blob/master/tests/ut/vector.d#L229

Ok, but this container copies the entire collection in postblit:
https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L104

So if you try to treat it like a range, you'll take a performance hit every time you trigger the postblit, which seems likely.

Isn't it reasonable to assume a range should act like a reference to data, rather than a copy of it?

On Sunday, 10 March 2019 at 17:36:09 UTC, bitwise wrote:
>
> Isn't it reasonable to assume a range should act like a reference to data, rather than a copy of it?

I think, this highly depends on data and the container.
What if data is stored inside the container? How the container can reference data if it is the one and only holder? In which format do you expect the container to release the raw data? How do you suppose to synchronize a data movement and references to it, if the container contains only references?
But what, if data may not be copied or moved, but the container manages the data by copy?
Is data assumed to be destroyed/invalidated after the container is destroyed? And why?

On Sunday, 10 March 2019 at 17:36:09 UTC, bitwise wrote:
> On Sunday, 10 March 2019 at 16:10:10 UTC, Atila Neves wrote:
>> On Sunday, 10 March 2019 at 15:12:03 UTC, bitwise wrote:
>>> [...]
>>
>> Like `const`, `scope` on a member function means that the `this` reference is `scope`.
>
> Interesting. I'll have to look up some examples.
>
>>> [...]
>>
>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L134
>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L159
>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L145
>>
>> static assert(isInputRange!(Vector!int));
>>
>>> [...]
>>
>> I would think so, especially since I wrote Vector to be @safe with dip1000. Look at this unit test for not being able to escape the payload for example:
>>
>> https://github.com/atilaneves/automem/blob/master/tests/ut/vector.d#L229
>
> Ok, but this container copies the entire collection in postblit:
> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L104

Well, yes. As it says in the module ddoc, it's my version of C++'s std::vector. If you don't want it to copy, pass by ref.

> So if you try to treat it like a range, you'll take a performance hit every time you trigger the postblit, which seems likely.

You can pass it by ref like I mention above, or you could slice it and pass the slice instead. Without DIP1000 passing the slice wouldn't be @safe but with it, it is.

Again, just like std::vector, which is pretty much always passed by const ref. But safer and with a default allocator.

> Isn't it reasonable to assume a range should act like a reference to data, rather than a copy of it?

It depends.

On Sunday, 10 March 2019 at 18:46:14 UTC, Atila Neves wrote:
> On Sunday, 10 March 2019 at 17:36:09 UTC, bitwise wrote:
>> On Sunday, 10 March 2019 at 16:10:10 UTC, Atila Neves wrote:
>>> On Sunday, 10 March 2019 at 15:12:03 UTC, bitwise wrote:
>>>> [...]
>>>
>>> Like `const`, `scope` on a member function means that the `this` reference is `scope`.
>>
>> Interesting. I'll have to look up some examples.
>>
>>>> [...]
>>>
>>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L134
>>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L159
>>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L145
>>>
>>> static assert(isInputRange!(Vector!int));
>>>
>>>> [...]
>>>
>>> I would think so, especially since I wrote Vector to be @safe with dip1000. Look at this unit test for not being able to escape the payload for example:
>>>
>>> https://github.com/atilaneves/automem/blob/master/tests/ut/vector.d#L229
>>
>> Ok, but this container copies the entire collection in postblit:
>> https://github.com/atilaneves/automem/blob/2a97acba94d6fe0bf9ba07fec99e86e46aa0f2a1/source/automem/vector.d#L104
>
> Well, yes. As it says in the module ddoc, it's my version of C++'s std::vector. If you don't want it to copy, pass by ref.
>
>> So if you try to treat it like a range, you'll take a performance hit every time you trigger the postblit, which seems likely.
>
> You can pass it by ref like I mention above, or you could slice it and pass the slice instead. Without DIP1000 passing the slice wouldn't be @safe but with it, it is.
>
> Again, just like std::vector, which is pretty much always passed by const ref. But safer and with a default allocator.
>
>> Isn't it reasonable to assume a range should act like a reference to data, rather than a copy of it?
>
> It depends.

This container seems good for short-lived usage. If I wanted to pass a bunch of filenames to a function for processing, I would consider this container Ok, because it would allocate once, then be consumed as a range without having to copy anything. For any kind of persistent storage though, I would consider the performance costs of using this container's range functionality too high.

At this point, I'm leaning toward just building separate containers. One would be a vector-like container that's @nogc and only has an indexer, but does not support iterators/ranges. The second would be a class-based container which *would* have range/iterator support since the ranges/iterators could safely contain pointers back to the GC-allocated collection for range checks and such.

I do think the idea of making the container itself the range is interesting, but what ruins it for me is the fact that ranges are consumable. So I'm wondering if it would work well to use an integer index for the range implementation instead of actually depleting the elements as you iterate, and have a "reset" function to refill the range so that no copying is necessary. I wonder how useful it would be to have some sort of NonConsumableRange concept that included reset() function.

  Bit