November 11, 2019
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
>

The DIP includes a discussion of std.array.appender:

> A workaround for deprecating:
> 
> a ~= b;
> is to use std.array.appender ...

The DIP should describe whether this includes both std.array.Appender and std.array.RefAppender or just std.array.Appender.


November 11, 2019
On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer wrote:
> On 11/11/19 11:47 AM, Uknown wrote:
>> A lot of people are bringing it up, so I'll bite. The problem with @nogc is that it doesn't cover all cases. Imagine the code given in the DIP like this instead:
>> 
>> ---lib.d
>> 
>> void f(int x[]) @safe pure nothrow
>> {
>>      x ~= 0;
>> }
>> 
>> ---main.d
>> 
>> void main() @safe
>> {
>>      import lib: f;
>>      import std.container : Array;
>> 
>>      Array!int x = [0, 1, 2, 3, 4];
>> 
>>      f(slice);
>>      // x's dtor will try to free an invalid pointer
>> }
>> 
>> Clearly here main does something that seems safe on the surface. But in actuality it is clearly unsafe code. And its hard to verify, because main and the libraries used are written by completely different people.
>
> No, you are misunderstanding a lot here.
>
> 1. f(slice), there is no symbol slice, I think maybe you mean x[]?

yes, my mistake

> 2. f's x is a *copy*, which means that appending to x here DOES NOT AFFECT main's x at all. Main's x will destroy perfectly fine, and all is well.

You're right, I shouldn't have used Array. I was just trying to point out how libraries could obscure such bugs. A different container with different semantics which exposes slices could cause issues.

> 3. If @nogc is added to main, then it won't compile, because f cannot be @nogc. Which is quite the point people are making.
>
> -Steve

The problem is that once the slice is created, there's no way to know how the underlying memory was allocated. And if you want to append to the slice, then that's hard. Because there's no way to append generically in the language without the GC. Don't get me wrong, I think this DIP is the wrong solution. But there is an issue to be addressed. We need a way to be able to append without invoking the GC. I personally think std.experimental.allocators + some other associated language modifications are the way forward.
November 11, 2019
On Monday, 11 November 2019 at 17:30:25 UTC, Steven Schveighoffer wrote:
> Sorry, I'm not sure why it's nitpicking. The example shown does not show what the author wanted it to show.

Alright, alright, I thought it was obvious what he meant, but I guess that is because I already agreed with his core viewpoint.


> True, GC-using code could be valid to call if it doesn't affect your type. But @nogc IS a valid way to use slices and ensure that they never use the GC to grow.

Yes, but you need to be careful so you don't split the ecosystem in two. So people should be able to call the best libraries they can find (GC or not).

I understand that this is difficult, but I think the intent of this DIP is pointing in the right direction, even if it isn't bulletproof or ready.  So I hope it isn't dismissed outright.


> But in fact, a better solution is to make sure the TYPE prevents it

Yes, a type that protects the underlying buffer somehow makes a lot of sense.

But then you need to encourage templated slicing somehow, so that you can write generic code that takes all kinds of slicing-types (and use introspection perhaps to choose how they are used).

Not so easy to get right, I guess... Hm.

> If the DIP said it wanted to introduce a type that enforces it will always point to the resource that is managed, that would be fine. It just can't overtake T[] to mean that. It's way way too disruptive. I'd be fine with new syntax as well (like T[new] or whatnot).

Ok, I understand.

Maybe a type that will share the binary interface of C++ span and string_view. They are supposed to be the ones to use in function C++ function signatures to increase interoperability between libraries etc.

Right now it is a bit difficult to say how span<> will be adopted in C++ libraries, but I would imagine that it might end up being used for matrices as well "span<span<T>>". So binary compatibility could make sense.


November 11, 2019
On Monday, November 11, 2019 11:03:35 AM MST Uknown via Digitalmars-d wrote:
> The problem is that once the slice is created, there's no way to know how the underlying memory was allocated. And if you want to append to the slice, then that's hard. Because there's no way to append generically in the language without the GC. Don't get me wrong, I think this DIP is the wrong solution. But there is an issue to be addressed. We need a way to be able to append without invoking the GC. I personally think std.experimental.allocators + some other associated language modifications are the way forward.

If you want to append to a dynamic array without using the GC, then just allocate a new block of memory with malloc or whatever, copy the elements from the dynamic array into it along with the elements being appended, slice that block of memory, and assign that dynamic array to the first one. Of course, you then have to code worrying about when it's safe to free that memory, but if your code needs something like reference counting to manage the memory, then it probably shouldn't be passing around dynamic arrays anyway. Dynamic arrays which are slices of malloc-ed memory or static arrays or whatnot work just fine so long as the code that does the slicing can know when the dynamic arrays which are slices of that memory go out of scope so that it can know when it's safe to free that memory, but if the dynamic arrays being passed around in a way that the code doing the allocating can't know when that memory isn't referenced anymore, then dynamic arrays are a bad fit - and that's the case whether this DIP is accepted or not. If the dynamic arrays are being passed around in a way that you don't know what their lifetimes are, then they need to be GC allocated.

- Jonathan M Davis



November 11, 2019
On Monday, 11 November 2019 at 18:01:03 UTC, Jon Degenhardt wrote:
> On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
>> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>>
>> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
>>
>
> The DIP includes a discussion of std.array.appender:
>
>> A workaround for deprecating:
>> 
>> a ~= b;
>> is to use std.array.appender ...
>
> The DIP should describe whether this includes both std.array.Appender and std.array.RefAppender or just std.array.Appender.

To clarify my previous comment, std.array.appender (lower-case) is a convenience function that can create both std.array.Appender and std.array.RefAppender. However, it is not obvious that std.array.RefAppender would be supported after acceptance of this DIP.
November 11, 2019
On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis wrote:
> How on earth do D dynamic arrays change their type? int[] is always int[].

Well, what we mean by "type" can of course be debated, but anyway:

Case 1:
    char[] a = new char[2];
    writeln(a.ptr);
    if (false) a.length = 1;
    a.length = 2;
    writeln(a.ptr);  // same pointer

Case 2:
    char[] a = new char[2];
    writeln(a.ptr);
    if (true) a.length = 1;
    a.length = 2;
    writeln(a.ptr); // different pointer

So, whether it extends inline depends what you did with the object dynamically. It isn't determined statically. Even though in this case that would be possible.

Kinda like the difference between an open and closed file, whether it is open or not depends on some dynamic external cause (if it exists on the files system).


November 11, 2019
On Monday, November 11, 2019 11:54:04 AM MST Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis
>
> wrote:
> > How on earth do D dynamic arrays change their type? int[] is always int[].
>
> Well, what we mean by "type" can of course be debated, but anyway:
>
> Case 1:
>      char[] a = new char[2];
>      writeln(a.ptr);
>      if (false) a.length = 1;
>      a.length = 2;
>      writeln(a.ptr);  // same pointer
>
> Case 2:
>      char[] a = new char[2];
>      writeln(a.ptr);
>      if (true) a.length = 1;
>      a.length = 2;
>      writeln(a.ptr); // different pointer
>
> So, whether it extends inline depends what you did with the object dynamically. It isn't determined statically. Even though in this case that would be possible.
>
> Kinda like the difference between an open and closed file, whether it is open or not depends on some dynamic external cause (if it exists on the files system).

So what if the pointer's value changed? Objects have their members mutated all the time. That doesn't make them different types. If you want to know whether appending to a dynamic array will result in a new buffer being allocated (with the elements being copied over, and the ptr member of the dynamic array being mutated), then check whether the capacity is large enough to contain the elements being appended. If it is, then the GC can expand the dynamic array in-place. If it isn't, then the GC has to allocate a new buffer, copy the elements over, and mutate the dynamic array's ptr and length members to point to the new buffer. And that's the same regardless of what kind of memory backs the dynamic array (it's just that there's never enough capacity to grow in-place if the underlying buffer wasn't allocated by the GC for dynamic arrays). I don't understand why you'd think that a dynamic array being in a different state based on what you did with it would in any way make it a different type. It's perfectly normal for the behavior of an object to change based on its state.

- Jonathan M Davis




November 11, 2019
On Monday, 11 November 2019 at 18:15:13 UTC, Jonathan M Davis wrote:
> accepted or not. If the dynamic arrays are being passed around in a way that you don't know what their lifetimes are, then they need to be GC allocated.

In the Prior Work section the DIP states:

«Rust's notion of pointers owning the memory they point to, unless the pointer is borrowed, is equivalent with this DIP's notion of a slice "borrowing" a reference to another object which manages the memory.»

So it does seem to imply that slices are borrowed views.

Conceptually, dynamic arrays would be borrowed from the GC, thus you don't need to know their lifetimes.

Lifetimes would come into play for non-dynamic-arrays, I guess, so scope... perhaps.



November 11, 2019
On Monday, 11 November 2019 at 19:18:53 UTC, Jonathan M Davis wrote:
> I don't understand why you'd think that a dynamic array being in a different state based on what you did with it would in any way make it a different type. It's perfectly normal for the behavior of an object to change based on its state.

Well, the conceptual state was exactly the same when accessed. So it is unusual for a core language type.

Anyway, there is no point in arguing this, as we first have to agree on what we mean by the word "type" at runtime vs "type" at compile time. Which would be very off topic...

(You could create a compile time ubertype that includes all other possible types, so that nothing can be said about any object at compile time, but you would still think that those objects had various types at runtime.)



November 11, 2019
I've never been comfortable with the fact that T[] can refer to any type of memory, GC-allocated or not (or T*, for that matter), but I think this DIP is not the right approach. It severely pessimizes one of the most common use cases of slices in order to prevent memory corruption when the slice does not refer to a GC'd array. While this could be a big gain, it is also definitely a huge loss in terms of how much code would be broken.

I completely agree, though, that the fact that ~ may change slices referring to non-GC'd memory to GC'd memory is not a good thing - I just think this DIP would break too much code.

I can't see a clear way forward, either, because at a fundamental level in the type system, D treats GC'd and non-GC'd pointers the same. Therefore, it would not be enough to introduce a new type that exclusively refers to GC-allocated memory (i.e., new T* / T[new], or whatever), because slicing would still have to be disabled for non-GC'd pointers, and appending would still have to be disabled for non-GC'd slices.

Would it be enough to put an assert in GC.realloc/extend/free that asserts the memory is GC-owned, maybe tied to a compiler switch?