On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
>
> All review-related feedback on and discussion of the DIP should occur in this thread. The review period will end at 11:59 PM ET on November 25, or when I make a post declaring it complete.
>
> At the end of Round 1, if further review is deemed necessary, the DIP will be scheduled for another round of Community Review. Otherwise, it will be queued for the Final Review and Formal Assessment.
>
> Anyone intending to post feedback in this thread is expected to be familiar with the reviewer guidelines:
>
> https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
>
> *Please stay on topic!*
>
> Thanks in advance to all who participate.

I get what the DIP is trying to do, but like Jonathon said, this solution is simply not worth the cost *at all*. Such a change would alienate the existing users, make a lot of good, working code break for no real reason, and cause a lot of other issues.

The better solution IMO is to instead set an allocator, either per thread or global, like std.experimental.allocator does. Then, change the runtime to allocate and expand slices by using that allocator, instead of the GC. The allocator could be GC by default. Maybe once that is done, slicing arbitrary pointers could be depreciated. Then the `@safe` aspect would be solved, and there would be practically 0 code breakage.

Something like this:

---a.d

import core.allocator; //hypothetical

void f(int[] x) @safe
{
    x ~= 0;
}

void main() @system
{
    int[] slice = (cast(int*)(malloc(10 * int.sizeof)))[0 .. 10]; //deprecated
    this_thread.allocator = Mallocator;
    int[] slice = new int[10];
    f(slice); //now safe, since mallocator is used instead of GC
}

On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
>
> All review-related feedback on and discussion of the DIP should occur in this thread. The review period will end at 11:59 PM ET on November 25, or when I make a post declaring it complete.
>
> At the end of Round 1, if further review is deemed necessary, the DIP will be scheduled for another round of Community Review. Otherwise, it will be queued for the Final Review and Formal Assessment.
>
> Anyone intending to post feedback in this thread is expected to be familiar with the reviewer guidelines:
>
> https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
>
> *Please stay on topic!*
>
> Thanks in advance to all who participate.

"This change is a __necessary__ part of evolving D toward being memory safe without using a GC"

This is a bold statement, that needs to be addressed with a little of explanation of why there are no alternatives.

I think it's not acceptable to suffer such a huge pain, without having a clear understanding that there was a deep analysis on potential alternative solutions, and an explanation of why they are not sufficient for the scope.

Just to be clear, why @nogc is not enough?

Thanks

On 11/11/19 5:27 AM, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
> 
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md 
> 
> 
> All review-related feedback on and discussion of the DIP should occur in this thread. The review period will end at 11:59 PM ET on November 25, or when I make a post declaring it complete.
> 
> At the end of Round 1, if further review is deemed necessary, the DIP will be scheduled for another round of Community Review. Otherwise, it will be queued for the Final Review and Formal Assessment.
> 
> Anyone intending to post feedback in this thread is expected to be familiar with the reviewer guidelines:
> 
> https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
> 
> *Please stay on topic!*
> 
> Thanks in advance to all who participate.

This is a question of terminology. The dynamic array type in D is specified to be a slice. This has extreme benefits that I think we all will lament if this goes away.

As I've said before, treating the slice type as the array type is super confusing and not technically true. A dynamic array owns its own data, the slice does not. In D, the dynamic array is created and maintained *typeless* in the GC. It's owned by the GC. A slice can point at a dynamic array, stack-allocated data, global/thread-local data, a malloc-allocated array, or any other piece of memory that you can possibly point at. As soon as you append to it, it NOW points at a GC-allocated array. This should be neither surprising, nor a problem, IF you adopt the right terminology.

A fix here could be to actually DEFINE the dynamic array type, and allow the user to fetch it, and understand how it works. You can continue to allow the slice to be an interface to it, or you can use it directly. Please email me if you want more details.

I 100% agree with Adam, just put @nogc if you don't want to use the GC. This is a huge breaking change, and the much better solution is to make the desired behavior opt-in. A middle ground could be a newly defined "slice" type that can be used to opt-in.

I'd say no to this change, unless we are talking D3. And if that's the case, let's start talking about all the other things we want to fix ;)

-Steve

On Monday, 11 November 2019 at 16:36:57 UTC, Paolo Invernizzi wrote:
> 
> "This change is a __necessary__ part of evolving D toward being memory safe without using a GC"
>
> This is a bold statement, that needs to be addressed with a little of explanation of why there are no alternatives.
>
> I think it's not acceptable to suffer such a huge pain, without having a clear understanding that there was a deep analysis on potential alternative solutions, and an explanation of why they are not sufficient for the scope.
>
> Just to be clear, why @nogc is not enough?
>
> Thanks

A lot of people are bringing it up, so I'll bite. The problem with @nogc is that it doesn't cover all cases. Imagine the code given in the DIP like this instead:

---lib.d

void f(int x[]) @safe pure nothrow
{
    x ~= 0;
}

---main.d

void main() @safe
{
    import lib: f;
    import std.container : Array;

    Array!int x = [0, 1, 2, 3, 4];

    f(slice);
    // x's dtor will try to free an invalid pointer
}

Clearly here main does something that seems safe on the surface. But in actuality it is clearly unsafe code. And its hard to verify, because main and the libraries used are written by completely different people.

On Monday, November 11, 2019 7:48:54 AM MST IGotD- via Digitalmars-d wrote:
> On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> > This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
> >
> > https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee5 6e4/DIPs/DIP1025.md
> >
> > All review-related feedback on and discussion of the DIP should occur in this thread. The review period will end at 11:59 PM ET on November 25, or when I make a post declaring it complete.
> >
> > At the end of Round 1, if further review is deemed necessary, the DIP will be scheduled for another round of Community Review. Otherwise, it will be queued for the Final Review and Formal Assessment.
> >
> > Anyone intending to post feedback in this thread is expected to be familiar with the reviewer guidelines:
> >
> > https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
> >
> > *Please stay on topic!*
> >
> > Thanks in advance to all who participate.
>
> "Enlarging a slice, using the append operator or by setting the .length property, makes use of the garbage collector. This fosters the incorrect notion that D slices require the GC."
>
> The title mentions arrays but the rest of the text is about slices.
>
> I need one clarification, this DIP regards slices not arrays? So an array is being able to be resized just as before. It's just that this will no longer apply for slices?
>
> If this only regards slices, then I think this is a sound change, even if it is breaking. A slice should not really know the underlying algorithm how the data is stored. It could be an array, it could be part of a buffer or a view into any other structure. Limiting the slice so that it cannot grow beyond its original boundaries makes sense.
>
> If this include arrays, that they aren't able to grow then I'm against this change but again it doesn't make any sense. Then it is not a dynamic array.

Dynamic arrays _are_ slices. T[] is a dynamic array which is a slice of a block of memory which may or may not be GC-allocated. Some folks do try to talk like dynamic arrays are the GC-allocated memory block that a slice refers to, but that's not the definition used by the language. As far as the language is concerned, T[] is a dynamic array no matter what memory it refers to, and the language has no specific term for a memory block allocated by the GC which is sliced by a dynamic array.

This DIP makes it so that it would no longer be possible to use any operations on a dynamic array which would increase its size, whereas right now those operations always work in code that isn't @nogc no matter what memory the dynamic array is a slice of. It's just that in the case where the dynamic array has no capacity to grow, you always get a reallocation. Whether the dynamic array referred to GC-allocated memory or not is irrelevant except that if the memory is not GC-allocated, there definitely isn't room to grow the dynamic array in-place, whereas if it was GC-allocated then there might be room to grow in-place. In either case, if there isn't room to grow in-place, then the GC allocates a new memory block, copies the elements into the new memory block, and alters the dynamic array to point to the new memory block.

- Jonathan M Davis

On 11/11/19 11:47 AM, Uknown wrote:
> A lot of people are bringing it up, so I'll bite. The problem with @nogc is that it doesn't cover all cases. Imagine the code given in the DIP like this instead:
> 
> ---lib.d
> 
> void f(int x[]) @safe pure nothrow
> {
>      x ~= 0;
> }
> 
> ---main.d
> 
> void main() @safe
> {
>      import lib: f;
>      import std.container : Array;
> 
>      Array!int x = [0, 1, 2, 3, 4];
> 
>      f(slice);
>      // x's dtor will try to free an invalid pointer
> }
> 
> Clearly here main does something that seems safe on the surface. But in actuality it is clearly unsafe code. And its hard to verify, because main and the libraries used are written by completely different people.

No, you are misunderstanding a lot here.

1. f(slice), there is no symbol slice, I think maybe you mean x[]?

2. f's x is a *copy*, which means that appending to x here DOES NOT AFFECT main's x at all. Main's x will destroy perfectly fine, and all is well.

3. If @nogc is added to main, then it won't compile, because f cannot be @nogc. Which is quite the point people are making.

-Steve

On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer wrote:
> No, you are misunderstanding a lot here.

This is nitpicking.

If you are going to have a good story with interfacing with C++ libraries and frameworks... you don't want this to fail accidentally:

ownerpointer = normalize(initialize(allocate()))

Where normalize is a D function that calls other GC library code that is well-behaving, except the one that accidentally grows the slice because it works in the library-author's pure GC code.

You have to be able to call GC code even if you want to modify non-GC memory.


Anyway, it also isn't true that Dynamic Arrays don't have a type. It isn't untyped. But the typesystem it uses is completely different from the rest of the language.

It appears to be quasi-dynamic effects based, whereas the rest of the language is mostly static strongly typed. Meaning, the type changes as you use it. Does any other object in D do that?

On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via Digitalmars-d wrote:
> On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer
>
> wrote:
> > No, you are misunderstanding a lot here.
>
> This is nitpicking.
>
> If you are going to have a good story with interfacing with C++ libraries and frameworks... you don't want this to fail accidentally:
>
> ownerpointer = normalize(initialize(allocate()))
>
> Where normalize is a D function that calls other GC library code that is well-behaving, except the one that accidentally grows the slice because it works in the library-author's pure GC code.
>
> You have to be able to call GC code even if you want to modify non-GC memory.
>
>
> Anyway, it also isn't true that Dynamic Arrays don't have a type. It isn't untyped. But the typesystem it uses is completely different from the rest of the language.
>
> It appears to be quasi-dynamic effects based, whereas the rest of the language is mostly static strongly typed. Meaning, the type changes as you use it. Does any other object in D do that?

How on earth do D dynamic arrays change their type? int[] is always int[]. I believe that the bit that Steven is talking about with regards to them being untyped is that inside druntime, it uses void*, because the code isn't templated - which arguably is something that should be fixed. Outside of druntime's innards though, a dynamic array always has the same type. All it really is is a struct containing a pointer and a size_t for length.

- Jonathan M Davis

On 11/11/19 12:12 PM, Ola Fosheim Grøstad wrote:
> On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer wrote:
>> No, you are misunderstanding a lot here.
> 
> This is nitpicking.
> 

Sorry, I'm not sure why it's nitpicking. The example shown does not show what the author wanted it to show.

When you have a function:

void f(int x[]) @safe pure nothrow
{
    x ~= 0;
}

There is no way to call that function in a way that will break safety, or alter any external data. It's pure, only the shallowly passed in slice is affected, nothing outside. This is truly a misunderstanding by the author (Uknown) of what a slice actually is.

The original example in the DIP shows more of a problem, but @nogc is certainly a solution there, which is why people are bringing it up.

> If you are going to have a good story with interfacing with C++ libraries and frameworks... you don't want this to fail accidentally:
> 
> ownerpointer = normalize(initialize(allocate()))
> 
> Where normalize is a D function that calls other GC library code that is well-behaving, except the one that accidentally grows the slice because it works in the library-author's pure GC code.

True, GC-using code could be valid to call if it doesn't affect your type. But @nogc IS a valid way to use slices and ensure that they never use the GC to grow.

But in fact, a better solution is to make sure the TYPE prevents it, and we absolutely have the mechanisms today to do this. An opt-in type (ironically, such as Array), would be sufficient to make sure you can still call gc-using code, without mucking with your type's ability to free the original pointer.

If the DIP said it wanted to introduce a type that enforces it will always point to the resource that is managed, that would be fine. It just can't overtake T[] to mean that. It's way way too disruptive. I'd be fine with new syntax as well (like T[new] or whatnot).

-Steve

On 11/11/19 12:28 PM, Jonathan M Davis wrote:
> On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via
> Digitalmars-d wrote:
>> Anyway, it also isn't true that Dynamic Arrays don't have a type.
>> It isn't untyped. But the typesystem it uses is completely
>> different from the rest of the language.
>>
>> It appears to be quasi-dynamic effects based, whereas the rest of
>> the language is mostly static strongly typed. Meaning, the type
>> changes as you use it. Does any other object in D do that?
> 
> How on earth do D dynamic arrays change their type? int[] is always int[]. I
> believe that the bit that Steven is talking about with regards to them being
> untyped is that inside druntime, it uses void*, because the code isn't
> templated - which arguably is something that should be fixed. Outside of
> druntime's innards though, a dynamic array always has the same type. All it
> really is is a struct containing a pointer and a size_t for length.

auto arr = "hello".dup;

ubyte[] arr2 = cast(ubyte[]) arr;

arr2 ~= 1; // fine appends in-place (possibly)
arr ~= 'a'; // fine, makes a copy

The type doesn't matter to druntime, only the length of data. That is the point.

Of course, this all breaks down if the type has a postblit/destructor, which IS stored by druntime. But I'm not sure if there are any runtime mechanisms to prevent it (I haven't touched that code much since the posblit/dtor code was added).

-Steve