On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
>> This is what the dead/alive example is supposed to express.
>> It is still true however that the rationale is not very convincing.
>> The first example indeed doesn't show any benefits of the DIP:
>> 
>> ```
>> int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
>> slice = slice ~ 1; // now guaranteed to make a copy
>> free(slice.ptr); // Still oops
>> ```
>
> Imagine these 3 lines are spread out over a large code base.

You realize he changed the code right? That is still valid code that exhibits the same problem even after this DIP gets integrated into DMD.


>> The following claim in the DIP is also unsubstantiated:
>> 
>>> This change is a necessary part of D evolving towards being memory safe without using
>>> a GC.
>
> Memory safety cannot be achieved without control over who is the owner of memory.

So is the GC being removed from D? Is this the ultimate goal? Cause the convenience of the language that is provided by using a GC is being removed along with these DIPs. At some point there's not going to be a reason to the GC because all of its' conveniences will have been removed at this rate.

On 11.11.19 11:27, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
> 
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md 
> 
> 
> All review-related feedback on and discussion of the DIP should occur in this thread. The review period will end at 11:59 PM ET on November 25, or when I make a post declaring it complete.
> 
> At the end of Round 1, if further review is deemed necessary, the DIP will be scheduled for another round of Community Review. Otherwise, it will be queued for the Final Review and Formal Assessment.
> 
> Anyone intending to post feedback in this thread is expected to be familiar with the reviewer guidelines:
> 
> https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
> 
> *Please stay on topic!*
> 
> Thanks in advance to all who participate.

Wrong season. April Fools' Day is in April.

On 11/11/2019 6:22 AM, Adam D. Ruppe wrote:
> This already works - and you can statically enforce it on a per-function level by @nogc. Or on a per-type basis with a trivial wrapper struct (except null doesn't implicitly construct - we should fix THAT). Or on a per-project basis with -betterC.
> 
> All this proposal does is break code that hasn't opted into those existing options.

Global solutions do exist, but they don't allow mixed mode code.

The idea is to make code mechanically auditable. It isn't auditable if creating a wrapper is expected.


>> Some communication about the overall goal here would be nice. Is there even a plan?
> I agree with this regardless though.

Be able to mechanically verify that code does not have memory safety bugs. We may not be able to get there 100%, but the closer the better.

On 11.11.19 22:41, Walter Bright wrote:
> 
> Memory safety cannot be achieved without control over who is the owner of memory.

The GC is a perfectly fine owner of memory. Please take a step back and stop gutting the language.

On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
> That's right. I had overlooked that.

It looked like you were aware since you mentioned "although that will generate more GC garbage if used in a loop" when referring to the workaround of `a = a ~ b`.

> Imagine these 3 lines are spread out over a large code base.

With or without this DIP, that would be problematic.

> On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
> Memory safety cannot be achieved without control over who is the owner of memory.
>
>> I would like to see an example of memory corruption in `@safe` code that can happen because of slice appending.
>
>   @trusted void myfree(void* p) { free(p); }
>   free(slice.ptr);

free cannot be @trusted on a raw pointer type.
In that definition, you can (from @safe code) give `myFree` a pointer to a local variable, global variable or GC pointer, all of which may corrupt memory. (And I'm ignoring the lack of @live here. Without @live you can obviously add double free / use-after-free to that list.)

The only way to make `free` safe is to encapsulate it in a struct or class that ensures the pointer that is being freed was allocated with malloc. If you have a custom type, then concatenation like ~= will not work unless you implement it, so disabling it on the built-in slice type does not do much good.

In fact, you can consider ~= only syntactic sugar for a call to _d_arrayappendcTX.
You can disable the ~= construct, but everywhere you would use that operator you can still do something functionally equivalent and change what kind of data (i.e. stack memory, heap memory) the slice refers to.

So the current proposal helps discouraging certain code, but it does not prevent anything.

On Monday, 11 November 2019 at 22:04:17 UTC, Steven Schveighoffer wrote:
> In order for the DIP to guarantee the pointer of a slice is still the pointer to memory to be freed, you have to *disable assignment*. Do we want to do that too?
>
> It also must disable popFront, or really any operation except for opIndex.

Actually, I think the example I gave was bad, since the DIP states that a slice is for borrowing, and then it should not be used for ownership. I retract it.

If a slice is a strictly borrowed view then you can have popFront etc, but not extending it.

This would basically be the same as C++ span with dynamic extents.


Then you would need something else for ownership of non-GC memory that can provide borrowed slices.

And yet again something else for write-buffer-views (e.g. a view on a field in a struct that should be written to, reallocation is bad in that situation).

So you need basically 3 different types. I think…

The idea is to be able to guarantee memory safety. I've spoken about this in my presentations and DConf, and that's clearly what DIP25 and DIP1000 were targeted at, and my proposals for an Ownership/Borrowing system for D.

This is where the industry is moving. Even C++ is heading that direction:

  http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

This proposal is written by Herb Sutter, and is no joke.

We either get on the bus or get run over by the bus.

On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:

> We either get on the bus or get run over by the bus.

Frankly speaking, I've read such argumentations a lot of times along many years in the D history: the concurrency chapter of the D Programming Language is here to testify that this type of argumentation should be avoided, in a technical discussion.

Let's move forward from that, and let's try to find a sound solution to the memory safety problem, but without killing the language in the attempt.

There's no hurry: D is _already_ a safe language, thanks to your intuition of 20 years ago to bake the GC.

On 11.11.19 23:28, Walter Bright wrote:
> 
> We either get on the bus or get run over by the bus.

That's a false dichotomy. E.g., you could also eat the engine of the bus for dinner and then get stranded in the desert with heavy metal poisoning.

On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via Digitalmars-d wrote:
> Memory safety cannot be achieved without control over who is the owner of memory.

And how does this DIP help with that? Dynamic arrays do not know or care where their memory came from, and they do not track lifetimes in any way, shape, or form. Doing anything that might append to them then guarantees that they're allocated by the GC, because that's the only way that they can grow. Without doing one of those operations, you would have to actually look at the ptr value of the dynamic array and ask the GC whether it's owned by the GC. Getting rid of the append operations does not change that. It's still impossible to know who owns the memory by simply looking at the type, and code that passes slices of non-GC allocated memory around has to be aware of the lifetime of those dynamic arrays and what's done with them to know whether any dynamic arrays referring to that memory still exist. The type system doesn't help with that beyond stuff like scope and pure limiting what can be done with the dynamic array that gets passed in. And getting rid of the append operations doesn't change that.

Sure, right now, if you pass a dynamic array which is a slice of malloc-ed memory to a function, that function could append to that dynamic array and result in a reallocation. Why is that a problem? It's not like that's going to cause a memory leak unless it was your only pointer to that memory, and if that's the case, having the dynamic array shrink on you would be just as big a problem, if not bigger. Either way, you have to know what the code is doing with the dynamic array if you're going to try to determine when it's safe to free that memory, because dynamic arrays can normally be passed around and copied with impunity, and they're not reference-counted. Their very nature pretty requires that if you're going to have a dynamic array which slices non-GC-allocated memory, you have to carefully control what you do with it, because nothing about the dynamic array manages the lifetime of the memory it points to. That's the case whether appending is allowed or not.

If for some reason, it is actually a problem for a piece of code if a dynamic array ever gets appended to, then requiring that the code be @nogc fixes that. It's then not possible to pass the dynamic array to any functions which might append to it.

Really, code that's using dynamic arrays which are slices of anything other than GC-allocated memory has to be very limited in how it passes them around regardless of whether appending is possible, because dynamic arrays don't manage their own memory at all. Code that wants to do much passing around of malloc-ed memory to anything other than a pure function where it's clear that the slices can't be squirreled away in any of the arguments or return value is going to tend to need to wrap that memory in something other than a dynamic array in order to manage its lifetime - either that, or it needs to be written in a way that the programmer can clearly know that anything that even might contain a reference to that memory is long gone before they're looking to free that memory (e.g. because it's allocated at the beginning of the program and freed at the end).

It looks to me like you're crippling the normal use case in attempt to deal with a less common one. And it doesn't look to me like it's really fixing anything for that less common use case.

Honestly, if it really is the case that we have to lose the ability to append to dynamic arrays in order to get @safety for malloc-ed memory, then I'd rather give up on @safety for malloc-ed memory and keep the ability to append to dynamic arrays. But from what I can tell, the only argument that the DIP has for why appending to dynamic arrays even could be a problem is because it might be problematic if a dynamic array which slices non-GC-allocated memory ends up being copied into newly allocated GC memory, because it's appended to. I dispute that that's a real problem, and even if it is one, @nogc prevents it without any language changes.

It should be clear to anyone seriously using dynamic arrays that

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
free(slice.ptr); // Oops!

is a problem. It's a given that appending can cause a reallocation. But even if appending weren't allowed, you could still have something like

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice = slice[1 .. $];
free(slice.ptr); // Oops!

and have problems, because the dynamic array isn't slicing the beginning of that block of memory anymore. So, allowing a dynamic array to shrink (or even be assigned to at all) causes similar problems as appending in this case. Either way, relying on the dynamic array itself to know which pointer to use to free that memory is just plain bad practice unless the code is definitely never going to do anything that would mutate the dynamic array itself (be it slicing, appending, or assignment from another array). In general, if code is going to slice malloc-ed memory to get a dynamic array, it should be keeping around a separate pointer which refers to the block that was allocated so that it can free it when no more dynamic arrays exist which point to that memory rather than relying on the dynamic array for that information. Either way, the general inability to track the lifetime of dynamic arrays outside of things like scope and pure means that you can't rely on the type system to track the lifetime of malloc-ed memory outside of functions that simply take an argument and return a result (with scope and pure used to ensure that no slices of that memory are holed away somewhere).

As for code such as

enum { dead, alive }
int[] cat = new int[6];
cat[5] = alive;
int[] b = cat;
b ~= 1;      // may or may not move b to new location
b[5] = dead; // indeterminate whether cat[5] is dead or alive

So what? It's a given that appending to a dynamic array could cause a reallocation. If that's a problem, the code can check the array's capacity first to see if appending would cause a reallocation. This behavior of arrays is nothing new and isn't specific to dynamic arrays which are slices of memory which wasn't allocated by the GC. If the code actually cares about whether a reallocation takes place, it can easily check whether it will or has taken place, and it can easily prevent it from taking place by requiring that the code be @nogc. The DIP does nothing to show why this behavior is actually a problem for @safe code.

- Jonathan M Davis