November 12, 2019
On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson wrote:
> (You mean moot point)
yes
>
> Thats what containers are for, to encapsulate the inherent unsafety and expose a safe interface.
Okay, from where I'm standing that sounds like the problem is with malloc/slicing
malloced memory, and not so much the GC.
On the other hand, the more functional my D becomes, the less I feel the need
for containers at all.

> encapsulating safe access to members with locks/atomics/etc.
>
> [1]:https://github.com/dlang/DIPs/blob/master/DIPs/DIP1024.md

Thank you, I'll have another read.
Sounds like this whole @safe-borrowing-rc needs a top down design DIP to
understand the complete context.


November 12, 2019
On Tuesday, 12 November 2019 at 11:12:30 UTC, Jonathan M Davis wrote:
> Dynamic arrays and manually managed memory are a _terrible_ combination, because dynamic arrays do nothing to manage lifetimes. They're really not any different from passing naked pointers around as far as lifetime tracking goes. This means that the only way that you can automatically manage that is to restrict where they can be stored. scope does that, but the result is that you can pretty much just pass the data into a function that operates on it and doesn't store it.

One option, albeit unorthodox, is to require heap-stored slices to be "concrete" and therefore "encode" the properties of what it is referencing.

Whereas you could allow slices used in the call-graph to be "abstract" (templated, but without the syntax) and then infer their concrete properties using static analysis.

So, the counter-argument would be "combinatorial explosion", but if the compiler deduce that the actual function that is being compiled generate the same code for a set of concrete slice types then you can emit a single copy of the compiled code.

Anyway, that ought to be sufficient to distinguish GC, reference counted and RAII objects. And it would also allow D slices to differ from C++ slices, but also allow you to write one function to cover both scenarios.



November 12, 2019
If we are to do this, we need to limit the breakage somehow. A few ideas, that could perhaps be used in some combination:

-Instead of disallowing appending on arrays, change the garbage collector so that it'll never append in place. Of course, this has the potential to be a performance killer so alone it won't be enough.

-Only disable GC appending in @safe or @live

-Come up with a way to mark some code or module to diallow GC appending, but do not disable it everywhere. Something that won't require yet another attribute in every function declaration.

-`scope` slices can be appended on.

Rust is already infamous for being hard to get into because of the borrow checker. So we definitely should not implement everything it does the same way. This DIP in it's current form would cause so much damage that if we fall behind because we won't do it, so be it.
November 12, 2019
On Tuesday, 12 November 2019 at 11:46:30 UTC, Robert Schadek wrote:
> On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson wrote:
>> Thats what containers are for, to encapsulate the inherent unsafety and expose a safe interface.
> Okay, from where I'm standing that sounds like the problem is with malloc/slicing
> malloced memory, and not so much the GC.
> On the other hand, the more functional my D becomes, the less I feel the need
> for containers at all.

Perhaps containers is the wrong word. I meant in in the sense data structures intersection wrapping, e.g. DList!int or Atomic!int. Both of those are inherently unsafe (blobs of pointers and shared memory respectively) but the container provides a safe way to interact with the data encapsulated by it by disallowing outside access in an unchecked manner.
November 12, 2019
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
>

I don't know.
C++ has std::vector and std::string_view, but our slices do these two things. It makes things easier sometimes but in the end you have to track ownership in your head, especially if you use @nogc and -betterC.

Does "dynamic arrays ARE slices" really hurt us?

This change would break a great number of libraries, as said in the thread. Personally I have zero interest in more memory safety.

On the other hand, maybe we can go "leap of faith" on this.
It would also help if there was a good std::vector replacement in the stdlib, with explicit .capacity.
November 12, 2019
On Tuesday, 12 November 2019 at 13:08:08 UTC, Nicholas Wilson wrote:
> Perhaps containers is the wrong word. I meant in in the sense data structures intersection wrapping, e.g. DList!int or Atomic!int. Both of those are inherently unsafe (blobs of pointers and shared memory respectively) but the container provides a safe way to interact with the data encapsulated by it by disallowing outside access in an unchecked manner.

I see.


November 12, 2019
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
> This is the feedback thread for the first round of Community Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
>
> https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

See here for my response: https://gist.github.com/John-Colvin/c9c0b79bc9d47ed8d57ec9e959d0542b


Included below for convenience and record:


The DIP provides 2 examples to justify itself:

1) unpredictable mutable aliasing after appends

2) unpredictable changes of ownership caused by appends

They can share the same cause (`~=`), but are two distinct symptoms, and can also have other causes. They have a unifying aspect which is unpredictable aliasing.

The breakage of the DIP is immense, so there needs to be a commensurately large upside and no reasonable alternative path with less breakage.


### Let's consider the first example:

```D
int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
free(slice.ptr); // Oops!
```

`free` is not an `@safe` operation and requires some conditions to be met for correct use. These conditions are as follows:
 - `free` happens after all accesses to the allocation referenced by the passed pointer.
 - the allocation referenced by the passed pointer was made by `{m,c,re}alloc`

as such, you can't really ever call `free` without completely trusting the path by which the data travelled, from inception (`{m,c,re}alloc`) to the call to `free`. This may be achieved by a variety of ways (e.g. a `MallocSlice` type that encapsulates the slice, or by having the path be very short and doing it manually) but fundamentally if I write a function taking a slice and then freeing the pointer, I cannot mark it as `@trusted`, regardless of whether `~=` is allowed or not.

That `~=` will cause a new array to be allocated unpredictably is not the problem, the problem is that it can happen at all. If we look at it probabilistically: the problem is not that the probability of re-allocation is in `(0, 1)`, the problem is that it's not `0`. *Someone can write `slice = slice ~ 1` and get the same behaviour, so my considerations when calling `free` have not changed by disallowing `~=`.*

### On to the second example:

```D
enum { dead, alive }
int[] cat = new int[6];
cat[5] = alive;
int[] b = cat;
b ~= 1;      // may or may not move b to new location
b[5] = dead; // indeterminate whether cat[5] is dead or alive
```

This is a sticky one. `~=` applied to data where there are >1 references and >0 are mutable is liable to cause problems of unpredictable action at a distance. Removing `~=` makes it slightly harder to do this, but seeing as it's so very easy to do the same thing in other ways (`a = b ? a : (a ~ 2)` and many other trivial examples) it arguably doesn't matter much. I could believe it makes it less likely to do by accident.

## Conclusion:

Overall, this change seems to not buy anyone much, but costs a lot. It correctly identifies a problem (or 2 problems, depending how you  look at it), but the proposed solution does not appear on a simple reading to solve the problem.

The DIP does not provide any examples of what is directly enabled by the change and I was not able to infer it from the current text.

*The author should explain what previously impossible/unsafe code can now be made possible/safe given the proposed change.*
November 12, 2019
On Tuesday, 12 November 2019 at 13:10:06 UTC, Guillaume Piolat wrote:
> [snip]
>
> I don't know.
> C++ has std::vector and std::string_view, but our slices do these two things. It makes things easier sometimes but in the end you have to track ownership in your head, especially if you use @nogc and -betterC.
>
> Does "dynamic arrays ARE slices" really hurt us?
>
> [snip]

I keep getting a little mixed up when referring to slices and dynamic arrays. So some
T[] x;
is a dynamic array and
auto y = x[a..b];
or
auto y = x[];
is a slicing operation on x. And right now, slicing x also returns a dynamic array unless there is some opSlice defined that does something different.

So one option would be to keep dynamic arrays as they are and change the slicing operations to return a slice type. They would be like dynamic arrays with the exception that you can't append to them or change the length. However, this would probably lead to a lot of breakage, even if you could have a .array function that can easily convert a slice to an array.

What about instead something like below with a type that has appending disabled. I had tried to make length const, but I figured it was easier to just return data's length. I'm not sure if this works in the more general case though. Regardless, the benefit is that this is completely opt-in and requires no code-breakage.

```
struct View(T : U[], U) {
    T data;
    alias data this;

    this(T x) {
     	data = x;
    }

    @disable void opOpAssign(string op)(U rhs)
        if (op == "~")
    {
        data ~= rhs;
    }

    @disable void opOpAssign(string op)(U[] rhs)
        if (op == "~")
    {
        data ~= rhs;
    }

    @disable T[] opBinary(string op)(U rhs)
        if (op == "~")
    {
        return data ~ rhs;
    }

    @disable T[] opBinary(string op)(U[] rhs)
        if (op == "~")
    {
        return data ~ rhs;
    }

    size_t length() {
     	return data.length;
    }
}

auto view(T : U[], U)(T x) {
	return View!(T, U)(x);
}

void main() {
    int[] x = [0, 3, 5];
    auto y = x[0..2].view;
}
```
November 12, 2019
On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:
> This DIP in it's current form would cause so much damage that if we fall behind because we won't do it, so be it.

It is going to fall behind either way. This DIP focuses on falling behind Rust/C++ safety with manual memory management. But it completely ignores falling behind on simplicity/easy of use/unrestricted use with other languages that have a GC. The focus seems to only be on Rust/C++ lately. By trying to be safe with manual memory management you lose what the GC offers, this is in part due to the fact that GC pointers and malloc pointers are indistinguishable. It's a clash between two different ideologies that don't mix well.
November 12, 2019
On Tuesday, 12 November 2019 at 15:54:22 UTC, John Colvin wrote:
> *The author should explain what previously impossible/unsafe code can now be made possible/safe given the proposed change.*

Looking at the examples again, do they even show memory corruption?


This would be a memory leak, not memory corruption:

  int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
  slice ~= 1;
  free(slice.ptr); // Oops!



This doesn't show memory corruption either, its a potential logic bug as it doesn't consider a side effect of array concatenation.


  enum { dead, alive }
  int[] cat = new int[6];
  cat[5] = alive;
  int[] b = cat;
  b ~= 1;      // may or may not move b to new location
  b[5] = dead; // indeterminate whether cat[5] is dead or alive


Neither of these examples are pertinent to @safe, which is aimed at reducing/removing memory corruption.

This would be a more relevant problem for @safe:

  int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
  free(slice.ptr);
  slice ~= 1;       // use after free

As far as I'm aware none of the current previous DIPs resolve this do they?