On Sunday, 24 November 2019 at 02:10:41 UTC, mipri wrote:
> There's not a choice between absolute perfect guarantees (with
> some other design) vs. a complete absence of guarantees (with
> this one). The choice is between language defaults, in how
> easily the guarantees can be defeated, in what you can expect
> of other people's code.

Walter's design has the explicit goal of giving 100% mechanically checkable memory safety for (limited) manual memory management.
Take for example this reply regarding a different proposal: [1]

> I haven't studied that. Is it a 100% [mechanically checkable memory safety] solution? I know there's one being worked on for C++, but it's an 80-90% solution.
>
> D needs to have a 100%, mechanically checkable, solution.
>
> We have 100% for transitive const and function purity. It's hard to get code to pass, but worth it when one succeeds. The Ownership/Borrowing solution is 100% which is what is appealing about it.

But now it appears to be more and more a linting tool for @system code that catches some errors but doesn't give any guarantees at all.
Or maybe it will, once your entire code base has transitioned to @live, and you somehow fixed the issue of mixing memory pools [2]? But then code that is made to work with the garbage collector will be crippled with overly strict aliasing rules from @live.

The current path seems to go towards a three-way split of D projects between memory management styles:
- @safe for code like Java/C#
- @system for code like C/C++
- @live for code like Rust

It would be really cool if ownership and borrowing could work alongside the garbage collector instead of introducing this hard split called `@live` for every function.

And if this was all just an experiment to test the waters before creating a final, complete design, I would not be concerned. But the way it is described in the blog post [3] and given the fact that DIP 1021 has been accepted [4], it seems like Walter is already committed to this specific design which has been pointed out to be seriously flawed by Timon Gehr multiple times now.

Walter's attitude seems to be "We plug the holes one by one" [5], but I don't see any holes being plugged recently. The test cases in the pull request do not display any memory corruption. Examples on the news group often include a wrongly @trusted free function (e.g. [6]), which do not show anything.

While I am excited that @safe is being worked on, it is completely unclear to me how any of the recent developments contribute to @safe as a 100% mechanically checked memory safety solution. If we're going to downgrade @safe to a 80% memory safety solution relying on strong defaults and well-informed programmers, then the specification needs to be updated to reflect that.

[1] https://forum.dlang.org/post/qlvihc$24nk$1@digitalmars.com
[2] https://github.com/dlang/dmd/pull/10586/files#diff-e96b0b4865baa6204208527156832d3fR252
[3] https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/
[4] https://forum.dlang.org/post/beselqdzploaeqfunzos@forum.dlang.org
[5] https://forum.dlang.org/post/qluqlv$c5m$1@digitalmars.com
[6] https://forum.dlang.org/post/qqckhn$1hss$1@digitalmars.com

On Saturday, 23 November 2019 at 23:40:05 UTC, Timon Gehr wrote:
> struct MP(T){ // owning, malloc-backed pointer
>     private T* payload;
>     @disable this();
>     @disable this(T*); // can't construct
>     @disable this(this); // can't copy (move-only type; therefore track
>                                         this type like you track
>                                         pointers in @live now)
>     pragma(inline,true){
>         private @system ~this(){} // only current module can drop
>                                   // values, in @system or @trusted code
>         ref T borrow()return{ return *payload; }
>     }
>     // can borrow out internal pointer
>     alias borrow this;
> }
>
> @safe MP!T malloc(); // type tracks allocator
> @trusted void free(MP!T); // @safe because pointer is known to be unique and malloc'd

To be honest I don't fully understand all the points you are making.

But that, that is a thing of beauty, exactly what I want.

The insight I gained from it is that you should not annotate functions, but rather, express the semantics you need by annotating a struct. Taken to its natural conclusion, that would make raw pointers @system. To use them in @safe you would need a wrapper struct with the semantics you need. This also scales really well, instead of adding yet another @annotation to every function every year, you just update your structs with the latest semantics you need.

If I have some time I am going to reread your latest posts, as I want to have a better understanding of what you are saying.

Thank you for fighting this fight.

On 24.11.19 03:33, mipri wrote:
> On Sunday, 24 November 2019 at 02:10:41 UTC, mipri wrote:
>> It's really hard to see you as only having sincere technical
>> objections to @live after reading this.
> 
> Even after rewriting this so many times,

Rewriting is a waste of time, because I respond to what you say, not how you say it. If you find yourself in such a situation, it can help to think about whether you actually want to say it. My drafts folder is overflowing with posts I never submitted.

> I reckon it still won't be received well.
> ...

I think a simple "I'm sorry, it wasn't my intention to question your motives" would have been more appropriate than predicting unreasonable animosity and implying it would be my fault. If I wanted to, I could choose to get offended by that just the same. :)

> So:
> 
> I'm actually very interested in criticisms of @live (I hope
> more people are testing it than is apparent from the posts
> here), and even of alternatives that won't happen. But I don't
> have a four-year degree with a major of "the last 300 years of
> your bitter disputes about language design", and every single
> post of yours has required that.

I don't think this is the case. In particular, the borrowing/ownership discussion is not very old.

> (I still have no idea what you
> could possibly mean with a remark like "It doesn't make
> @safe code any more expressive.")
> ...

It's a summary of some of the other points in the post.

@safe restricts code to be memory safe in a way that is checked by the compiler, such that only @trusted functions can be a potential source of memory unsafety in a @safe program. (Basically, @trusted functions are at the same level of trust as the compiler implementation of @safe, so that not everything at this level of trust has to be implemented in the compiler, which makes sense.)

If @live @safe code can interact arbitrarily with @safe code, @live @safe cannot establish that the invariants that @live attempts to preserve (every memory location has a unique mutable reference to it or many non-mutable references, pointers are not leaked, etc) actually hold. Therefore, we cannot use @live invariants to write @safe code whose safety depends on those invariants. This means @live does not improve the expressiveness of @safe code: it does not allow us to write new and interesting @safe code that we could not write before. Walter however claimed that @live enables safe manual memory management in @safe code

What I am complaining about is a discrepancy between the stated goals of @live and what it actually accomplishes. The goal is to close the gap, to keep the quality of D high. I believe this is Walter's sincere goal too, this is why he is asking for feedback in the first place.

My arguments are not very complicated, but necessarily a bit abstract, because Walter is not providing any concrete examples of @safe code that are helped by @live that I could then break immediately by applying that abstract reasoning. The burden of proof shouldn't even be on me, because if @live @safe indeed enables safe manual memory management, he can demonstrate it by providing a code example that I can't break.

> I realize it's tiresome to repeat things that you think are
> already established, though.

What's tiresome is when people keep responding with nonsense or personal attacks. I have no problem at all with people asking for additional details that I didn't think to provide, or let alone people responding with good points!

> Please feel free to disregard my input.
> 

I will disregard your personal attack, but I don't see any reason to disregard your input.

On 24.11.19 03:10, mipri wrote:
> On Saturday, 23 November 2019 at 23:40:05 UTC, Timon Gehr wrote:
>> ...
>> but if they
>> will, it will inevitably infect libraries and suddenly, yes, I
>> will have to deal with it.
> 
> How will you have to deal with it? Code can't require that
> their callers have @live.

It will inevitably _assume_ that its @safe callers have @live (otherwise it wouldn't have chosen to bother with @live @safe). So sure, you can violate that assumption, but then you risk memory corruption in so-called @safe functions. It's memory safety by convention.

> It's your preferred alternative that
> would necessarily entangle users of libraries.
> 

No, it's a take it or leave it situation. It would enable @safe implementations of functions that can't have @safe implementation at all right now. I'm not taking away anyone's option to write @system code.
As I said, if the goal of @live is just to provide some linting in @system code, that's fine, but the stated goals are actually more ambitious.

On Sunday, 24 November 2019 at 14:07:31 UTC, Timon Gehr wrote:
> This means @live does not improve
> the expressiveness of @safe code: it does not allow us to write
> new and interesting @safe code that we could not write before.
> Walter however claimed that @live enables safe manual memory
> management in @safe code

OK, I get it. The problem was that 'expressiveness' is so
strongly associated with code size or neatness. What you're
wanting is not something like "D code can become less verbose
or look more modern", but that more of the D code that's
currently possible, can be possibly marked @safe.

That's not a weird thing to want at all. Especially not if
there's a long term goal of making @safe the default. It can't
be the default if too much valid code isn't valid @safe.

On 24.11.19 13:30, Dennis wrote:
> ...

Nice summary and collection of references. Thanks!

> 
> The current path seems to go towards a three-way split of D projects between memory management styles:
> - ...
> - ...
> - @live for code like Rust

I agree with everything but this. For instance, Rust does not do any lifetime checking for raw pointers. In Rust, you can't access raw pointers in safe code. This makes sense for Rust, because there is no built-in GC or @safe mutable global variables acting as the default owners of built-in pointers. In D, @safe code can manipulate raw pointers just fine, because of the GC. Lifetime checking for raw pointers still does not make sense in @safe code though.

On 24.11.19 14:42, Sebastiaan Koppe wrote:
> On Saturday, 23 November 2019 at 23:40:05 UTC, Timon Gehr wrote:
>> struct MP(T){ // owning, malloc-backed pointer
>>     private T* payload;
>>     @disable this();
>>     @disable this(T*); // can't construct
>>     @disable this(this); // can't copy (move-only type; therefore track
>>                                         this type like you track
>>                                         pointers in @live now)
>>     pragma(inline,true){
>>         private @system ~this(){} // only current module can drop
>>                                   // values, in @system or @trusted code
>>         ref T borrow()return{ return *payload; }
>>     }
>>     // can borrow out internal pointer
>>     alias borrow this;
>> }
>>
>> @safe MP!T malloc(); // type tracks allocator
>> @trusted void free(MP!T); // @safe because pointer is known to be unique and malloc'd
> 
> To be honest I don't fully understand all the points you are making.
> ...

Most points are in support of this kind of strategy.

> But that, that is a thing of beauty, exactly what I want.
> 
> The insight I gained from it is that you should not annotate functions, but rather, express the semantics you need by annotating a struct. Taken to its natural conclusion, that would make raw pointers @system.

This is indeed what Rust does. In D, raw pointers without additional semantics can actually be fine in @safe code (for example, `new int` returns an `int*`, and so does `&x` for a module-level (thread-local) `int x;`), but @trusted and @system code has to be careful what raw pointers it passes to @safe functions.

> To use them in @safe you would need a wrapper struct with the semantics you need. This also scales really well, instead of adding yet another @annotation to every function every year, you just update your structs with the latest semantics you need.
> 
> If I have some time I am going to reread your latest posts, as I want to have a better understanding of what you are saying.
> ...

Feel free to ask if specific things are unclear. I had a major deadline yesterday and couldn't spend a lot of my time to make my posts more detailed. (And often, it is not so clear a priori which points deserve elaboration.)

> Thank you for fighting this fight.

Sure! I can't help it. :)

On Sunday, 24 November 2019 at 14:51:31 UTC, Timon Gehr wrote:
>
> This is indeed what Rust does. In D, raw pointers without additional semantics can actually be fine in @safe code (for example, `new int` returns an `int*`, and so does `&x` for a module-level (thread-local) `int x;`), but @trusted and @system code has to be careful what raw pointers it passes to @safe functions.
>

I just saw this article that might be helpful:
https://plv.mpi-sws.org/rustbelt/stacked-borrows/

I'm dealing with sick kiddos today so haven't had time to go over it in much (any) detail, but it looks like an attempt to solve the aliasing problem caused by raw pointers when passed to otherwise safe code.

On Sunday, 24 November 2019 at 21:51:08 UTC, Doc Andrew wrote:
> I just saw this article that might be helpful:
> https://plv.mpi-sws.org/rustbelt/stacked-borrows/
>
> I'm dealing with sick kiddos today so haven't had time to go over it in much (any) detail, but it looks like an attempt to solve the aliasing problem caused by raw pointers when passed to otherwise safe code.

Looks more like it is defining what authors should not do in unsafe code for Rust to make optimizations under the assumption that unsafe code is wellbehaved. Then they provide an interpreter that dynamically borrow checks tests by running the code... In order to capture situations that rust cannot deal with statically. Or something to that effect.

TLDR; they specify undefined behaviour in order to enable optimization.

Probably the opposite of what you are looking for?

On Monday, 25 November 2019 at 00:10:41 UTC, Ola Fosheim Grøstad wrote:
> Looks more like it is defining what authors should not do in unsafe code for Rust to make optimizations under the assumption that unsafe code is wellbehaved. Then they provide an interpreter that dynamically borrow checks tests by running the code... In order to capture situations that rust cannot deal with statically. Or something to that effect.
>
> TLDR; they specify undefined behaviour in order to enable optimization.
>

Yeah, that's about as much as I was able to take from it yesterday. I wasn't sure if the stacked borrow technique might be useful for catching some of the errors that Timon pointed out with the interface between @live and @system code. I think most of us would consider the UB an error, rather than an issue with optimization?