D has made a lot of progress recently on memory safety with -preview=dip1000, thanks in no small part to the work of Dennis Korpel. This progress has in turn enabled the creation of SafeRefCounted by Ate Eskola, which will hopefully be available in the next release of Phobos.

The next logical step on this journey is a version of SafeRefCounted with support for std.experimental.allocator. Unfortunately, this step is where we run into a roadblock.

SafeRefCounted is allowed make a @trusted call to free when it knows it holds the only pointer to its payload, because it knows (from the C standard) that free will not corrupt memory when called under those circumstances.

However, an allocator-aware version of SafeRefCounted that calls a generic Allocator.deallocate function instead of free specifically has literally no idea what that function will do, and therefore cannot mark that call as @trusted, ever, under any circumstances.

The only solution is to somehow allow deallocate (and by extension free) to have a @safe interface on its own—which isn't possible in the current D language. At minimum, it would require something like an isolated qualifier (h/t deadalnix for the link), which would guarantee that a pointer is the only pointer to a particular block of memory. Some form of ownership/borrow checking would also work, of course.

In any case, this is not something that can be solved in library code. A language change is necessary.

On Sunday, 25 September 2022 at 12:03:08 UTC, Paul Backus wrote:

Well Atila started it, so he has a bit of credit. I "stole" his PR at some point.

I think it can. We need to agree on what the deallocator can and cannot do. If the deallocator then does something disallowed, then its the deallocator that's to blame, not SafeRefCounted. The deallocator, unless it's an intentional memory leaker or the GC, cannot be @safe anyway, so there's no way for the user to cause UB without writing incorrect @trusted or @system code.

I do concur that reading how isolated works is worth it for anyone thinking how to improve @live.

On Sunday, 25 September 2022 at 12:48:14 UTC, Dukc wrote:

This is "safety by convention"--the exact thing we're trying to get away from by using @safe. It can work in example code and small projects, but it doesn't scale, because the effort required to maintain safety scales exponentially with program size (proportional to the number of code paths).

To make this work, we need the compiler to enforce the rules about what the deallocator can and cannot do.

On Sunday, 25 September 2022 at 13:12:11 UTC, Paul Backus wrote:

Why? The deallocator is going to be @system anyway.

This means that it does not matter whether there is an allocator-enabled reference counter. In either case it's safety by convention inside the deallocators, but compiler enforced safety in client code assuming the deallocator is written correctly.

On Sunday, 25 September 2022 at 14:04:31 UTC, Dukc wrote:

If you have isolated, the deallocator can be made @safe by having it take an isolated pointer as its argument.

You are right that it is not enough just to have the compiler enforce the rules on the deallocator--the calling code also has to know that the rules are being enforced.

On Sunday, 25 September 2022 at 14:23:22 UTC, Paul Backus wrote:

A language change like that is ideal in the long term. My point, however, is that we're not blocked on waiting for one. An allocator-enabled safe reference counter is possible with the present language (except perhaps for some yet-undiscovered issue). The downside that the deallocators will have to be reviewed against not only causing UB by themselves, but also against leaking a pointer so that it can be reached by @safe code.

On Sunday, 25 September 2022 at 14:45:38 UTC, Dukc wrote:

Well, of course, if you are willing to write unsound @trusted code, anything can be made @safe. :)

On Sunday, 25 September 2022 at 14:52:09 UTC, Paul Backus wrote:

The SafeRefCounted destructor wouldn't be any less sound that it's now. If free did something it's not supposed to, the @trusted attribute on the destructor would be invalid. It isn't any different with a custom deallocator.

On Sunday, 25 September 2022 at 15:07:04 UTC, Dukc wrote:

free's interface is defined by the C language standard. There are a lot of safeguards in place, both social and technical, to prevent a non-conformant implementation of free from making its way into your program. Therefore, it is not a big risk for SafeRefCounted's @trusted destructor to rely on the behavior of free conforming to the C language standard.

No such safeguards exist to prevent users of a hypothetical allocator-aware SafeRefCounted from passing in a custom deallocator that violates safety when called from the same destructor. So the risk of allowing memory corruption in @safe code is much, much, higher when using a custom deallocator than it is when using free.

On Sunday, 25 September 2022 at 14:23:22 UTC, Paul Backus wrote:

That would be nice, but doesn't that tantamount to only being able to deallocate something when you can proof there are no other aliases? At which point you might as well use that proof and have the compiler call deallocate for you ;)