Jump to page: 1 27  
Page
Thread overview
Allocator-aware @safe reference counting is still not possible
Jan 22, 2023
Atila Neves
Jan 22, 2023
Paul Backus
Jan 23, 2023
Paul Backus
Jan 23, 2023
Paul Backus
Jan 23, 2023
Dukc
Jan 23, 2023
Paul Backus
Jan 24, 2023
Dukc
Jan 24, 2023
Dukc
Jan 24, 2023
Paul Backus
Jan 24, 2023
Dukc
Jan 24, 2023
Dukc
Jan 23, 2023
jmh530
Jan 23, 2023
Atila Neves
Jan 23, 2023
Paul Backus
Jan 23, 2023
Paul Backus
Jan 23, 2023
Paul Backus
Jan 24, 2023
Dukc
Jan 26, 2023
FeepingCreature
Jan 26, 2023
FeepingCreature
Jan 22, 2023
RTM
Jan 24, 2023
Nick Treleaven
Jan 24, 2023
jmh530
Jan 28, 2023
Nick Treleaven
Jan 30, 2023
Timon Gehr
Jan 30, 2023
Dukc
Jan 30, 2023
Paul Backus
Jan 30, 2023
Paul Backus
Jan 30, 2023
Paul Backus
Jan 30, 2023
Paul Backus
Jan 30, 2023
Paul Backus
Jan 30, 2023
Dukc
Jan 30, 2023
Paul Backus
Jan 30, 2023
Paul Backus
Feb 01, 2023
jmh530
Jan 30, 2023
Dukc
Jan 30, 2023
Paul Backus
Jan 30, 2023
Timon Gehr
Jan 31, 2023
Dukc
Feb 04, 2023
patescross
Feb 04, 2023
Paul Backus
Jan 30, 2023
jmh530
Jan 30, 2023
Timon Gehr
Jan 30, 2023
jmh530
Feb 01, 2023
Atila Neves
Feb 01, 2023
Dukc
January 22, 2023

https://forum.dlang.org/post/jsuraddtynhjoaikqprs@forum.dlang.org

On Sunday, 25 September 2022 at 12:03:08 UTC, Paul Backus wrote:

>

D has made a lot of progress recently on memory safety with -preview=dip1000, thanks in no small part to the work of Dennis Korpel. This progress has in turn enabled the creation of SafeRefCounted by Ate Eskola, which will hopefully be available in the next release of Phobos.

The next logical step on this journey is a version of SafeRefCounted with support for std.experimental.allocator. Unfortunately, this step is where we run into a roadblock.

SafeRefCounted is allowed make a @trusted call to free when it knows it holds the only pointer to its payload, because it knows (from the C standard) that free will not corrupt memory when called under those circumstances.

However, an allocator-aware version of SafeRefCounted that calls a generic Allocator.deallocate function instead of free specifically has literally no idea what that function will do, and therefore cannot mark that call as @trusted, ever, under any circumstances.

The only solution is to somehow allow deallocate (and by extension free) to have a @safe interface on its own—which isn't possible in the current D language. At minimum, it would require something like an isolated qualifier (h/t deadalnix for the link), which would guarantee that a pointer is the only pointer to a particular block of memory. Some form of ownership/borrow checking would also work, of course.

In any case, this is not something that can be solved in library code. A language change is necessary.

I'm pretty much convinced we need isolated. This is very similar to why the language as it exists today doesn't allow a library author to write a vector type that can be appended to, which... is the main reason one would use a vector to begin with.

Some allocators (GC?) might have a @safe deallocate function but most (all except the GC?) can't due to aliasing, and that requires isolated.

January 23, 2023
On 23/01/2023 4:28 AM, Atila Neves wrote:
> I'm pretty much convinced we need isolated.

I'm not. When I first got a link to that paper I certainly didn't understand even the basic concepts. Variable based borrow checker is much easier to understand comparatively.

My general feeling is allocators get used in two scenarios:

- Controlled: this is your self contained data structure type scenario with RC. Safe, because if it wasn't the data structure wouldn't work.
- Uncontrolled: No lifetimes, either global heavy or only used with function body, which means fat slices and pointers (no thanks). Unsafe and cannot be made it (due to things like globals).

So there is no point in trying to make memory lifetimes of uncontrolled safe, because you shouldn't be doing this! Use a data structure instead.

That just leaves controlled, where @localsafe would be desirable (so you could call @system RCAllocator api). And having a better lifetime management strategy to and with RC (i.e. eliding & order of destruction via borrow checker).

Throw in value type exceptions as well, and ROM aware RC hooks; we'd be in a good place I think for this.
January 22, 2023

On Sunday, 22 January 2023 at 15:28:53 UTC, Atila Neves wrote:

>

I'm pretty much convinced we need isolated.
Seems redundant to dip1021 (@live).

January 22, 2023
On Sunday, 22 January 2023 at 15:50:27 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 23/01/2023 4:28 AM, Atila Neves wrote:
>> I'm pretty much convinced we need isolated.
>
> I'm not. When I first got a link to that paper I certainly didn't understand even the basic concepts. Variable based borrow checker is much easier to understand comparatively.

A borrow checker would also work. I think `isolated` is probably a better fit for D, but it's certainly not the only option.

> That just leaves controlled, where @localsafe would be desirable (so you could call @system RCAllocator api).

Can you explain more about @localsafe? I don't understand how this is different from Dukc's proposal in the linked thread.
January 23, 2023
On Sunday, 22 January 2023 at 15:50:27 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 23/01/2023 4:28 AM, Atila Neves wrote:
>> I'm pretty much convinced we need isolated.
>
> I'm not. When I first got a link to that paper I certainly didn't understand even the basic concepts. Variable based borrow checker is much easier to understand comparatively.
>

I can’t claim to be an expert, but I’ve read some on isolated and some smart people here like the idea. The borrow checker certainly has some similarities. With the borrow checker, you can have an unlimited number of const references or a single mutable reference. Isolated means you can at most have a single mutable way of accessing some data. So it’s missing that either/or-ness of the borrow checker, if I understand it correctly. Another question is whether an affine type or qualifier is better than @live for handling the borrow checker behavior. If such a thing existed would there be a demand for isolated?




January 23, 2023
On 23/01/2023 10:31 AM, Paul Backus wrote:
> Can you explain more about @localsafe? I don't understand how this is different from Dukc's proposal in the linked thread.

Its @safe except you can call non-safe functions. Same goes for nogc and pure.

Basically it verifies that you didn't do something stupid without limiting what you can call (like callbacks).

This is something I've been wanting for a while now due to potential mistakes with creating contexts with callbacks.
January 23, 2023
On 23/01/2023 3:07 PM, jmh530 wrote:
> Another question is whether an affine type or qualifier is better than @live for handling the borrow checker behavior. If such a thing existed would there be a demand for isolated?

Yes I believe a type qualifier (scope), would be better suited towards a borrow checker than any other solution.

Essentially a borrow checker just says, an owning reference lifetime must exceed that of a borrowed reference. It guarantees the right order of death for them.

Its so simple, we already have a ton of logic to support this with DIP1000! We don't need new syntax, just some smarter semantics surrounding owning/borrowing.

Just to be clear, I don't think @live solves any problem that the D community has. It is useless.

Here is how much @live is used (note we know it is severely incomplete):

https://issues.dlang.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&component=dmd&f0=OP&f1=OP&f2=assigned_to&f3=CP&f4=CP&j1=OR&list_id=243775&o2=substring&query_format=advanced&short_desc=%40live&short_desc_type=allwordssubstr&v2=live

None. Not a one bug report. Nobody uses it today. Because we don't need it.
January 23, 2023
On Sunday, 22 January 2023 at 15:50:27 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 23/01/2023 4:28 AM, Atila Neves wrote:
>> [...]
>
> I'm not. When I first got a link to that paper I certainly didn't understand even the basic concepts. Variable based borrow checker is much easier to understand comparatively.
>
> [...]

I don't understand your post. I've never seen anything resembling the "uncontrolled" above. I think the way to use allocators is via smart pointers/containers and iff it's strictly required.
January 23, 2023
On 23/01/2023 9:31 PM, Atila Neves wrote:
> On Sunday, 22 January 2023 at 15:50:27 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> On 23/01/2023 4:28 AM, Atila Neves wrote:
>>> [...]
>>
>> I'm not. When I first got a link to that paper I certainly didn't understand even the basic concepts. Variable based borrow checker is much easier to understand comparatively.
>>
>> [...]
> 
> I don't understand your post. I've never seen anything resembling the "uncontrolled" above. I think the way to use allocators is via smart pointers/containers and iff it's strictly required.

I have written uncontrolled allocator usage. Its also in druntime/phobos plenty. After all its basically just a life cycle is known but compiler can't prove anything useful related to it. See any usage of malloc/free (including internally to the GC).

But yes, I think the way to go is some sort of controlled representation for normal usage (which requires borrowing, which @live does not solve).

```d
Vector!int vector;
vector ~= 3;

auto borrowed = vector[0];
func(borrowed);

void func(scope ref int value) {
	
}
```

Basically right now we're missing the lifetime checks surrounding borrowed & function parameter. Everything else is do-able right now, even if it isn't as cheap as it could be (like RC eliding).
January 23, 2023
On Monday, 23 January 2023 at 07:06:04 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 23/01/2023 10:31 AM, Paul Backus wrote:
>> Can you explain more about @localsafe? I don't understand how this is different from Dukc's proposal in the linked thread.
>
> Its @safe except you can call non-safe functions. Same goes for nogc and pure.

In this context, that makes it no different from @trusted.

The problem is that, in a generic allocator-aware container, if you write a @trusted/@localsafe call to RCAllocator.deallocate, there is nothing to stop someone from writing a custom allocator with a deallocate function like this:

struct NaughtyAllocator
{
    // ...

    @system void deallocate(void[] block)
    {
         corruptMemory();
    }
}

...and then RCAllocator.deallocate will dispatch to this function, and you will end up corrupting memory in @safe code.
« First   ‹ Prev
1 2 3 4 5 6 7