On 31/01/2023 5:27 AM, Paul Backus wrote:
> I don't understand why you keep bringing this up--it's totally beside the point. Obviously most users should not need to use the allocator API directly.

In the above case, with @system you could pass in whatever pointer you want or extract the pointer out and lose any protection that the given struct could provide. If the struct is done simply, you could do this with @safe easily as well :/ So not really appropriate for an allocator to return.

So you remain in audit only territory of it.

> However, if you are *implementing* a data structure like a dynamic array, and you want to support user-supplied custom allocators, then the only way your data structure can be @safe is if the allocator API uses this kind of wrapper type to present a @safe interface.

No?

The data structure is the one doing the lifetime management. It takes over from the language to keep the guarantees in check.

Ultimately, data structures and algorithms typically need auditing to make sure they are doing what they are supposed to be doing. I strongly believe adding a whole lifetime tracking feature in what should be library code won't have the ROI in adding it to the language.

On that note, Dennis has done some work to replace some scope inference which should offer the infrastructure required to plug the last big hole in lifetime tracking of borrowed memory from a data structure :) https://github.com/dlang/dmd/pull/14492

On Monday, 30 January 2023 at 17:01:38 UTC, Richard (Rikki) Andrew Cattermole wrote:
> On 31/01/2023 5:27 AM, Paul Backus wrote:
>> I don't understand why you keep bringing this up--it's totally beside the point. Obviously most users should not need to use the allocator API directly.
>
> In the above case, with @system you could pass in whatever pointer you want or extract the pointer out and lose any protection that the given struct could provide. If the struct is done simply, you could do this with @safe easily as well :/ So not really appropriate for an allocator to return.
>
> So you remain in audit only territory of it.

We can use DIP 1035's @system variables to ensure that the wrapper struct's internals are not meddled with.

>> However, if you are *implementing* a data structure like a dynamic array, and you want to support user-supplied custom allocators, then the only way your data structure can be @safe is if the allocator API uses this kind of wrapper type to present a @safe interface.
>
> No?
>
> The data structure is the one doing the lifetime management. It takes over from the language to keep the guarantees in check.

It seems as though you have completely failed to grasp the essential point of my original post [1].

In order for a data structure to provide these guarantees to its users, the allocator must, in turn, provide certain guarantees *to the data structure*. And if the data structure does not know in advance which allocator it will be using (e.g., if it is calling a generic RCAllocator.deallocate function), then those guarantees must be *encoded in the type system* somehow, so that @safe code cannot accidentally break them.

[1] https://forum.dlang.org/post/jsuraddtynhjoaikqprs@forum.dlang.org

On 31/01/2023 6:18 AM, Paul Backus wrote:
> It seems as though you have completely failed to grasp the essential point of my original post [1].
> 
> In order for a data structure to provide these guarantees to its users, the allocator must, in turn, provide certain guarantees *to the data structure*. And if the data structure does not know in advance which allocator it will be using (e.g., if it is calling a generic RCAllocator.deallocate function), then those guarantees must be *encoded in the type system* somehow, so that @safe code cannot accidentally break them.

Unfortunately I have indeed completely grasped it.

I genuinely do not believe it will have the ROI that others seem to think it will.

Walter has shown that he does not want the DFA that I strongly suspect is required to do this properly and therefore trying to solve this will only result in wasted effort when we could get almost there without it within our (ok sometimes artificial) limitations.

On Monday, 30 January 2023 at 17:22:56 UTC, Richard (Rikki) Andrew Cattermole wrote:
> Unfortunately I have indeed completely grasped it.
>
> I genuinely do not believe it will have the ROI that others seem to think it will.
>
> Walter has shown that he does not want the DFA that I strongly suspect is required to do this properly and therefore trying to solve this will only result in wasted effort when we could get almost there without it within our (ok sometimes artificial) limitations.

As far as I am aware, it is impossible to have all three of the following:

1. @safe containers.
2. User-supplied allocators.
3. No language changes.

Given Walter and Atila's stance on memory safety, (1) is non-negotiable, so the question is whether we prefer (1)+(2) (implement isolated or something similar) or (1)+(3) (do not allow users to define their own allocators). I myself am not sure which of these would have better ROI.

In this post, it sounds as though you are advocating for (1)+(3). But in your previous post [1], you replied to my claim that (1)+(2) requires not-(3) with "No? The data structure is the one doing the lifetime management," which suggested to me that you believe (1)+(2)+(3) is actually possible somehow.

If you agree with me that (1)+(2)+(3) is impossible, then I have no objections to your other claims. If you believe that (1)+(2)+(3) is possible, on the other hand, then I would very much like to hear how.

[1] https://forum.dlang.org/post/tr8t5i$2gt5$1@digitalmars.com

We'll have language changes for helping with this, I can't see us not making them if opportunities arise ;) Especially if they are small and have multiple use cases.

Now, 1. and 2. I want to differentiate between effectively @safe, and actually machine check-able @safe.

I do not believe we will ever have fully machine check able. That means significant DFA, we really don't have the required people to design and implement this. DIP1000 is a good example of this, since it doesn't support indirection with multiple lifetimes being involved in a variable.

Effectively @safe means as much code is machine checked, but we want to isolate to library code the unsafe parts, where we ensure those guarantees for what we can by auditing instead.

So no I don't think we can have 1/2 without hiring some people, but we can get close enough to it with what resources we do have, just by telling people to not use something and push them instead to use things that do offer it as long you don't do something outright stupid (which lets face it, they probably won't be using anything other than the default global allocator).

On Monday, 30 January 2023 at 17:51:52 UTC, Richard (Rikki) Andrew Cattermole wrote:
> Effectively @safe means as much code is machine checked, but we want to isolate to library code the unsafe parts, where we ensure those guarantees for what we can by auditing instead.

Yes; I assumed that this went without saying. Even with language features like an isolated qualifier to help us, it will still be necessary for the data structures and allocators to use @trusted code internally.

> So no I don't think we can have 1/2 without hiring some people, but we can get close enough to it with what resources we do have, just by telling people to not use something and push them instead to use things that do offer it as long you don't do something outright stupid (which lets face it, they probably won't be using anything other than the default global allocator).

I am afraid that this description is far too vague for me to understand what you have in mind here. Are you advocating for (1)+(3), (2)+(3), or maybe some hybrid of both? Like, if you use one of the Officially Blessed allocators, the container will be @safe, and if you use a 3rd-party custom allocator, it'll be @system?

On 31/01/2023 7:19 AM, Paul Backus wrote:
>> So no I don't think we can have 1/2 without hiring some people, but we can get close enough to it with what resources we do have, just by telling people to not use something and push them instead to use things that do offer it as long you don't do something outright stupid (which lets face it, they probably won't be using anything other than the default global allocator).
> 
> I am afraid that this description is far too vague for me to understand what you have in mind here. Are you advocating for (1)+(3), (2)+(3), or maybe some hybrid of both? Like, if you use one of the Officially Blessed allocators, the container will be @safe, and if you use a 3rd-party custom allocator, it'll be @system?

If we hired some people then yes 1&2 (but not 3). Otherwise we are stuck with changing the goal posts from perfect to good enough for the time being. As Andrei use to say: perfect is the enemy of the good; which is unfortunately what I think we need to strongly consider here. We have what appears to be a pretty decent path forward for 'good'.

It shouldn't matter what allocator library you use. Either it works or it doesn't. The compiler shouldn't care whose code it is, only the lifetime patterns.

On Monday, 30 January 2023 at 18:33:57 UTC, Richard (Rikki) Andrew Cattermole wrote:
> If we hired some people then yes 1&2 (but not 3). Otherwise we are stuck with changing the goal posts from perfect to good enough for the time being. As Andrei use to say: perfect is the enemy of the good; which is unfortunately what I think we need to strongly consider here. We have what appears to be a pretty decent path forward for 'good'.

I'm not opposed to the idea of moving the goalposts in principle, I am just not sure where you are proposing that we move them to. :) It seems like every time I ask, I get a different answer.

> It shouldn't matter what allocator library you use. Either it works or it doesn't. The compiler shouldn't care whose code it is, only the lifetime patterns.

For example: I agree with this paragraph in principle, but isn't this one of the goalposts that you were just saying we should be willing to move?

On 31/01/2023 7:49 AM, Paul Backus wrote:
> On Monday, 30 January 2023 at 18:33:57 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> If we hired some people then yes 1&2 (but not 3). Otherwise we are stuck with changing the goal posts from perfect to good enough for the time being. As Andrei use to say: perfect is the enemy of the good; which is unfortunately what I think we need to strongly consider here. We have what appears to be a pretty decent path forward for 'good'.
> 
> I'm not opposed to the idea of moving the goalposts in principle, I am just not sure where you are proposing that we move them to. :) It seems like every time I ask, I get a different answer.

I'm not expressing myself clearly enough but the goal posts I'm recommending are the same as at the start. Check as much as possible with @safe, but let data structures be responsible for as much of the life time guarantees wrt. allocators as possible.

Perfect solution: all code is mechanically checked.

Good solution: library and under the hood code gets audited, user code of the library code gets mechanically checked.

Let's have a good solution, then try to figure out if we can remove restrictions and get a perfect solution in the library code as well.

>> It shouldn't matter what allocator library you use. Either it works or it doesn't. The compiler shouldn't care whose code it is, only the lifetime patterns.
> 
> For example: I agree with this paragraph in principle, but isn't this one of the goalposts that you were just saying we should be willing to move?

I'm half and half. Stuff like @localsafe would mean that I think that allocators should all be @system in API. I'm genuinely reconsidering it for my own because of this thread.

But in terms of hard coding into the compiler std.allocators vs sidero.base.allocators yeah no. It shouldn't do that, which is what the quoted statement was about. If both API's match, then the compiler should treat them the same way in terms of lifetime management.