In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.

So I thought I'd make a refCounted struct that uses the GC [1]. The concept is simple -- allocate the refCounted payload in a GC block, then pin the block as a root. Once all references are gone, remove the root. But the memory stays behind to keep things memory safe (if, for example, you saved a pointer to it outside a reference count object). The memory will be in an initial state, but not invalid.

This means that if you include it in e.g. an array or a class, then it still should work correctly (the memory is guaranteed to be present, and anything it points at).

Of course, you can have cycles that prevent it ever from being cleaned up. But most of the time, this is for auto cleaning up stack items. So maybe that's OK?

Let me know what you think. It sucks that we have no valid way to do reference counting in safe code, because std.io and iopipe highly depend on it.

-Steve

[1] https://github.com/schveiguy/iopipe/blob/makesafe/source/iopipe/refc.d

On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>
> So I thought I'd make a refCounted struct that uses the GC [1]. The concept is simple -- allocate the refCounted payload in a GC block, then pin the block as a root. Once all references are gone, remove the root. But the memory stays behind to keep things memory safe (if, for example, you saved a pointer to it outside a reference count object). The memory will be in an initial state, but not invalid.
>
> This means that if you include it in e.g. an array or a class, then it still should work correctly (the memory is guaranteed to be present, and anything it points at).
>
> Of course, you can have cycles that prevent it ever from being cleaned up. But most of the time, this is for auto cleaning up stack items. So maybe that's OK?
>
> Let me know what you think. It sucks that we have no valid way to do reference counting in safe code, because std.io and iopipe highly depend on it.
>
> -Steve
>
> [1] https://github.com/schveiguy/iopipe/blob/makesafe/source/iopipe/refc.d

Is it not enough to add a flag to RefCounted that tells it to destroy (but not deallocate) the ref counted object, instead of deallocating it? Then it could be conditionally @safe.

On 5/1/20 10:52 PM, Meta wrote:
> Is it not enough to add a flag to RefCounted that tells it to destroy (but not deallocate) the ref counted object, instead of deallocating it? Then it could be conditionally @safe.

You mean for Phobos? Phobos uses C malloc/free, so no. Something has to deallocate it.

If you mean something else, I'm not sure what you mean.

-Steve

On Saturday, 2 May 2020 at 03:31:55 UTC, Steven Schveighoffer wrote:
> On 5/1/20 10:52 PM, Meta wrote:
>> Is it not enough to add a flag to RefCounted that tells it to destroy (but not deallocate) the ref counted object, instead of deallocating it? Then it could be conditionally @safe.
>
> You mean for Phobos? Phobos uses C malloc/free, so no. Something has to deallocate it.
>
> If you mean something else, I'm not sure what you mean.
>
> -Steve

Err... right. I wasn't thinking clearly.

On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>
> So I thought I'd make a refCounted struct that uses the GC [1]. The concept is simple -- allocate the refCounted payload in a GC block, then pin the block as a root. Once all references are gone, remove the root. But the memory stays behind to keep things memory safe (if, for example, you saved a pointer to it outside a reference count object). The memory will be in an initial state, but not invalid.
>
> This means that if you include it in e.g. an array or a class, then it still should work correctly (the memory is guaranteed to be present, and anything it points at).
>
> Of course, you can have cycles that prevent it ever from being cleaned up. But most of the time, this is for auto cleaning up stack items. So maybe that's OK?
>
> Let me know what you think. It sucks that we have no valid way to do reference counting in safe code, because std.io and iopipe highly depend on it.

For my small memory buffer mgmt library I use next solution - library user can't have raw pointers to memory (I understand this is not your case with file etc), It can only have
unique_ptr to the mutable memory chunk which user can fill with some data(from file/network), and also can convert this unique_ptr to ref_counted (to immutable view of this memory), and this destroys uniq_ptr, so it can't be used anymore. It is safe to keep ref_counted in arrays.

PS. it is better to make things @nogc from beginning

>
> -Steve
>
> [1] https://github.com/schveiguy/iopipe/blob/makesafe/source/iopipe/refc.d

On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>
> [...]

Are you aware of the stalled work to get reference counting in druntime?

https://github.com/dlang/druntime/pull/2679

https://github.com/dlang/druntime/pull/2646

https://github.com/dlang/druntime/pull/2760

On 5/2/20 7:14 AM, Seb wrote:
> On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
>> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>>
>> [...]
> 
> Are you aware of the stalled work to get reference counting in druntime?
> 
> https://github.com/dlang/druntime/pull/2679
> 
> https://github.com/dlang/druntime/pull/2646
> 
> https://github.com/dlang/druntime/pull/2760

No, but these are not what I'm focused on.

I want a @safe API, I don't care about @nogc in iopipe. The problem with letting the GC clean up your non-memory resources is that it may never happen, or it may happen at a much later time. A process can have a lot of memory, it's open file descriptors are much more limited.

But the memory used to implement said resources I'm fine leaving to the GC to clean up. In other words, I want my files to close as soon as I no longer need them, but the shell being used to store the file info (e.g. buffer etc) can stick around while something has a pointer to it.

I don't know if reference counting without GC can be made safe for D.

ikod has a good plan, don't give access to the actual data. But it falls apart in practice, because then you can't use standard functions, everything has to use reference counted pointers and arrays. Not only that, but a generic library can easily give away the data without you wanting it to. You have to be really cautious about how you use the reference counting.

I like the plan of letting the GC ensure the memory is always valid. I'm just not sure about the problem of cycles. That's the one place this might fall down.

-Steve

On 5/2/20 2:16 PM, Steven Schveighoffer wrote:
> ikod has a good plan, don't give access to the actual data. But it falls apart in practice, because then you can't use standard functions, everything has to use reference counted pointers and arrays. Not only that, but a generic library can easily give away the data without you wanting it to. You have to be really cautious about how you use the reference counting.

As an example, if you have something like:

struct S
{
   int[100] buf;

   @disable this(this);

   int[] opSlice() { return buf[]; }
}

Now, reference counting this struct, you can prevent direct access to the S instance, but just calling x[] now gives you naked access to the data.

How do you solve this issue in D and still make it @safe without using the GC?

-Steve

On Saturday, May 2, 2020 12:16:13 PM MDT Steven Schveighoffer via Digitalmars-d wrote:
> I like the plan of letting the GC ensure the memory is always valid. I'm just not sure about the problem of cycles. That's the one place this might fall down.

As I understand it, as long as nothing that the program still has access to refers to the objects with circular references, the cycle won't be a problem, and the GC will be able to collect them. I recall Andrei talking in the past about having reference counting which did basically what you're describing with the GC being left to handle the cycles.

- Jonathan M Davis

On 03/05/2020 6:23 AM, Steven Schveighoffer wrote:
> How do you solve this issue in D and still make it @safe without using the GC?

(head)const + DIP25/1000 would be a good place to begin.

I would love to have a headconst tied to lifetimes. Most of the work has already been done, it just needs exposing in a nice way. I'm not sure if @live is the right way to do this though.