On Saturday, 2 May 2020 at 11:14:23 UTC, Seb wrote:
> On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
>> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>>
>> [...]
>
> Are you aware of the stalled work to get reference counting in druntime?
>
> https://github.com/dlang/druntime/pull/2679
>
> https://github.com/dlang/druntime/pull/2646
>
> https://github.com/dlang/druntime/pull/2760

Do you know what happened to the whole __mutable ?

On Saturday, 2 May 2020 at 18:23:36 UTC, Steven Schveighoffer wrote:
> On 5/2/20 2:16 PM, Steven Schveighoffer wrote:
>> ikod has a good plan, don't give access to the actual data. But it falls apart in practice, because then you can't use standard functions, everything has to use reference counted pointers and arrays. Not only that, but a generic library can easily give away the data without you wanting it to. You have to be really cautious about how you use the reference counting.
>
> As an example, if you have something like:
>
> struct S
> {
>    int[100] buf;
>
>    @disable this(this);
>
>    int[] opSlice() { return buf[]; }
> }
>
> Now, reference counting this struct, you can prevent direct access to the S instance, but just calling x[] now gives you naked access to the data.
>
> How do you solve this issue in D and still make it @safe without using the GC?

Yes there is no magic and there are lot of limitations and inconveniences, but at least I know where data can leak.

Here is gist with code sample and comments: https://gist.github.com/ikod/2c35851581b59677a0d9511812592df0

>
> -Steve

On 5/2/20 2:38 PM, Jonathan M Davis wrote:
> On Saturday, May 2, 2020 12:16:13 PM MDT Steven Schveighoffer via
> Digitalmars-d wrote:
>> I like the plan of letting the GC ensure the memory is always valid. I'm
>> just not sure about the problem of cycles. That's the one place this
>> might fall down.
> 
> As I understand it, as long as nothing that the program still has access to
> refers to the objects with circular references, the cycle won't be a
> problem, and the GC will be able to collect them. I recall Andrei talking in
> the past about having reference counting which did basically what you're
> describing with the GC being left to handle the cycles.

This is different. I'm pinning the blocks so they won't be collected until all "appropriate" (e.g. "counted") references are no more.

Essentially, what I want is ref counting for resource management, but instead of freeing the memory, I'm releasing it for the GC to clean up.

So there is definitely the possibility of cycles.

-Steve

On 5/2/20 4:28 PM, ikod wrote:
> Yes there is no magic and there are lot of limitations and inconveniences, but at least I know where data can leak.
> 
> Here is gist with code sample and comments: https://gist.github.com/ikod/2c35851581b59677a0d9511812592df0
>

OK, so essentially you need to have a lot of trusted escapes. I'm looking for something that doesn't need that by default for useful code.

Though that does look correct in terms of memory safety.

My biggest problem with doing something like that is that for instance, an iopipe buffered output stream uses reference counting to ensure that once all references to the output stream are done, the final data in the buffer is flushed to the output. This is a perfect fit for reference counting, but of course, it can't be made @safe because iopipe provides direct buffer access (that is part of the design).

So I want something @safe that provides direct buffer access, and also can clean up whatever needs cleaning synchronously (e.g. closing files, flushing data, etc). Developers are not going to be keen to a buffered file flushing it's buffer at sometime in the future (or never).

AND I would like it to be storable inside a GC block (many people don't realize that std.typecons.RefCounted isn't valid to put in the GC when you have multiple threads).

-Steve

On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.

I was solving a much more constrained problem, but I wrote a couple of one-pass input ranges over a set of files, providing open file access to each in turn. Has the nice benefit of closing the file immediately when it is popFront'ed off the range. There's nothing preventing the caller from holding a copy of the underlying File object, but it is only open while it is the front element of the range. If the concept might be of interest, the code is here:
* https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L1968
* https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L2344

--Jon

On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>
> [...]

I’m late to the party.

I think you could use one of the allocation schemes proposed by Alexandru Jercaianu in his dconf talk [2]

[2] - https://youtu.be/kaA3HPgowwY

On 10/27/20 7:55 PM, Eduard Staniloiu wrote:
> On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
>> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>>
>> [...]
> 
> I’m late to the party.
> 
> I think you could use one of the allocation schemes proposed by Alexandru Jercaianu in his dconf talk [2]

Thanks, I had forgotten about this (you can see my bald head in the video, so I was there ;)

It's an interesting opportunity. Perhaps there is room for a RefCounted type that allows multiple memory allocation schemes. I think the concept I came up with back then is pretty solid, it's since been released in iopipe. That will possibly allow iopipes in @nogc code that need ref counting.

The concept is the same -- I want to synchronously clean up the type, but leave the memory for something else to clean up, or flag as an error.

-Steve

On Tuesday, 27 October 2020 at 23:55:04 UTC, Eduard Staniloiu wrote:
> On Saturday, 2 May 2020 at 02:27:10 UTC, Steven Schveighoffer wrote:
>> In trying to make iopipe @safe, I came to the realization that having auto-managed items such as files and the like (std.io Files and Sockets are non-copyable), you need to rely on some form of @safe reference counting. Unfortunately std.typecons.RefCounted is not and cannot be safe. This is because it allocates in the C heap, and deallocates regardless of whether anyone has ever squirreled away a reference.
>>
>> [...]
>
> I’m late to the party.
>
> I think you could use one of the allocation schemes proposed by Alexandru Jercaianu in his dconf talk [2]
>
> [2] - https://youtu.be/kaA3HPgowwY

That's an interesting talk actually. What's the current status?