Jump to page: 1 2
Thread overview
Future of SafeRefCounted
Feb 22
Dukc
Feb 22
Dukc
Feb 23
Dukc
Feb 23
Dennis
Feb 24
Dukc
Feb 25
Dukc
Mar 06
Dukc
February 22

SafeRefCounted depends on DIP1000 to work. Because of that, it was requested early in the PR review adding it that it should only be @safe when DIP1000 is on, not otherwise. This sounded obviously a good idea back then, so the destructor of SafeRefCounted has a static if that checks for DIP1000 and does the deallocation inside a @trusted lambda if the switch is enabled, directly otherwise.

This makes the destructor @system for non-DIP1000 client code, @safe for DIP1000 code. But, I've come to think it'd have been better to leave it @safe for all code.

Have you looked at std.dirEntries recently? It has a compile-time template parameter called useDIP1000, that is always set automatically and is even checked with a static assert to be set to be default option. Isn't this crazy? But believe or not, the unit tests don't pass without the template parameter!

The issue is, the return value of DirEntries is implemented with SafeRefCounted. This means it, like SafeRefCounted itself, is safe iff DIP1000 is on. Without the useDIP1000 flag, the linker can attempt linking to non-DIP1000 dirEntries from dip1000 client code or vice-versa, causing an ABI mismatch since the safety of the function affects it's name mangling.

Imagine some average user trying to define their own type with SafeRefCounted and getting linker errors like that. When I encountered the linker errors, I knew that I had just made SafeRefCounter safety to depend on the DIP1000 flag, but it was still very tricky to figure out what was happening and how to solve it. We certainly can't expect the same from users.

Now, maybe this would let non-DIP1000 users to write unsafe code in @safe. I'm not actually sure if that's the case, but assuming it is I still think it would be the lesser evil here. Remember, @safe has pointer escaping holes in non-DIP1000 code anyway. I don't see how the possibility of the std.typecons.borrow argument function escaping the reference to the payload is any different from the possibility of leaking a slice of a static array in the stack.


Which leads to another issue: language modules and Phobos 3.

My understanding is the current plan with Phobos 3 is to not use DIP1000 in it. SafeRefCounted depends on DIP1000. Without it it isn't any better than the old RefCounted type.

Surprisingly, this doesn't automatically mean it can't be done in Phobos 3. Within the present language it wouldn't be possible, but modules are going to change the calculus.

In the future when DIP1000 is on or off per module as opposed to compilation unit, it is going to the client code which has to be DIP1000 for ref counting to be safe. The reference counted type can be implemented with our without DIP1000 just as well (although non-DIP1000 implementation of borrow would probably need a currently missing ability to manually inspect the scopeness of the argument function - currently the safety is inferred by calling the argument function with a scope argument which probably doesn't work outside DIP1000. Also @safety unit tests of it would need DIP1000.).

Modules are also one more reason to do away with inspecting for DIP1000. With modules, the current definition of the inspection

private auto dip1000Test(int x) {return *&x;}
// Used in the `static if`
package(std) enum dip1000Enabled
    = is(typeof(&dip1000Test) : int function(int) @safe);

is going to inspect Phobos, not the client code. While we're probably going to have a way to inspect client code instead, IMO it is a code smell if you're inspecting the module / completion flags of your client code.

What are your thoughts?

February 22

On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

>

But believe or not, the unit tests don't pass without the template parameter!

Correction: Unit tests do pass, but the test suite as whole gives a linker error in apparently unrelated place.

February 23

On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

>

Which leads to another issue: language modules and Phobos 3.

Wow, how did I write about modules all the way down without noticing? I meant editions in all but one instance. So:

Which leads to another issue: language editions and Phobos 3.

My understanding is the current plan with Phobos 3 is to not
use DIP1000 in it. SafeRefCounted depends on DIP1000. Without
it it isn't any better than the old RefCounted type.

Surprisingly, this doesn't automatically mean it can't be done
in Phobos 3. Within the present language it wouldn't be
possible, but editions are going to change the calculus.

In the future when DIP1000 is on or off per module as opposed
to compilation unit, it is going to the client code which has
to be DIP1000 for ref counting to be safe. The reference
counted type can be implemented with our without DIP1000 just
as well (although non-DIP1000 implementation of borrow would
probably need a currently
missing
ability to
manually inspect the scopeness of the argument function -
currently the safety is inferred by calling the argument
function with a scope argument which probably doesn't work
outside DIP1000. Also @safety unit tests of it would need
DIP1000.).

Editions are also one more reason to do away with inspecting for
DIP1000. With editions, the current definition of the inspection

private auto dip1000Test(int x) {return *&x;}
// Used in the `static if`
package(std) enum dip1000Enabled
    = is(typeof(&dip1000Test) : int function(int) @safe);

is going to inspect Phobos, not the client code. While we're
probably going to have a way to inspect client code instead,
IMO it is a code smell if you're inspecting the edition /
completion flags of your client code.

What are your thoughts?

February 23

On Saturday, 22 February 2025 at 19:05:49 UTC, Dukc wrote:

>

This sounded obviously a good idea back then, so the destructor of
SafeRefCounted has a static if that checks for DIP1000 and
does the deallocation inside a @trusted lambda if the switch
is enabled, directly otherwise.
(...)
What are your thoughts?

To me it always looked hacky, and I'd get rid of it as well.

On the other hand, I think reference counting sucks in general, and see little value in using it in Phobos. Especially in functions like dirEntires, as if that is going to be called a lot in performance critical real-time code.

If you want to get rid of the GC dependency, you can use arena allocation. (I talked about this last DConf)

February 24

On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

>

On the other hand, I think reference counting sucks in general, and see little value in using it in Phobos. Especially in functions like dirEntires, as if that is going to be called a lot in performance critical real-time code.

I disagree, reference counting is a great tool for this use case. Not because of the memory, but because if the directory handles. Were the dirEntries implementation garbage collected, the directory handles would get released only when the collection happens, which could never happen if the program doesn't use much RAM.

Admittedly this kind of reference counting doesn't mandate using SafeRefCounted - the reference counter could just free the directory handles and leave the RAM management to the GC, and this would be totally @safe.

When I think of it, maybe DirEntries should have the SafeRefCounted field as an untyped void[x] field instead and have a manually written @trusted destructor to call the SafeRefCounted destructor, so the ABI of dirEntries/DirEntries isn't affected. Ironically this means there's no benefit anymore from using SafeRefCounted over the old RefCounted.

February 24

On Monday, 24 February 2025 at 17:18:55 UTC, Dukc wrote:

>

On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

>

On the other hand, I think reference counting sucks in general, and see little value in using it in Phobos. Especially in functions like dirEntires, as if that is going to be called a lot in performance critical real-time code.

I disagree, reference counting is a great tool for this use case. Not because of the memory, but because if the directory handles. Were the dirEntries implementation garbage collected, the directory handles would get released only when the collection happens, which could never happen if the program doesn't use much RAM.

For resource management with deterministic lifecycle, I usually prefer non-copyable structs with refs. Only in the rare case when they need to be accessible in multiple parts of the code would I resort to ref counting.

February 24
On Saturday, February 22, 2025 12:05:49 PM MST Dukc via Digitalmars-d wrote:
> What are your thoughts?

Personally, I'm very much hoping that DIP 1000 gets killed off. There are some niche cases where it's useful, but in general, it's just too disruptive. In order to support it in general, code that doesn't care at all about it has to be marked with scope in order for the code that does care about it to be able to use that code. And the reason that the current plan is for Phobos v3 to not do anything with DIP 1000 is because a number of us feel that way. Atila is looking to see if he can come up with a way to make it work without being so disruptive, but I question that that's going to work, whereas others like Rikki are looking to do something else entirely which will theoretically cover more than DIP 1000 does. So, I don't know what we're going to end up with, but realistically, if a type requires DIP 1000 in order to be @safe, then it's likely not going to be @safe in Phobos v3.

As for RefCounted specifically, I haven't looked at its implementation recently, so I don't know why it can't be @safe, but off the top of my head, I don't see why it can't be possible to create something like RefCounted that is @safe. Maybe some part of what the current design does would have to be disallowed, but it sure seems like it should be possible to have reference counting work with @safe so long as you don't allow references to the internals to escape. And if for some reason, it can't be, well, then either the code using it will need to wrap it in a way that allows for @safe, or it can't be @safe.

And for the vast majority of code, nothing like RefCounted is needed. The GC largely solves the problem with regards to memory management, and the cases where you want or need to do something else will probably want a solution that's tied to whatever allocators are being used rather than a more general ref-counting solution (assuming that reference counting is wanted at all). It's mostly just system resources that need to be deterministically freed / destroyed where reference counting becomes necessary. So, IMHO, we do need a solution for reference counting in Phobos v3, but it's also niche enough that I don't think that it's the end of the world if some @trusted is required for it - much as I agree that it would ideally be @safe. But again, I'd have to sit down and work through the problem to say for sure whether it can be @safe without DIP 1000. Either way, as things stand, DIP 1000 is not going to be a solution for making anything in Phobos v3 @safe.

- Jonathan M Davis



February 25
On 25/02/2025 1:17 PM, Jonathan M Davis wrote:
> On Saturday, February 22, 2025 12:05:49 PM MST Dukc via Digitalmars-d wrote:
> 
> 
>     What are your thoughts?
> 
> 
> Personally, I'm very much hoping that DIP 1000 gets killed off. There are some niche cases where it's useful, but in general, it's just too disruptive. In order to support it in general, code that doesn't care at all about it has to be marked with scope in order for the code that does care about it to be able to use that code. And the reason that the current plan is for Phobos v3 to not do anything with DIP 1000 is because a number of us feel that way. Atila is looking to see if he can come up with a way to make it work without being so disruptive, but I question that that's going to work, whereas others like Rikki are looking to do something else entirely which will theoretically cover more than DIP 1000 does. So, I don't know what we're going to end up with, but realistically, if a type requires DIP 1000 in order to be @safe, then it's likely not going to be @safe in Phobos v3.

I'm exploring it from both directions.

Trying to model more things (without doing whole program analysis), and have the default be "to give up" when it cannot analyze what it does or doesn't understand.

Basically get the best of both worlds by having a fast and a slow DFA engine.

It'll mean that PhobosV3 would have DFA language features, with some guarantees. There may even be whole modules that support the slow DFA, but we'll see about it as a stretch goal.

February 25

On Sunday, 23 February 2025 at 20:02:03 UTC, Dennis wrote:

>

If you want to get rid of the GC dependency, you can use arena allocation. (I talked about this last DConf)

In general I approve of using arena allocators. RC is useful, but I'd like to avoid RC as much as possible in Phobos. We have a GC, and we're getting a vastly improved one. For cases where the GC cannot be used, almost anything is better than RC.

February 25
On Tuesday, 25 February 2025 at 00:17:19 UTC, Jonathan M Davis wrote:
> On Saturday, February 22, 2025 12:05:49 PM MST Dukc via Digitalmars-d wrote:
>> What are your thoughts?
>
> Personally, I'm very much hoping that DIP 1000 gets killed off. There are some niche cases where it's useful [...]

I use dip1000 in two ways, firstly to avoid allocations on the heap and secondly to safely share references to objects with limited lifetime. Often both at the same time.

Avoiding allocations (GC or malloc) allows the program to run faster with fewer resources. I personally value efficient and safe programs, and IMO it should be a core focus of the language.

I do see challenges with dip1000's usage, but relegating it by saying its only useful for niche cases neglects a whole set of efficient programs. I understand not everyone cares about them to the same extent, but don't forget there is a chicken-and-egg problem as well: if it's relatively hard to write efficient programs, you'll tend to filter out the people who care about them.

> Either way, as things stand, DIP 1000 is not going to be a solution for making anything in Phobos v3 @safe.

That is a real shame. That leaves implementing them either @system, inefficiently or not at all.
« First   ‹ Prev
1 2