On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis wrote:
> On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl via Digitalmars-d wrote:
>> On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis
>>
>> wrote:
>> > The issue isn't the object being destroyed. It's what it refers to via its member variables. For instance, what if an object were to remove itself from a shared list when it's destroyed (e.g. because it's an observer in the observer pattern). The object has a reference to the list, but it doesn't own it.
>>
>> So, even a thread-local object that has references to a shared
>> list
>> has to handle those as shared, even in its non-shared destructor.
>> I can't follow your argument.
>
> You can't just strip off shared. To do so defeats the purpose of shared. If you have something like
>
> struct S
> {
>      shared List<Foo> _list;
>
>      ~this()
>      {
>         ...
>      }
> }

This is fine. What dmd does now is strip shared off of the `this` pointer, not the member variables. There's only a problem if the sharedness of the member variable(s) depends on sharedness of the enclosing object.

> then inside of the destructor, _list is not treated as shared,

It is.

Atila

On Thursday, 20 July 2017 at 10:15:26 UTC, Kagamin wrote:
> On Wednesday, 19 July 2017 at 20:59:03 UTC, Atila Neves wrote:
>> Not necessarily - the reference counted smart pointer doesn't have to be `shared` itself to have a `shared` payload.
>
> Yes, but it can be done either way. It's actually what Jack is trying to do: make stdout shared and reference counted: https://issues.dlang.org/show_bug.cgi?id=15768#c7
>
>> I'm not even entirely sure what the advantage of it being `shared` would be, or even what that would really mean.
>
> It will be thread safe and its lifetime will be automatically managed.
>
>> You've definitely made me wonder about complicated cases, but I'd argue that they'd be rare. Destructors are (bar manually calling them) run in one thread. I'm having trouble imagining a situation where two threads have references to a `shared` object/value that is going to be destroyed deterministically.
>
> A mutex, a file, a socket, any shareable resource. Though I agree that reference counting of shared resources should be optimized by thread local counters.

Mutexes and sockets are classes, so not destroyed deterministically.

Anything that is `shared` is likely to be a reference (pointer, class...), or a global. Either way the compiler-generated destructor call isn't going to exist, which means it's probably ok to cast away shared when the compiler inserts the automatic call to a destructor at the end of scope.

Atila

On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:
> On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:
>> On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:
>>> TL;DR: Issue 17658 [1] makes using shared very annoying/practically impossible.
>>>
>>> [...]
>>
>> I fixed this already, should be in the next release.
>>
>> Atila
>
> Are you sure? Because DMD nightly still errors:
>
> https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D
>
>
> (maybe we should remove the ban on URL shorteners for our own sites)

This works fine in dmd 2.075:

struct A
{
    this(string a) {}
    this(string a) shared {}

    ~this() {}

    this(this T)(this) {} // you can reflect to find out if shared
}

void main()
{
    auto nonShared = A("");
    auto shared_ = shared A("");
    auto nonSharedCopy = nonShared;
    auto sharedCopy = shared_;
}


Atila

On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:
> Mutexes and sockets are classes, so not destroyed deterministically.

They should, like any unmanaged resources, e.g. by wrapping in a smart pointer. Imagine 10000 lingering tcp connections accumulated over time due to poor timing of destruction, it becomes a stress test for the server.

> Anything that is `shared` is likely to be a reference (pointer, class...), or a global. Either way the compiler-generated destructor call isn't going to exist, which means it's probably ok to cast away shared when the compiler inserts the automatic call to a destructor at the end of scope.

These are contradictory. Does the automatic destructor call exist or not?

Hmm, if proper implementation of a shared smart pointer is impossible, it probably means that such smart pointer should be typed unshared when passed around. But then it doesn't make sense to call unshared destructor on shared smart pointer anyway, because it's not designed to be typed shared. In this case absence of shared destructor will indicate that the object doesn't support being shared and the compiler should reject the code.

On Friday, 21 July 2017 at 11:57:11 UTC, Kagamin wrote:
> On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:
>> Mutexes and sockets are classes, so not destroyed deterministically.
>
> They should, like any unmanaged resources

I tend to agree, although

> e.g. by wrapping in a smart pointer.

objects that manage their own lifetime limit the design space unnecessarily.
It's better to use normal structs to wrap resources (RAII) and then build whatever object lifetime management scheme one wants (including reference counting) on top of that.

On Friday, 21 July 2017 at 11:57:11 UTC, Kagamin wrote:
> On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:
>> Mutexes and sockets are classes, so not destroyed deterministically.
>
> They should, like any unmanaged resources, e.g. by wrapping in a smart pointer. Imagine 10000 lingering tcp connections accumulated over time due to poor timing of destruction, it becomes a stress test for the server.
>
>> Anything that is `shared` is likely to be a reference (pointer, class...), or a global. Either way the compiler-generated destructor call isn't going to exist, which means it's probably ok to cast away shared when the compiler inserts the automatic call to a destructor at the end of scope.
>
> These are contradictory. Does the automatic destructor call exist or not?

What I'm trying to say is that `shared` values will usually be references or globals and therefore there won't be a compiler-generated call to the destructor at the end of scope.

When the compiler _does_ generate a call to the destructor, the value is unlikely to be shared.

Since then I've thought that sending a value to another thread does indeed create a shared value with defined scope. Or, for that matter, calling any function that takes shared values (but those are rare, so it'll usually be `send`).

I think I've not done a good job of explaining the destructor fix: what it changes is the code the compiler writes for you when variables go out of scope, i.e. the automatic destructor call casts away shared. It already did the same thing for immutable, otherwise you wouldn't be able to put immutable values on the stack.

Atila

On Friday, July 21, 2017 08:37:51 Atila Neves via Digitalmars-d wrote:
> On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis wrote:
> > On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl
> >
> > via Digitalmars-d wrote:
> >> On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis
> >>
> >> wrote:
> >> > The issue isn't the object being destroyed. It's what it refers to via its member variables. For instance, what if an object were to remove itself from a shared list when it's destroyed (e.g. because it's an observer in the observer pattern). The object has a reference to the list, but it doesn't own it.
> >>
> >> So, even a thread-local object that has references to a shared
> >> list
> >> has to handle those as shared, even in its non-shared
> >> destructor.
> >> I can't follow your argument.
> >
> > You can't just strip off shared. To do so defeats the purpose of shared. If you have something like
> >
> > struct S
> > {
> >
> >      shared List<Foo> _list;
> >
> >      ~this()
> >      {
> >
> >         ...
> >
> >      }
> >
> > }

Wow. I've been doing too much C++ lately apparently, since I used <>. :|

> This is fine. What dmd does now is strip shared off of the `this` pointer, not the member variables. There's only a problem if the sharedness of the member variable(s) depends on sharedness of the enclosing object.

What happens with something like

struct S
{
    Foo* _foo;

    ~this() {...}
}

shared S s;

Inside the destructor, is what _foo points to still treated as shared: shared(Foo)*? i.e. is the outer layer of shared the only layer being made thread-local - like what would supposedly happen with synchronized classes? If so, then that largely solves the problem. The only issue is pointers to the member variables, which would not be @safe but would still technically be possible, in which case something outside the struct could still reference the member variables from another thread. But given that that's going to blow up in your face soon thereafter anyway, since the object is being destroyed (and thus you screwed up making sure that your @system code was @safe), that's probably fine.

However, if _foo is treated as Foo* instead of shared(Foo)* in the destructor, then there definitely is a problem. The fact that when the member variable is explicitly shared, it continues to be treated as shared definitely reduces the problem, but it doesn't fully close the hole. The parts of the member variables not directly in the object still need to be treated as shared, because they aren't necessarily owned or controlled by the object and could legally and @safely be manipulated from other threads even while the destructor is running. So, are they still treated as shared, or are they treated as fully thread-local? I would have thought that they'd still be treated as thread-local given that the shared part is then only known to the variable that was marked as shared and not the destructor itself, since it's the same destructor for thread-local and shared objects.

And if the destructor treats the member variables as completely thread-local (rather than just the outer layer as thread local) even when the object itself was shared, then I don't think that this is a viable solution. It would either need to be made illegal to make an object shared if it has indirections and a destructor, or it needs to have a shared destructor.

- Jonathan M Davis

On Friday, 21 July 2017 at 22:02:41 UTC, Jonathan M Davis wrote:
> On Friday, July 21, 2017 08:37:51 Atila Neves via Digitalmars-d wrote:
>> On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis wrote:
>> > On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl
>> >
>> > via Digitalmars-d wrote:
>> >> On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis
>> >>
>> >> wrote:
>> >> > The issue isn't the object being destroyed. It's what it refers to via its member variables. For instance, what if an object were to remove itself from a shared list when it's destroyed (e.g. because it's an observer in the observer pattern). The object has a reference to the list, but it doesn't own it.
>> >>
>> >> So, even a thread-local object that has references to a shared
>> >> list
>> >> has to handle those as shared, even in its non-shared
>> >> destructor.
>> >> I can't follow your argument.
>> >
>> > You can't just strip off shared. To do so defeats the purpose of shared. If you have something like
>> >
>> > struct S
>> > {
>> >
>> >      shared List<Foo> _list;
>> >
>> >      ~this()
>> >      {
>> >
>> >         ...
>> >
>> >      }
>> >
>> > }
>
> Wow. I've been doing too much C++ lately apparently, since I used <>. :|
>

I noticed, but I wasn't going to say anything ;)

>> This is fine. What dmd does now is strip shared off of the `this` pointer, not the member variables. There's only a problem if the sharedness of the member variable(s) depends on sharedness of the enclosing object.
>
> What happens with something like
>
> struct S
> {
>     Foo* _foo;
>
>     ~this() {...}
> }
>
> shared S s;
>
> Inside the destructor, is what _foo points to still treated as shared: shared(Foo)*?

No. This is what I meant by the sharedness depening on the enclosing object. However, there's a workaround:

struct Foo { }


struct S {

    Foo* _foo;
    bool _isShared;

    this(this T, U)(U foo) if(is(T == shared) && is(U == shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
        static if(is(T == shared)) _isShared = true;
        _foo = foo;
    }

    ~this() {
        import std.stdio: writeln;
        _isShared ? writeln("shared dtor") : writeln("non-shared dtor");
    }
}

void main() {
    auto f = Foo();
    auto sf = shared Foo();
    auto s = S(&f);
    auto ss = shared S(&sf);
}


It's annoying to use that bool up memory-wise, but I assume it's not a big deal for most applications.

In any case, that example wouldn't have worked anyway before my change to dmd - even creating the S struct would've been a compiler error.

Atila

On Monday, July 24, 2017 2:30:01 PM MDT Atila Neves via Digitalmars-d wrote:
> >> This is fine. What dmd does now is strip shared off of the `this` pointer, not the member variables. There's only a problem if the sharedness of the member variable(s) depends on sharedness of the enclosing object.
> >
> > What happens with something like
> >
> > struct S
> > {
> >
> >     Foo* _foo;
> >
> >     ~this() {...}
> >
> > }
> >
> > shared S s;
> >
> > Inside the destructor, is what _foo points to still treated as
> > shared: shared(Foo)*?
>
> No. This is what I meant by the sharedness depening on the enclosing object. However, there's a workaround:
>
> struct Foo { }
>
>
> struct S {
>
>      Foo* _foo;
>      bool _isShared;
>
>      this(this T, U)(U foo) if(is(T == shared) && is(U ==
> shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
>          static if(is(T == shared)) _isShared = true;
>          _foo = foo;
>      }
>
>      ~this() {
>          import std.stdio: writeln;
>          _isShared ? writeln("shared dtor") : writeln("non-shared
> dtor");
>      }
> }
>
> void main() {
>      auto f = Foo();
>      auto sf = shared Foo();
>      auto s = S(&f);
>      auto ss = shared S(&sf);
> }
>
>
> It's annoying to use that bool up memory-wise, but I assume it's not a big deal for most applications.
>
> In any case, that example wouldn't have worked anyway before my change to dmd - even creating the S struct would've been a compiler error.

The problem with this is that this means that shared is not being properly enforced by the compiler. Your workaround is a way for the programmer to figure out if the object is shared and do something differently based on that, but for the compiler to do what it's supposed to be doing with shared (e.g. prevent non-atomic operations), any indirections in the member variables must continue to be typed as shared inside the destructor, and that's clearly not happening right now, which is a serious problem IMHO. The situation may be better thanks to your changes in that some stuff is now possible that should be possible and was not before, but it's not completely sound as far as the type system goes, and we really should be fixing it so that shared is properly enforced rather than just blindly stripped off.

- Jonathan M Davis