Hi all,

I know that many (most?) D users don't like using classes or old, manually controlled, concurrency using "shared" & co., but still, since they *are* in the language, I think they should at least be usable.

After having had my share (no pun intended) of problems using shared, I've finally settled for the following:

* Encapsulate all the shared stuff in classes (personal preference, easier to pass around).
* When possible, try to use "shared synchronized" classes, because even if there are potential losses of performance, the simplicity is often worth it. This mean that the classed is declared:

```
shared synchronized class A { }
```

and now, the important point:

* Make all _private non-reference fields_ of shared, synchronized classes __gshared.

AIUI the access of those fields is already guaranteed to be safe by the fact that *all* the methods of the class are already synchronized on "this", and nothing else can access them.

Of course, assuming you then don't escape references to them, but I think that would be a *really* silly thing to do, at least in the most common case... why on earth are they then private in the first place?.

Now, the question is, would it make sense to have the compiler do this for me in a transparent way? i.e. the compiler would automatically store private fields of shared *and* synchronized classes in the global storage.

Bonus points if it detects and forbids escaping references to them, although it could also be enough to warn the user.

This way I think there would an easy and sane way of using shared, because many of its worst quirks (for one, try using a struct like SysTime that overrides OpAssign, but not for shared objects, as a field) would be transparently dealt with.

A.

On Thursday, 13 September 2018 at 13:53:49 UTC, Arafel wrote:
> * Make all _private non-reference fields_ of shared, synchronized classes __gshared.

so __gshared implies static. Are you sure that's what you want?

On 09/13/2018 04:27 PM, Adam D. Ruppe wrote:
> On Thursday, 13 September 2018 at 13:53:49 UTC, Arafel wrote:
>> * Make all _private non-reference fields_ of shared, synchronized classes __gshared.
> 
> so __gshared implies static. Are you sure that's what you want?

Indeed it isn't! Why must __gshared be static?? (BTW, thanks a lot, you have just saved me a lot of debugging!!).

Then, how on earth are we supposed to have a struct like SysTime as a field in a shared class? Other than the "fun" of having a shared *pointer* to such a struct that then you can cast as non-shared as needed...

But let's say that this solution seems quite...... sub-optimal... at so many levels...

On Thursday, 13 September 2018 at 14:43:51 UTC, Arafel wrote:
>  Why must __gshared be static?? (BTW, thanks a lot, you have just saved me a lot of debugging!!).

The memory location differences of shared doesn't apply to class members. All members are stored with the instance, and shared only changes the type. (Contrast to global variables, where shared changes where they are stored - the default is to put them in thread-local storage, and shared moves it back out of that.)

Class static variables btw follow the same TLS rules. A static member is really the same as a global thing, just in a different namespace.


Now, the rule of __gshared is it puts it in that global memory storage using the unshared type. Unshared type you like here, but also, since normally, class members are stored in the object, changing the memory storage to the global shared area means it is no longer with the object... thus it becomes static.

> Then, how on earth are we supposed to have a struct like SysTime as a field in a shared class? Other than the "fun" of having a shared *pointer* to such a struct that then you can cast as non-shared as needed...

What you want is an unshared type without changing the memory layout. There's no syntax for this at declaration, but there is one at usage point: you can cast away attributes on an lvalue:

shared class Foo {
        void update() {
                // the cast below strips it of all attributes, including shared,
                // allowing the assignment to succeed
                cast() s = Clock.currTime;
        }
        SysTime s;
}



Using the private field with a public/protected/whatever accessor method, you can encapsulate this assignment a little and make it more sane to the outside world.

struct Unshared(T)
{
    private T value;
    T get() shared { return cast(T)value; }
    alias get this;
    void opAssign(T v) shared { value=cast(shared)v; }
}

shared synchronized class A
{
    private Unshared!(int[]) a;
    int[] f()
    {
        return a;
    }
}

On 09/13/2018 05:16 PM, Kagamin wrote:
> struct Unshared(T)
> {
>      private T value;
>      T get() shared { return cast(T)value; }
>      alias get this;
>      void opAssign(T v) shared { value=cast(shared)v; }
> }
> 
> shared synchronized class A
> {
>      private Unshared!(int[]) a;
>      int[] f()
>      {
>          return a;
>      }
> }
> 

Doesn't work:

```
import std.datetime.systime;

struct Unshared(T)
{
    private T value;
    T get() shared { return cast(T)value; }
    alias get this;
    void opAssign(T v) shared { value=cast(shared)v; }
}

shared synchronized class A {
    private Unshared!SysTime t;

    this() {
        t = Clock.currTime;
    }
}

void main() {
    shared A a = new shared A;
}
```

Gives you:

onlineapp.d(6): Error: non-shared const method std.datetime.systime.SysTime.opCast!(SysTime).opCast is not callable using a shared mutable object
onlineapp.d(6):        Consider adding shared to std.datetime.systime.SysTime.opCast!(SysTime).opCast
onlineapp.d(8): Error: template std.datetime.systime.SysTime.opAssign cannot deduce function from argument types !()(shared(SysTime)) shared, candidates are:
/dlang/dmd/linux/bin64/../../src/phobos/std/datetime/systime.d(612):    std.datetime.systime.SysTime.opAssign()(auto ref const(SysTime) rhs)
onlineapp.d(12): Error: template instance `onlineapp.Unshared!(SysTime)` error instantiating

On 09/13/2018 05:16 PM, Kagamin wrote:
> struct Unshared(T)
> {
>      private T value;
>      T get() shared { return cast(T)value; }
>      alias get this;
>      void opAssign(T v) shared { value=cast(shared)v; }
> }
> 
> shared synchronized class A
> {
>      private Unshared!(int[]) a;
>      int[] f()
>      {
>          return a;
>      }
> }
> 

My current attempt, still work in progress:


```
import std.stdio;
import std.datetime.systime;

shared struct GShared(T) {
	ubyte[T.sizeof] payload;

    this(T t) {
		*(cast(T*) &payload) = t;
	}
    this(shared T t) {
		*(cast(T*) &payload) = cast() t;
	}
	void opAssign(T t) {
 		*(cast(T*) &payload) = t;
	}
	void opAssign(shared T t) {
		*(cast(T*) &payload) = cast() t;
	}
    ref T data() {
        return *(cast(T*) &payload);
    }
    alias data this;
}

shared synchronized class A {
    this() {
	    t = Clock.currTime;
    }

    void printIt() {
        writeln(t);
    }

    private:
    GShared!SysTime t;
}

void main() {
    shared A a = new shared A;
    a.printIt;
}
```

On 09/13/2018 05:11 PM, Adam D. Ruppe wrote:
> On Thursday, 13 September 2018 at 14:43:51 UTC, Arafel wrote:
>>  Why must __gshared be static?? (BTW, thanks a lot, you have just saved me a lot of debugging!!).
> 
> The memory location differences of shared doesn't apply to class members. All members are stored with the instance, and shared only changes the type. (Contrast to global variables, where shared changes where they are stored - the default is to put them in thread-local storage, and shared moves it back out of that.)
> 
> Class static variables btw follow the same TLS rules. A static member is really the same as a global thing, just in a different namespace.
> 
> 
> Now, the rule of __gshared is it puts it in that global memory storage using the unshared type. Unshared type you like here, but also, since normally, class members are stored in the object, changing the memory storage to the global shared area means it is no longer with the object... thus it becomes static.
> 
>> Then, how on earth are we supposed to have a struct like SysTime as a field in a shared class? Other than the "fun" of having a shared *pointer* to such a struct that then you can cast as non-shared as needed...
> 
> What you want is an unshared type without changing the memory layout. There's no syntax for this at declaration, but there is one at usage point: you can cast away attributes on an lvalue:
> 
> shared class Foo {
>          void update() {
>                  // the cast below strips it of all attributes, including shared,
>                  // allowing the assignment to succeed
>                  cast() s = Clock.currTime;
>          }
>          SysTime s;
> }
> 
> 
> 
> Using the private field with a public/protected/whatever accessor method, you can encapsulate this assignment a little and make it more sane to the outside world.

Thanks a lot!! I remember having tried casting shared away, and ending up with a duplicate, but I have just tried it now and indeed it seems to work, will have to try with more complex use cases (comparing, adding dates and intervals, etc.), but it looks promising.

The problem might have been that I think I tried:

shared SysTime s_;
SysTime s = cast () s_; // Now I've got a duplicate! Ugh!

Because that works with classes... but (in hindsight) obviously not with value types.

I still think that it would be useful:

1) Allow __gshared for non-static fields to have this meaning, it would make it much more intuitive. A library solution is perhaps possible, but cumbersome.

2) Make it (sometimes) automatic as the original proposal.

Of course 1) is the most important part.

A.

On Thursday, September 13, 2018 7:53:49 AM MDT Arafel via Digitalmars-d wrote:
> Hi all,
>
> I know that many (most?) D users don't like using classes or old, manually controlled, concurrency using "shared" & co., but still, since they *are* in the language, I think they should at least be usable.
>
> After having had my share (no pun intended) of problems using shared, I've finally settled for the following:
>
> * Encapsulate all the shared stuff in classes (personal preference,
> easier to pass around).
> * When possible, try to use "shared synchronized" classes, because even
> if there are potential losses of performance, the simplicity is often
> worth it. This mean that the classed is declared:
>
> ```
> shared synchronized class A { }
> ```
>
> and now, the important point:
>
> * Make all _private non-reference fields_ of shared, synchronized classes __gshared.
>
> AIUI the access of those fields is already guaranteed to be safe by the fact that *all* the methods of the class are already synchronized on "this", and nothing else can access them.
>
> Of course, assuming you then don't escape references to them, but I think that would be a *really* silly thing to do, at least in the most common case... why on earth are they then private in the first place?.
>
> Now, the question is, would it make sense to have the compiler do this for me in a transparent way? i.e. the compiler would automatically store private fields of shared *and* synchronized classes in the global storage.
>
> Bonus points if it detects and forbids escaping references to them, although it could also be enough to warn the user.

Have you read the concurrency chapter in The D Programming Language by Andrei? It sounds like you're trying to describe something vere similar to the synchronized classes from TDPL (which have never been fully implemented in the language). They would make it so that you had a class with shared members but where the outer layer of shared was stripped away inside member functions, because the compiler is able to guarantee that they don't escape (though it can only guarantee that for the outer layer). Every member function is synchronized and no direct access to the member variables outside of the class (even in the same module) is allowed. It would make shared easier to use in those cases where it makes sense to wrapped everything protected by a mutex in a class (though since it can only safely strip away the outer layer of shared, it's more limited than would be nice, and there are plenty of cases where it doesn't make sense to stuff something in a class just to use it as shared).

> This way I think there would an easy and sane way of using shared, because many of its worst quirks (for one, try using a struct like SysTime that overrides OpAssign, but not for shared objects, as a field) would be transparently dealt with.

The fact that most operations are not allowed with shared is _on purpose_. If anything, too many operations are currently legal. What's really supposed to be happening is that every single operation on a shared object is either guaranteed to be thread-safe, or it's illegal. And if it's illegal, that means that you either need to use atomics to do an operation (since they're thread-safe), or you need to protect the shared object with a mutex and temporarily cast away shared while the mutex is locked so that you can actually do something with the object - and then make sure that no thread-local references exist when the mutex is released.

Something like copying a shared object shouldn't even be legal in general. An object that defines opAssign prevents it now, but the fact that it's legal on any type where copying is not guaranteed to be thread-safe is a bug. It's one of those details of shared that has never been fully fleshed out like it should be. Walter and Andrei have been discussing finishing shared, but it hasn't been a high enough priority for it actually get fully sorted out yet. Once it is, unless you're dealing with a type that isn't guaranteed to be thread-safe when copying it, it won't be legal copy it without first casting away shared. Anything less than that would violate what shared is supposed to do.

What you should be thinking when dealing with any shared object and whether a particular operation should be allowed is whether that operation is guaranteed to be thread-safe. If the compiler can't guarantee that the operation is thread-safe, then it's not supposed to be legal. The main area that Walter and Andrei haven't agreed upon yet is how much the compiler can or should do to ensure that something is thread-safe rather than just making an operation illegal (e.g. whether memory barriers should be involved). So, _maybe_ some operations will end up as legal thanks to the compiler adding extra code to do something to ensure thread-safety, but in most situations, it's just going to be illegal.

So, ultimately, every type is either going to need to be designed such that it simply does not work as shared, or it manages the thread-safety stuff for you. If the object is not designed to be used as shared, then that means that if you want to, you need to protect it with a mutex (be that with synchronized or directly using mutexes) and cast away shared correctly when the object is protected by the mutex. It's annoying, but it prevents thread-safety bugs, and it allows the compiler (and the programmer) to treat the rest of the program as thread-local. On the other hand, if the type is designed to be used as shared (so it actually has shared member functions), then that means that the type itself is going to need to deal with all of the thread-safety stuff internally (be that by using mutexes and casting or using atomics or whatever).

If we're going to find ways to make shared require less manual work, it means finding a way to protect a shared object (or group of shared objects) with a mutex in a way that is able to guarantee that when you operate on the data, it's protected by that mutex and that no reference to that data has escaped. TDPL's synchronized classes are one attempt to do that, but the requirement that no references escape (so that shared can safely be cast away) makes it so that only the outer layer of shared can be cast away, and it's extremely difficult to do better than that with having holes such that it isn't actually guaranteed to be thread-safe when shared is cast away. Maybe someone will come up with something that will work, but I wouldn't bet on it. Either way, I don't see how any solution is going to be acceptable which does not actually guarantee thread-safety, because it would be violating the guarantees of shared otherwise. A programmer can choose to cast away shared in an unsafe manner (or use __gshared) and rely on their ability to ensure that the code is thread-safe rather than letting shared do its job, but that's not the sort of thing that we're going to do with a language construct, and given that the compiler assumes that anything that isn't shared or immutable is thread-local, it's very much a risky thing to do.

As for __gshared, it's intended specifically for C globals, and using it for anything else is just begging for bugs. Because the compiler assumes that anything which is not marked as shared or immutable is thread-local, having such an object actually be able to be mutated by another thread risks subtle bugs of the sort that shared was supposed to prevent in the first place. Unfortunately, due to some of the difficulties in using shared and some of the misunderstandings about it, a number of folks have just used __gshared instead of shared, but once you do that, you're risking subtle bugs, because that's not at all what __gshared is intended for. If you're using __gshared for anything other than a C global, it's arguably a bug. Certainly, it's a risky proposition.

- Jonathan M Davis

On 09/13/2018 09:49 PM, Jonathan M Davis wrote:
> 
> Have you read the concurrency chapter in The D Programming Language by
> Andrei? It sounds like you're trying to describe something vere similar to
> the synchronized classes from TDPL (which have never been fully implemented
> in the language). They would make it so that you had a class with shared
> members but where the outer layer of shared was stripped away inside member
> functions, because the compiler is able to guarantee that they don't escape
> (though it can only guarantee that for the outer layer). Every member
> function is synchronized and no direct access to the member variables
> outside of the class (even in the same module) is allowed. It would make
> shared easier to use in those cases where it makes sense to wrapped
> everything protected by a mutex in a class (though since it can only safely
> strip away the outer layer of shared, it's more limited than would be nice,
> and there are plenty of cases where it doesn't make sense to stuff something
> in a class just to use it as shared).
> 

I hadn't read the book, but that's indeed the gist of what I'm proposing. I think it could be enough to restrict it to value types, where it's easier to assume (and even check) that there are no external references.

> 
> [snip]
> 
> If we're going to find ways to make shared require less manual work, it
> means finding a way to protect a shared object (or group of shared objects)
> with a mutex in a way that is able to guarantee that when you operate on the
> data, it's protected by that mutex and that no reference to that data has
> escaped. TDPL's synchronized classes are one attempt to do that, but the
> requirement that no references escape (so that shared can safely be cast
> away) makes it so that only the outer layer of shared can be cast away, and
> it's extremely difficult to do better than that with having holes such that
> it isn't actually guaranteed to be thread-safe when shared is cast away.
> Maybe someone will come up with something that will work, but I wouldn't bet
> on it. Either way, I don't see how any solution is going to be acceptable
> which does not actually guarantee thread-safety, because it would be
> violating the guarantees of shared otherwise. A programmer can choose to
> cast away shared in an unsafe manner (or use __gshared) and rely on their
> ability to ensure that the code is thread-safe rather than letting shared do
> its job, but that's not the sort of thing that we're going to do with a
> language construct, and given that the compiler assumes that anything that
> isn't shared or immutable is thread-local, it's very much a risky thing to
> do.
> 

I completely agree with this argument, however please note that there must be a sensible way to work with shared, otherwise we enter in the "the perfect is the enemy of the good" area.

For reference types it's somehow workable, because you can just cast away and store it in a new variable:

```
class A {
	this() { }
}

shared synchronized class B {
	this(A a) {
		a_ = cast (shared) new A; // no shared this()
	}
	void foo() {
		A a = cast () a_;
		// Work with it
	}
	private:
	A a_;
}
```

It's still somewhat cumbersome, specially if you have many such members, but still doable.

However, this is not possible for value types, and it makes it nigh on impossible to work with them in a sensible way. You have either to use pointers, or cast away every type you want to use it. None of them are what I would call "practical".

While not the biggest problem (see the later point), I still think that synchronized classes are a good compromise, specially with the restriction of only applying to full value types (no internal references allowed). Of course it is still perhaps possible to bypass that mechanism, but so is the case with many other ones (assumeUnique?).

If it's hard enough to do by mistake, it can be assumed that the people messing with it should know what they are doing.

Finally, you suggest using __gshared, and I'm not sure you're not having the same misunderstanding I had: __gshared implies "static", so it's not a valid solution for class fields in most cases.

> As for __gshared, it's intended specifically for C globals, and using it for
> anything else is just begging for bugs. Because the compiler assumes that
> anything which is not marked as shared or immutable is thread-local, having
> such an object actually be able to be mutated by another thread risks subtle
> bugs of the sort that shared was supposed to prevent in the first place.
> Unfortunately, due to some of the difficulties in using shared and some of
> the misunderstandings about it, a number of folks have just used __gshared
> instead of shared, but once you do that, you're risking subtle bugs, because
> that's not at all what __gshared is intended for. If you're using __gshared
> for anything other than a C global, it's arguably a bug. Certainly, it's a
> risky proposition.
> 

As I said, the current semantics of __gshared doesn't allow it to be a "drop-in" replacement of "shared". I also agree that it's not what it was meant for, and that changing that right now would risk breaking a lot of code.

However, I think that there should be *some* way in the language itself to express that without having to cast all over the place: access this member of the shared class as if it were local, I'll take care of controlling the access to it.

You can use a wrapper type, and that's what I'm trying to do right now, but I'm pretty sure there will be a ton of corner cases and interactions that will make it really hard to work reliably in a generic way.

Then, implementing a kind of "synchronized classes" that would automate this when possible would of course be a further and welcome improvement.

A.