Given the pain of shared usage with std and pretty much every library in existence, I cowboyed the server without this qualifier. One of the mechanisms required atomic class reference compare-and-set, and the class reference is not shared, because it would otherwise require, like, 30 or 40 casts to non-shared in other places co compile. I was then mortified to learn that core.atomic operate on shared references\pointers\values, and that I had to do stuff like:
```
cas(cast(shared(C)*) &unshared_c, cast(shared C) unshared_c, cast(shared C) unshared_c);
atomicStore(*(cast(shared(C)*) &unshared_c), cast(shared(C)) unshared_c);
```

Does shared impose some alignment constraints in the backend, that are needed for core.atomic? Do these functions really need to work only with shared?

Side question: how hard is it for a semantic analysis to implicitly remove all shared qualifiers from all symbols referenced inside "synchronized" block (including "this" pointer). It would solve about 90% of my gripes with shared, I believe.

On Wednesday, July 04, 2018 09:25:27 Boris-Barboris via Digitalmars-d wrote:
> Given the pain of shared usage with std and pretty much every
> library in existence, I cowboyed the server without this
> qualifier. One of the mechanisms required atomic class reference
> compare-and-set, and the class reference is not shared, because
> it would otherwise require, like, 30 or 40 casts to non-shared in
> other places co compile. I was then mortified to learn that
> core.atomic operate on shared references\pointers\values, and
> that I had to do stuff like:
> ```
> cas(cast(shared(C)*) &unshared_c, cast(shared C) unshared_c,
> cast(shared C) unshared_c);
> atomicStore(*(cast(shared(C)*) &unshared_c), cast(shared(C))
> unshared_c);
> ```
>
> Does shared impose some alignment constraints in the backend, that are needed for core.atomic? Do these functions really need to work only with shared?

It wouldn't make any sense for atomics to operate on anything other than shared data. If the data isn't shared, then it doesn't need atomics.

There are some aspects of shared that still need to be fixed up (and Walter and Andrei have been discussing it), but the basic idea with shared is that you shouldn't be able to operate on it directly unless the compiler can guarantee that what you're doing is thread-safe. There's some debate as to whether that should involve the compiler inserting stuff for you to make stuff thread-safe or mean that you can't do much of anything to shared unless you protect the data and cast it away (Andrei tends to prefer the former, whereas Walter prefers the latter, and it sounds like Andrei is coming around to Walter's way of thnking, but we'll see). Most of the way that shared currently works, it goes with not allowing stuff and does not insert anything to make anything thread-safe, but that isn't fully implemented, because some stuff like copying shared data still works even though it's not thread-safe.

At this point, to operate on anything that's shared, either means using atomics or protecting the data with a mutex (be that with a synchronized block / function or a mutex object) and temporarily casting away shared while operating on the data. Afterwards, the mutex is released, and at that point, there should just be only shared references to the data. About the only time that operating directly on a shared object then makes sense is when it manages the atomics or mutex and associated cast internally.

Of course, the fact that you can't operate directly on shared can get annoying, but it's a side effect of the fact that non-shared objects are supposed to be guaranteed to be thread-local, and it prevents the programmer from accidentally operating on shared data in a manner that isn't thread-safe. So, by and large, it's preventing bugs - though we certainly need to lock it down better and preferably find some ways to make it more user-friendly where we can (synchronized classes are supposed to help with that but have yet to be finished).

Unfortunately, we have done a poor job of messaging how shared is supposed to be used, and many expect to be able to operate on shared data like normal data like you'd do in languages like C++ or Java, and so they get frustrated fast. The atomics and mutexes are exactly what you'd be doing in those languages. It's just that they don't protect you against accidentally operating on shared data and thus don't have shared in the type system and don't require casts. If we can finish locking down shared in the spec and implementation and then properly message how to use it, I expect that that will fix many of the problems (though some aspects of it are bound to always be annoying).

Many seem to just use __gshared as a way out, but it's really only intended for interacting with C global variables (and even then, only very carefully - especially if it's an aggregate type), and it results in shared data being treated as thread-local by the compiler, which can result in subtle, nasty bugs.

> Side question: how hard is it for a semantic analysis to implicitly remove all shared qualifiers from all symbols referenced inside "synchronized" block (including "this" pointer). It would solve about 90% of my gripes with shared, I believe.

It's a very difficult problem. Synchronized classes were proposed in an attempt to solve it, but even if they were fully implemented, they'd only help partially, because they can only strip off the outermost layer of shared. In order for the compiler to cast away shared for you, it has to be able to guarantee that there are no unprotected references to that data, and because D doesn't have any kind of ownership system in the language, we don't have a clean way to do it.

Given D's lack of ownership model, any solution that might work would almost certainly have to have some way to associate a mutex with an object (or set of objects) and prevent all operations on the object which could allow any unprotected references to the data to escape. That's what synchronized classes try to do, but they'd have to somehow be locked down even further in order to guarantee that anything other than that the data directly in the object is protected. As soon as you can do something like return an unprotected pointer or reference from a member function, then the compiler can't guarantee that the data is protected and thus can't implicitly remove that layer of shared.

If anyone can come up with a clean and sane way to fully protect an object with an associated mutex, then we could do better with implicitly removing shared, but I don't know of any way that it could be done without either locking down what you can do with such an object so thoroughly that it borders on unusable or adding ownership semantics to the language, and ownership semantics would complicate the language so much that I'd be shocked if we ever had them. So, right now, it's looking like synchronized classes are probably the best that we're going to get, though maybe someone smart will come up with something.

- Jonathan M Davis

On Wednesday, 4 July 2018 at 10:47:12 UTC, Jonathan M Davis wrote:
> At this point, to operate on anything that's shared, either means using atomics or protecting the data with a mutex (be that with a synchronized block / function or a mutex object) and temporarily casting away shared while operating on the data. Afterwards, the mutex is released, and at that point, there should just be only shared references to the data. About the only time that operating directly on a shared object then makes sense is when it manages the atomics or mutex and associated cast internally.

Good overview. Some syntactic sugar should ease the casting pain, because it's the casting that I dislike the most, especially wheh shared has to be casted away from multiple unrelated entities guarded by one mutex chain or some high-level synchronization rule, unknown to the compiler. If there would be a construct that does not involve additional line of code and explicit type specification (or some typeof + Unqual shenanigans), I would probably have no problem with this approach. Currently, the casting seems to be the correct way, and it's far from user-friendly. Maybe such thing should not be a cast at all, and some kind of built-in property, idk.

On Wednesday, 4 July 2018 at 10:47:12 UTC, Jonathan M Davis wrote:
> At this point, to operate on anything that's shared, either means using atomics or protecting the data with a mutex (be that with a synchronized block / function or a mutex object) and temporarily casting away shared while operating on the data. Afterwards, the mutex is released, and at that point, there should just be only shared references to the data. About the only time that operating directly on a shared object then makes sense is when it manages the atomics or mutex and associated cast internally.
>
> ...
> It's a very difficult problem. Synchronized classes were proposed in an attempt to solve it, but even if they were fully implemented, they'd only help partially, because they can only strip off the outermost layer of shared. In order for the compiler to cast away shared for you, it has to be able to guarantee that there are no unprotected references to that data, and because D doesn't have any kind of ownership system in the language, we don't have a clean way to do it.
>

Once we have DIP1000, can accessing shared class members in a synchronized class (optionally? implicitly?) result in scoped rvalues?

On Thursday, 5 July 2018 at 11:31:15 UTC, FeepingCreature wrote:
> On Wednesday, 4 July 2018 at 10:47:12 UTC, Jonathan M Davis wrote:
>>[...]
>
> Once we have DIP1000, can accessing shared class members in a synchronized class (optionally? implicitly?) result in scoped rvalues?

We already have DIP1000. I used it to avoid casting and bringing your own mutex here:

https://github.com/atilaneves/fearless

Atila

On Wednesday, 4 July 2018 at 10:47:12 UTC, Jonathan M Davis wrote:
> [cut]
>
> - Jonathan M Davis

I think I've just read a similar explanation about shared written by you some months ago. It seems that a lot of users doesn't understand what shared really is.

Probably a pinned article should be posted on dlang homepage :)

Andrea

On Thursday, July 05, 2018 11:31:15 FeepingCreature via Digitalmars-d wrote:
> On Wednesday, 4 July 2018 at 10:47:12 UTC, Jonathan M Davis wrote:
> > At this point, to operate on anything that's shared, either means using atomics or protecting the data with a mutex (be that with a synchronized block / function or a mutex object) and temporarily casting away shared while operating on the data. Afterwards, the mutex is released, and at that point, there should just be only shared references to the data. About the only time that operating directly on a shared object then makes sense is when it manages the atomics or mutex and associated cast internally.
> >
> > ...
> > It's a very difficult problem. Synchronized classes were
> > proposed in an attempt to solve it, but even if they were fully
> > implemented, they'd only help partially, because they can only
> > strip off the outermost layer of shared. In order for the
> > compiler to cast away shared for you, it has to be able to
> > guarantee that there are no unprotected references to that
> > data, and because D doesn't have any kind of ownership system
> > in the language, we don't have a clean way to do it.
>
> Once we have DIP1000, can accessing shared class members in a synchronized class (optionally? implicitly?) result in scoped rvalues?

I'm not quite sure what you're asking exactly, but if you're wondering if scope can help indicate that a reference hasn't escaped from a synchronized class, then I don't think so. e.g. something like

mySynchObj.foo().func();

or

scope foo = mySynchObj.foo();
foo.bar();

would already be a problem, even though scope prevents the reference from being kept around, because another thread could come in and access foo or what it refers to at the same time. The mutex would have been released as soon as the member function returned. Maybe some changes could be made to how synchronized works so that the mutex would be left locked until the expression terminated, and then if all you're allowed to do is call a member function on the scope reference that's returned, then maybe it could work - but even then, you have the problem of a reference escaping via that member function call, and even a scoped reference is unacceptable, because then it could escape the mutex's lock, even if it didn't last long - and that's assuming that the mutex was made to stay locked for the duration of the expression rather than for the duration of the member function call. scope is designed around making sure that the lifetime of a reference, ref, pointer, or dynamic array is not longer than the lifetime of the object it refers to and not around the lifetime of a function call or mutex lock.

It has to be guaranteed that any time the data is accessed, the mutex protecting it is locked. If it could be guaranteed that the data never escapes - even temporarily - from any member function call (i.e. they never return any references to any member variables or anything they refer to, and all of the member functions are pure), and it's guaranteed that the data in the member variables was all created inside the class (so any data that was passed to a member function - including the constructor - and then assigned to or passed to a member variable is a value type or it was a reference type that was duped - though we don't really have a way to guarantee at this point that data was duped), then I think that it would be possible to strip away the other layers of shared and call free functions inside member functions. But that would be a pretty big restriction as well as probably a pain to code correctly in the compiler.

I don't know. Some improvements may be possible there, but it's very tricky. You have to be able to guarantee that no data that ever has shared implicitly removed from it is ever accessed while the mutex isn't locked, and the more you try to allow, the harder it is to plug the holes. Much as I agree that the casting sucks, I'm increasingly of the opinion that the extra complications required to implicitly remove shared from even part of an object quickly become too much to be worth it. Ultimately, I think that the only real downside to the casting is that it requires that the programmer get it right, whereas any time we can automate it, the compiler guarantees it. Having the compiler guarantee it is very desirable, but the extra restrictions required to make that guarantee can't be too restrictive, or it quickly becomes saner to just do the cast even if it makes you verify it yourself.

- Jonathan M Davis