1. Make synchronized an attribute of fields, classes and methods
2. synchronized methods are synchronized(this) wrapping inconditions, outconditions, invariant checks and the main body. They are thus actually safe, as opposed to the status quo where invariants have to be synchronized separately and are thus Worse Than Useless™.
3. synchronized classes consist of synchronized fields and synchronized public methods
3.1. Only synchronized subclasses may inherit from synchronized classes, and vice versa.
4. immutable data is implicitly synchronized. Local variables are implicitly synchronized.
5. Synchronized methods must not allow mutable references to escape, unless they are to synchronized objects.
6. in @safe, only synchronized methods may access synchronized fields.

Advantages:
* simple
* does something useful
Disadvantages:
* inability to generate giant forum debates (fingers crossed)

Addendum: synchronized methods must be pure.

No setting a global variable from two different threads for you!

On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature wrote:
> 1. Make synchronized an attribute of fields, classes and methods
> ...
>
> Advantages:
> * simple
> * does something useful
> Disadvantages:
> * inability to generate giant forum debates (fingers crossed)

:) What about structs? Or all the cases where you don't need 'synchronized'? You can write a lock-free queue with a few integers and an array.

On Tuesday, 23 October 2018 at 12:50:30 UTC, Stanislav Blinov wrote:
> On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature wrote:
>> 1. Make synchronized an attribute of fields, classes and methods
>> ...
>>
>> Advantages:
>> * simple
>> * does something useful
>> Disadvantages:
>> * inability to generate giant forum debates (fingers crossed)
>
> :) What about structs? Or all the cases where you don't need 'synchronized'? You can write a lock-free queue with a few integers and an array.

That's why I tied it to @safe, so you can always write synchronized @trusted code that needs manual thread safety. That said, clever threading solutions, atomics and the like are mostly out of scope for this proposal, which is primarily aimed at simple oop-based code.

ping

On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature wrote:
> Disadvantages:
> * inability to generate giant forum debates (fingers crossed)

I consider the point proven, but the downside of being unable to generate giant flamewars is that you eventually drop off the frontpage.

I really think the shared viewpoint on thread-owned data and how to move data from thread to thread is very alien to how thread safety is handled in practice. In practice, a lot of data is shared; I would argue a large fraction of *all* mutable data is shared, and access is protected using synchronization in the owning class. That is, objects, not threads, are the owners and overseers of mutable data.

Whatever solution for thread safety we arrive at should account for this.

ping

On Friday, October 26, 2018 7:21:50 AM MDT FeepingCreature via Digitalmars-d wrote:
> ping
>
> On Tuesday, 23 October 2018 at 08:56:39 UTC, FeepingCreature
>
> wrote:
> > Disadvantages:
> > * inability to generate giant forum debates (fingers crossed)
>
> I consider the point proven, but the downside of being unable to generate giant flamewars is that you eventually drop off the frontpage.
>
> I really think the shared viewpoint on thread-owned data and how to move data from thread to thread is very alien to how thread safety is handled in practice. In practice, a lot of data is shared; I would argue a large fraction of *all* mutable data is shared, and access is protected using synchronization in the owning class. That is, objects, not threads, are the owners and overseers of mutable data.
>
> Whatever solution for thread safety we arrive at should account for this.

Really, all shared is about is preventing common bugs related to multithreading by segregating data shared across threads and making operations which aren't guaranteed to be thread-safe illegal (the main benefit then being that you know that everything else then isn't shared across threads). The language doesn't do that entirely correctly at the moment (e.g. copying and assigning shared data is currently legal in spite of not being thread-safe) but ultimately, when you use shared, what you're doing is pretty much exactly what you'd do in C/C++ except that the compiler prevents many bugs with regards to data shared across threads, and because of the extra type information in place to enable the restrictions to give you that protection, you're often forced to cast away shared in @trusted code to then do what you would have done in C++. But all of the atomics and mutexes and whatnot are the same as what you'd be doing in C++. The basic design patterns are the same. It's the fact that you you're getting compiler errors when you do something that's not thread-safe that's different, and it's the fact that you have to cast away shared after locking a mutex that's different (since C++ doesn't have shared).

But aside from that, what you're doing is the same. The fact that the compiler prevents you from doing basic stuff while the variable is shared and requires casting (unless you use atomics) frequently causes folks to get mad at shared and misunderstand it, but that's really the compiler preventing you from shooting yourself in the foot. Once you understand what's going on, it's really just the same as C++ but with some extra casts.

So, shared is very low level ultimately. It's about how the data is stored and accessed, not about how higher level constructs are written. We can then build other, higher level mechanisms on top of that - such as std.concurrency - but at the language level, whether you're passing data across threads or having multiple threads access data on the same thread isn't really part of the design. The language is just concerned about the fact that the data is shared.

The only higher level threading mechanism that the language has is synchronized, and all that really is is a built-in mutex, which in conjunction whith synchronized classes (if they're ever fully implemented) would allow for the implicit casting away of the top level of shared inside of member functions, but that's really as far as it goes, and without some kind of ownership model in the language, it's _very_ hard to go much farther than that - either with implicitly removing shared in code or with having passing ownership across threads work without explicit casts.

Certainly, I think that any attempt to move shared away from being a low level construct indicating simply that the data is shared across threads is mis-guided. It's what other constructs for sharing data across threads are built upon. So, if someone wants to push for some new mechanism that improves sharing data across threads (rather than passing it across threads) that's built on top of shared, that's fine. If someone wants to push for some new mechanism that improves sharing data by passing it across threads, that's fine. But shared is a low level construct and should stay that way. It's the base for everything else.

And to be honest, I don't really expect it to be reasonable to add much in the way of higher level mechanisms for sharing data to D at the language level - not with it being a systems language. Most stuff would require restricting it further and/or introducing a serious ownership model to it, which we really don't want to do. Even synchronized classes are pretty questionable, because they can only safely remove the outer layer of shared, which is arguably better than nothing, but it doesn't get you very far. Without a full-blown ownership model, we're pretty much reduced to manual casting when mutexes or synchronized are used, and the language really isn't going to be much help. If someone can come up with some bright ideas, great. All the more power to them. Maybe we can improve the situation. But again, they should be built on top of shared as a low level mechanism, not try to change what shared is.

- Jonathan M Davis

On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis wrote:
> But shared is a low level construct and should stay that way. It's the base for everything else.
>

This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates.

Why not do something useful with synchronized instead?

> And to be honest, I don't really expect it to be reasonable to add much in the way of higher level mechanisms for sharing data to D at the language level - not with it being a systems language. Most stuff would require restricting it further and/or introducing a serious ownership model to it, which we really don't want to do. Even synchronized classes are pretty questionable, because they can only safely remove the outer layer of shared, which is arguably better than nothing, but it doesn't get you very far. Without a full-blown ownership model, we're pretty much reduced to manual casting when mutexes or synchronized are used, and the language really isn't going to be much help. If someone can come up with some bright ideas, great. All the more power to them. Maybe we can improve the situation. But again, they should be built on top of shared as a low level mechanism, not try to change what shared is.
>
> - Jonathan M Davis


Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?

On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature via Digitalmars-d wrote:
> On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis
>
> wrote:
> > But shared is a low level construct and should stay that way. It's the base for everything else.
>
> This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates.
>
> Why not do something useful with synchronized instead?

__gshared is intended for interacting with C globals and that's it. If it's used for anything else, it's not being used for its intended purpose. Certainly, using it for D objects is just begging for bugs. __gshared should _never_ be used for D objects. I know that some folks do, and sometimes folks get away with it, but it's just asking for trouble, and there's no guarantee that such code will continue to work, because it's violating compiler guarantees when it does so. __gshared is designed solely as a backdoor for dealing with C and that is it. On the whole, shared really is _not_ broken and _does_ work. Yes, it has some rough edges that need fixing, and not everything in core.sync has been updated to use it properly, necessitating more casts than should be necessary when using those constructs than should be necessary, but shared as a whole is sound and always has been. The biggest problem with it is that most folks don't understand how to use it properly.

> > And to be honest, I don't really expect it to be reasonable to add much in the way of higher level mechanisms for sharing data to D at the language level - not with it being a systems language. Most stuff would require restricting it further and/or introducing a serious ownership model to it, which we really don't want to do. Even synchronized classes are pretty questionable, because they can only safely remove the outer layer of shared, which is arguably better than nothing, but it doesn't get you very far. Without a full-blown ownership model, we're pretty much reduced to manual casting when mutexes or synchronized are used, and the language really isn't going to be much help. If someone can come up with some bright ideas, great. All the more power to them. Maybe we can improve the situation. But again, they should be built on top of shared as a low level mechanism, not try to change what shared is.
>
> Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?

Manual casting is required if you're actually using shared with mutexes or synchronized. If you're using __gshared, it's not, but that's only because you're lying to the compiler about whether the variables are shared are not, which means that you're circumventing the type system, and you're risking subtle bugs in your programs, because the guarantees that the type system is suppose make aren't actually in place.

Your synchronized field proposal is basically just a variation of synchronized classes with the extra caveat that somehow mutable references to the member variables can't escape, and for some reason, not everything in the class has to be synchronized. It's like you're trying to have synchronized classes without talking about shared. But you can't take shared out of the mix. Per the type system, anything that isn't shared is considered thread-local by the compiler, and trying to do anything like this without shared would be a serious violation of the type system and the compiler's guarantees.

Instead of talking about safely casting away the outer layer of shared inside the class, you're talking about preventing escaping. synchronized classes as described in TDPL already guaranteed that the outer layer can't escape, since it lives directly in the class. What they can't guarantee is that the rest can't escape, and the language doesn't provide any way for such a guarantee. scope (even with DIP 1000) isn't transitive, so it doesn't really do the trick. So, I don't know how we could guarantee that _nothing_ escapes, though if nothing escapes, then all levels of shared could be implicitly cast away instead of just the outer layer, which would be nice.

Also, even if we _could_ somehow prevent anything from escaping, I would point out that mutability has nothing to do with this (or at least constness doesn't), because reading has many of the same threading issues as writing. In order to avoid threading issues, the data must actually be immutable. So, allowing const references to escape would still violate thread-safety. Access to data must be synchronized unless it is immutable - whether we're talking about reading or writing is irrelevant.

synchronized classes work (or would work if fully implemented) by guaranteeing that a single level can't escape - the one level that it _can_ guarantee can't escape - and thus are able to properly synchronize that level. If scope were transitive, then maybe we could do better, but since it isn't (and Walter insists that it can't reasonbly be transitive), I'm not sure that it's really reasonable to prevent escaping transitively.

In any case, your proposal is just a variation of synchronized classes. So, while it may state it in a new way, the basic concept really isn't anything new. And my point still stands that anything that we do with regards to thread safety must be built on top of shared, because that is the building block of sharing data across threads in D. Whether you like the specifics of shared or not, the idea that the compiler can rely on the assumption that data is _not_ shared across threads unless it's marked as such is a key feature of D. And attempting to use __gshared to get around it is just shooting yourself in the foot.

- Jonathan M Davis

On Wednesday, 31 October 2018 at 07:57:59 UTC, Jonathan M Davis wrote:
> On Wednesday, October 31, 2018 1:00:32 AM MDT FeepingCreature via Digitalmars-d wrote:
>> On Tuesday, 30 October 2018 at 18:38:48 UTC, Jonathan M Davis
>>
>> wrote:
>> > But shared is a low level construct and should stay that way. It's the base for everything else.
>>
>> This is simply not correct. People didn't stop having a need for threading while D was figuring out how to unbreak shared for ten years and counting. Fact is, the base for everything else is __gshared and synchronized, and it will *continue* to be __gshared and synchronized unless __gshared is removed and the Thread constructor is changed to only take shared delegates.
>>
>> Why not do something useful with synchronized instead?
>
> __gshared is intended for interacting with C globals and that's it. If it's used for anything else, it's not being used for its intended purpose. Certainly, using it for D objects is just begging for bugs. __gshared should _never_ be used for D objects. I know that some folks do, and sometimes folks get away with it, but it's just asking for trouble, and there's no guarantee that such code will continue to work, because it's violating compiler guarantees when it does so. __gshared is designed solely as a backdoor for dealing with C and that is it. On the whole, shared really is _not_ broken and _does_ work. Yes, it has some rough edges that need fixing, and not everything in core.sync has been updated to use it properly, necessitating more casts than should be necessary when using those constructs than should be necessary, but shared as a whole is sound and always has been. The biggest problem with it is that most folks don't understand how to use it properly.
>
>> > And to be honest, I don't really expect it to be reasonable to add much in the way of higher level mechanisms for sharing data to D at the language level - not with it being a systems language. Most stuff would require restricting it further and/or introducing a serious ownership model to it, which we really don't want to do. Even synchronized classes are pretty questionable, because they can only safely remove the outer layer of shared, which is arguably better than nothing, but it doesn't get you very far. Without a full-blown ownership model, we're pretty much reduced to manual casting when mutexes or synchronized are used, and the language really isn't going to be much help. If someone can come up with some bright ideas, great. All the more power to them. Maybe we can improve the situation. But again, they should be built on top of shared as a low level mechanism, not try to change what shared is.
>>
>> Sorry, but can you please actually comment on my proposal in concrete terms rather than handwave about how we're reduced to manual casting (which is not necessary for concurrency at the moment, so I'm not sure why you think it's unavoidable), and how the language surely cannot be of much help without even commenting on a proposal for how it *could* actually be of help?
>
> Manual casting is required if you're actually using shared with mutexes or synchronized. If you're using __gshared, it's not, but that's only because you're lying to the compiler about whether the variables are shared are not, which means that you're circumventing the type system, and you're risking subtle bugs in your programs, because the guarantees that the type system is suppose make aren't actually in place.
>
This does not match what I'm actually seeing in practice. If this is how it is, then this admonishment should be written in the D style guide in big, bolded letters.
Generally we try to write cast-free code if possible, so casting away shared feels wrong. That's not really an argument, of course, but to me semantic casting indicates "no compiler, you are mistaken about the properties of this data." The compiler should never *require* casting.

> Your synchronized field proposal is basically just a variation of synchronized classes with the extra caveat that somehow mutable references to the member variables can't escape, and for some reason, not everything in the class has to be synchronized.

Everything in the class has to be synchronized. I'm not sure where I said differently.

> It's like you're trying to have synchronized classes without talking about shared. But you can't take shared out of the mix. Per the type system, anything that isn't shared is considered thread-local by the compiler, and trying to do anything like this without shared would be a serious violation of the type system and the compiler's guarantees.
>

Again, imo this belongs in either the spec or the DStyle guide in big bold letters.

> Instead of talking about safely casting away the outer layer of shared inside the class, you're talking about preventing escaping. synchronized classes as described in TDPL already guaranteed that the outer layer can't escape, since it lives directly in the class. What they can't guarantee is that the rest can't escape, and the language doesn't provide any way for such a guarantee.

Not sure what you mean by "outer layer" here or this entire paragraph.

> scope (even with DIP 1000) isn't transitive, so it doesn't really do the trick. So, I don't know how we could guarantee that _nothing_ escapes, though if nothing escapes, then all levels of shared could be implicitly cast away instead of just the outer layer, which would be nice.
>
> Also, even if we _could_ somehow prevent anything from escaping, I would point out that mutability has nothing to do with this (or at least constness doesn't), because reading has many of the same threading issues as writing. In order to avoid threading issues, the data must actually be immutable. So, allowing const references to escape would still violate thread-safety. Access to data must be synchronized unless it is immutable - whether we're talking about reading or writing is irrelevant.

Indeed, const is useless for threading, which is why my proposal explicitly only notes immutable data as implicitly synchronized. We're on the same page.

> synchronized classes work (or would work if fully implemented) by guaranteeing that a single level can't escape - the one level that it _can_ guarantee can't escape - and thus are able to properly synchronize that level. If scope were transitive, then maybe we could do better, but since it isn't (and Walter insists that it can't reasonbly be transitive), I'm not sure that it's really reasonable to prevent escaping transitively.
>

The point of the proposal is not precisely to prevent escaping, but to prevent the escape of *data structures/references that offer the ability to mutate data unprotected.* Remember that in this model, the object is the unit of data ownership. So the two things that are allowed to escape are immutable data (and non-reference copied data), and synchronized classes, because they can guarantee that they're taking care of protecting their data.

> In any case, your proposal is just a variation of synchronized classes. So, while it may state it in a new way, the basic concept really isn't anything new.

I didn't say it was! I said it was *useful* and *straightforward*.

> And my point still stands that anything that we do with regards to thread safety must be built on top of shared, because that is the building block of sharing data across threads in D. Whether you like the specifics of shared or not, the idea that the compiler can rely on the assumption that data is _not_ shared across threads unless it's marked as such is a key feature of D. And attempting to use __gshared to get around it is just shooting yourself in the foot.
>
> - Jonathan M Davis


And yet, there's no way to even *state* that the Thread constructor must not take a delegate that can access unshared data, and none of the current proposals list one. Should we demand that only shared data can be passed to new threads? That's going to limit their usefulness. With synchronized, we can deprecate the delegate constructor, then pass the thread data as synchronized objects to a pure function, thereby actually guaranteeing thread safety.