I've followed the AGM a little and think, why not go with the "disable access to shared" approach, but instead of allowing it to be casted away, introduce a "locked scope" lock { } which need to be within a function and allow access to shared variables only within such blocks?
It's still up to the user to ensure that such locked blocks encapsulate the semantic blocks which need to be kept together to safely modify shared variables, but at least the compiler can make sure they are not modified outside locked blocks and that at any time only one such block is executed on any thread. And such blocks should be scarce so they can be examined carefully like trusted and everything build upon is verifiable safe.

On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes Scherkl wrote:
> I've followed the AGM a little and think, why not go with the "disable access to shared" approach, but instead of allowing it to be casted away, introduce a "locked scope" lock { } which need to be within a function and allow access to shared variables only within such blocks?
> It's still up to the user to ensure that such locked blocks encapsulate the semantic blocks which need to be kept together to safely modify shared variables, but at least the compiler can make sure they are not modified outside locked blocks and that at any time only one such block is executed on any thread. And such blocks should be scarce so they can be examined carefully like trusted and everything build upon is verifiable safe.

And leave shared in the useless state that it currently has?
+ introducing a new syncronized keyword? (locked)?

doesn't sound too compelling.

On Saturday, 11 May 2019 at 12:08:00 UTC, Stefan Koch wrote:
> On Saturday, 11 May 2019 at 09:51:41 UTC, Dominikus Dittes Scherkl wrote:
>> I've followed the AGM a little and think, why not go with the "disable access to shared" approach, but instead of allowing it to be casted away, introduce a "locked scope" lock { } which need to be within a function and allow access to shared variables only within such blocks?
>> It's still up to the user to ensure that such locked blocks encapsulate the semantic blocks which need to be kept together to safely modify shared variables, but at least the compiler can make sure they are not modified outside locked blocks and that at any time only one such block is executed on any thread. And such blocks should be scarce so they can be examined carefully like trusted and everything build upon is verifiable safe.
>
> And leave shared in the useless state that it currently has?
No, as I said: use the "access disabled" (outside of locked blocks) approach, as several people advocated for.

> + introducing a new synchronized keyword? (locked)?
Need not be a keyword. It's also possible to use Lock() and Unlock() functions, but they need to be present in the same scope, so that the compiler can check, that what was locked really gets unlocked again close by (to keep it easily parsable). Or the block reuses the shared keyword, just to not burn another word for identifiers...

This is to avoid to need a cast and trusted stuff. I think shared and trusted should not be conflated, so that they can use a different review process (e.g. someone specialized in looking for synchronization problems)
>
> doesn't sound too compelling.

On Saturday, May 11, 2019 3:51:41 AM MDT Dominikus Dittes Scherkl via Digitalmars-d wrote:
> I've followed the AGM a little and think, why not go with the
> "disable access to shared" approach, but instead of allowing it
> to be casted away, introduce a "locked scope" lock { } which need
> to be within a function and allow access to shared variables only
> within such blocks?
> It's still up to the user to ensure that such locked blocks
> encapsulate the semantic blocks which need to be kept together to
> safely modify shared variables, but at least the compiler can
> make sure they are not modified outside locked blocks and that at
> any time only one such block is executed on any thread. And such
> blocks should be scarce so they can be examined carefully like
> trusted and everything build upon is verifiable safe.

All that really would do is add a bit of extra syntax around locking a mutex and casting away shared. It doesn't really change anything with regards to what happens. The operations are the same, just wrapped. It's also arguably worse in that it implies that the compiler is actually making something safe for you, and it isn't. You're still doing the cast. It's still up to you to make sure that the concurrency aspects are sorted out correctly. It's just that what's actually happening is less obvious. I'd argue that we really don't want anything to just remove shared for us if the compiler can't actually guarantee that doing so is safe, which it can't do in this case.

Also, mutexes aren't the only concurrency mechanism. You have to deal with stuff like condition variables and semaphores. And even with mutexes, the actual code flow may not be as simple as

* lock mutex
* cast away shared
* do stuff as thread-local
* make sure no thread-local references to the data exist
* unlock mutex

That's the basic case and what your suggestion targets, but if your code flow needs to be even slightly different, it won't work, whereas casting gives a lot more freedom. It might make sense to add such a helper on top of casting, but casting is still needed.

Regardless, it's clear that Walter and Andrei want to fully look at the issue - with concurrency and memory model experts - to make sure that they're doing the right thing rather than simply doing something now that might be wrong. It's almost certainly the case that part of whatever we get will involve disabling reading from and writing to shared objects, and casting is pretty much going to have to be involved with that, but the details still need to be worked out, and working out all of those details (like the memory model) could very well affect what we get in unexpected ways.

It may ultimately make sense to add helper stuff that hides casts, but we really need to figure out the details of how shared really works before looking at adding more user-friendly helpers on top of it.

- Jonathan M Davis

On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:


> All that really would do is add a bit of extra syntax around locking a mutex and casting away shared.
No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.

> It doesn't really change anything with regards to what happens.
correct. It's a safe wrapper that actually make shared usefull in one way.

> Also, mutexes aren't the only concurrency mechanism. You have to deal with stuff like condition variables and semaphores. And even with mutexes, the actual code flow may not be as simple as
>
> * lock mutex
> * cast away shared
No, that's exactly NOT necessary anymore, because the compiler can deal with this in a safe and checked manner for this special case.
> * do stuff as thread-local
> * make sure no thread-local references to the data exist
> * unlock mutex

Of course. But that would still be possible if you are willing to cast and deal with trusted functions. But it would not be required anymore for the 99% of usecases where simple mutex would be enough.

>
> That's the basic case and what your suggestion targets, but if your code flow needs to be even slightly different, it won't work, whereas casting gives a lot more freedom. It might make sense to add such a helper on top of casting, but casting is still needed.
Yes and it would be far from my opinion to suggest otherwise.
Casting is always possible and need be in a systems language.
But its not safe, and shared should be possible to be used in at least one way without getting into unsafe territory, I think.
>
> It may ultimately make sense to add helper stuff that hides casts, but we really need to figure out the details of how shared really works before looking at adding more user-friendly helpers on top of it.
And that I think is simple: forbid access without cast (outside locked blocks). Its the best (and most KISS) idea that has come up so far - and adding locked blocks seems logical to me, to allow one safe way to use shared.

On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl via Digitalmars-d wrote:
> On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:
> > All that really would do is add a bit of extra syntax around locking a mutex and casting away shared.
>
> No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.

Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data.

The only proposal that we've had thus far that is actually able to make enough guarantees to safely remove shared is the synchronized classes proposal in TDPL (which has never been implemented). The compiler would only be able to remove shared at all with TDPL synchronized classes, because it would guarantee that no references to the data directly in the class escaped the class, and even then, it could only safely remove the outer layer of shared, because anything that isn't directly in the class could still have references to it elsewhere. So, TDPL synchronized classes would actually be very limited in usefulness, because they simply can't guarantee enough to strip much of shared away, and they're only able to do that much because of how restrictive they are. Something as free-form as simply indicating that a mutex is locked in a block of code is nowhere near enough to guarantee that accessing any variables in that code is thread-safe.

- Jonathan M Davis

On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
> On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl via Digitalmars-d wrote:
>> On Saturday, 11 May 2019 at 15:24:19 UTC, Jonathan M Davis wrote:
>> > All that really would do is add a bit of extra syntax around locking a mutex and casting away shared.
>>
>> No, it makes it possible to use mutex in safe functions without needing to cast and which the compiler CAN guarantee to really be safe.
>
> Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time.
But that piece of code is system, which explicitly allows you to shoot into your foot. If you want to stay safe, don't do that.

> And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code
Yes, the lock block need a list of vars that it allows to be modified

lock(var1, var2, ...)
{
}

two mutexes can only be executed at parallel if their parameter set is disjunct.


> And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist.
ok, so we need in addition that a reference to a shared var need not be lived beyond the end of the locked block or be immutable. Bad, but seems necessary.

On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
> On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl via Digitalmars-d wrote:
>> [...]
>
> Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data.
>
> [...]

I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers.

What I mean is that something like:

void doSharedStuff(scope shared(Foo) foo)
{
}

Will be able to safely lock/unlock foo and cast away shared'ness in the function's scope.
The compiler can provide guarantees here that foo will not escape.

Sounds like what Atila did https://forum.dlang.org/post/vocustlarsadbankufjk@forum.dlang.org

On Tuesday, May 14, 2019 8:32:45 AM MDT Radu via Digitalmars-d wrote:
> On Monday, 13 May 2019 at 16:20:30 UTC, Jonathan M Davis wrote:
> > On Monday, May 13, 2019 9:52:02 AM MDT Dominikus Dittes Scherkl
> >
> > via Digitalmars-d wrote:
> >> [...]
> >
> > Actually, it doesn't. All you've done is lock a mutex and cast away shared. There is no guarantee that that mutex is always used when that object is accessed. There could easily be another piece of code somewhere that casts away shared without locking anything at the same time. And even if this were the only mechanism for removing shared, you could easily use it with the same object and a completely different mutex in another piece of code, thereby making the mutex useless. The type system has no concept of ownership and no concept of a mutex being associated with any particular object aside from synchronized classes (and those don't even currently require that the mutex always be used). And even if the compiler could check that all uses of a particular variable were locked with a mutex, that still wouldn't be enough, because other references to the same data could exist. So, with the construct you've proposed, there's no way for the compiler to guarantee that no other thread is accessing the data at the same time. All it guarantees is that a mutex has been locked, not that it's actually protecting the data.
> >
> > [...]
>
> I had this idea for some time, not sure I can best articulated now, but I think a workable solution for shared is closely linked with the dip1000 - scoped pointers.
>
> What I mean is that something like:
>
> void doSharedStuff(scope shared(Foo) foo)
> {
> }
>
> Will be able to safely lock/unlock foo and cast away shared'ness
> in the function's scope.
> The compiler can provide guarantees here that foo will not escape.

Sure, it can guarantee that no reference will escape that function, but all that's required is that another reference to the same data exist elsewhere, and another thread could muck with the object while the mutex was locked. There's no question that helpers could be created which would help users avoid mistakes when casting when casting away shared, but the compiler can't actually make the guarantee that casting away shared is thread-safe.

- Jonathan M Davis