On Tue, Jun 18, 2019 at 9:46 AM Manu <turkeyman@gmail.com> wrote:
>
> Is this valid?
>
> int x;
> void fun(scope ref shared(int) x) { ... }
> fun(x); // implicit promotion to shared in this case
>
> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
>
> With scope, we can guarantee that the reference doesn't escape the callee.
> Since the argument is local to the calling thread, and since the
> calling thread can not be running other code at the same time as the
> call is executing, there is no way for any code to execute with a
> thread-local assumption while the callee makes shared assumptions.
>
> I think this might be safe?

I guess this is the problem case; as is often the problem case:

int x;
void fun(scope ref shared(int) x, ref int y) { ... }
fun(x, x); // aliasing

On 18.06.19 02:49, Manu wrote:
> On Tue, Jun 18, 2019 at 9:46 AM Manu <turkeyman@gmail.com> wrote:
>>
>> Is this valid?
>>
>> int x;
>> void fun(scope ref shared(int) x) { ... }
>> fun(x); // implicit promotion to shared in this case
>>
>> This appears to promote a thread-local to shared. The problem with
>> such promotion is that it's not valid that a thread-local AND a shared
>> reference to the same thing can exist at the same time.
>>
>> With scope, we can guarantee that the reference doesn't escape the callee.
>> Since the argument is local to the calling thread, and since the
>> calling thread can not be running other code at the same time as the
>> call is executing, there is no way for any code to execute with a
>> thread-local assumption while the callee makes shared assumptions.
>>
>> I think this might be safe?
> 
> I guess this is the problem case; as is often the problem case:
> 
> int x;
> void fun(scope ref shared(int) x, ref int y) { ... }
> fun(x, x); // aliasing
> 

Walter is thinking about making it unsafe to pass two mutable pointers to the same object into a function by `ref`. For simple cases like your example, where x is a local variable, I think a slightly adapted version of the algorithm he has in mind would work too.

On 6/17/2019 7:03 PM, Timon Gehr wrote:
> Walter is thinking about making it unsafe to pass two mutable pointers to the same object into a function by `ref`. For simple cases like your example, where x is a local variable, I think a slightly adapted version of the algorithm he has in mind would work too.

I suggest instead making `x` shared. Then cast it to unshared when protected with a mutex.

All this trying to find ways to convert local references to shared make code very very difficult to reason about, even if you didn't make a mistake. Worse, making them "implicit" conversions mean they can lie hidden in the code, so you don't even know it's happening.

On Tuesday, 18 June 2019 at 02:54:10 UTC, Walter Bright wrote:
> I suggest instead making `x` shared. Then cast it to unshared when protected with a mutex.

Doesn't this imply implementing rust-style borrowing?
How do you ensure that a reference to unshared isn't retained after the mutex is released?

On Tue, Jun 18, 2019 at 4:30 PM Ola Fosheim Grøstad via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Tuesday, 18 June 2019 at 02:54:10 UTC, Walter Bright wrote:
> > I suggest instead making `x` shared. Then cast it to unshared when protected with a mutex.
>
> Doesn't this imply implementing rust-style borrowing?
> How do you ensure that a reference to unshared isn't retained
> after the mutex is released?

The access function returns a scoped lock object with a ref count. You may take many read ref's, a write access has a maximum of 1 ref. You may not take a read and a write lock at the same time.

On Tuesday, 18 June 2019 at 09:01:47 UTC, Manu wrote:
> The access function returns a scoped lock object with a ref count. You may take many read ref's, a write access has a maximum of 1 ref. You may not take a read and a write lock at the same time.

But, you could take the address of an attribute or object and retain it?  So this is only by convention? Or is the type system able to ensure this?

On Tuesday, June 18, 2019 2:57:20 PM MDT Ola Fosheim Grostad via Digitalmars-d wrote:
> On Tuesday, 18 June 2019 at 09:01:47 UTC, Manu wrote:
> > The access function returns a scoped lock object with a ref count. You may take many read ref's, a write access has a maximum of 1 ref. You may not take a read and a write lock at the same time.
>
> But, you could take the address of an attribute or object and retain it?  So this is only by convention? Or is the type system able to ensure this?

scope only makes guarantees for @safe code. As soon as anything is @system or @trusted, all bets are off. So, _if_ all of the code involved is @safe, then the address of the scope object couldn't be taken, and no references to the scope object could escape. However, all that's required is a piece of @trusted code with access to the scope object, and all protections go out the window. At that point, it's up to the programmer to ensure that the code doesn't violate the guarantees that the compiler expects @safe code to make - including whatever guarantess go along with scope in @safe code.

- Jonathan M Davis

On Tuesday, 18 June 2019 at 21:20:13 UTC, Jonathan M Davis wrote:
> escape. However, all that's required is a piece of @trusted code with access to the scope object, and all protections go out the window. At that point, it's up to the programmer to

So if I implement a container with @trusted method that grabs a reference to the object, which is reasonable, then it will fail? Or is there a "temporarily unshared" type that the container implementor could test for? Or is the argument that this would also fail for new/delete of reference counting so that you will try to reuse all the ref counting semantics on the shared level?

Is the idea to treat a grabbed lock as "generating a new object" and deal with it as refernce counting? Then you need to check that all locks have been released to avoid deadlocks/starvation...

This will only work in simple scenarios though...

On Tuesday, June 18, 2019 3:42:50 PM MDT Ola Fosheim Grostad via Digitalmars-d wrote:
> On Tuesday, 18 June 2019 at 21:20:13 UTC, Jonathan M Davis wrote:
> > escape. However, all that's required is a piece of @trusted code with access to the scope object, and all protections go out the window. At that point, it's up to the programmer to
>
> So if I implement a container with @trusted method that grabs a reference to the object, which is reasonable, then it will fail?

Storing a reference to a scope object in @trusted code would violate what scope is supposed to guarantee. So, such code would be badly written code just like @trusted code that screws up bounds-checking with pointer arithmetic is bad code. Temporarily taking the address of a scope object and doing something with it but making sure that no such references exist when the function exits could be reasonable in @trusted code, because the programmer would have ensured that the guarantees that scope is supposed to provide would have been met. So, you _can_ do stuff that scope doesn't allow which doesn't actually violate scope's guarantees, but if your @trusted code does anything which violates the guarantees that @safe is supposed to make, then the programmer who verified that it was okay to mark it as @trusted screwed up.

> Or is there a "temporarily unshared" type that the container implementor could test for? Or is the argument that this would also fail for new/delete of reference counting so that you will try to reuse all the ref counting semantics on the shared level?
>
> Is the idea to treat a grabbed lock as "generating a new object" and deal with it as refernce counting? Then you need to check that all locks have been released to avoid deadlocks/starvation...
>
> This will only work in simple scenarios though...

@safe really only deals with memory safety, not thread-safety. Converting between shared and thread-local does require a cast, which is @trusted, because you're basically stepping outside of the type system. At that point, the code involved has to then deal with the threading stuff properly, but as far as the type system is concerned, an object is either thread-local or shared, and you're on your own if you use casts to change something from one to the other even temporarily. @safe is only involved insofar as the cast is @system.

What Manu is proposing is a scenario where the type system is able to guarantee that no references to the variable escape and that based on that assumption, temporarily converting to shared wouldn't violate the guarantees that come with the object actually being thread-local. That _does_ rely on the code then either all being @safe or that any @trusted bits are correctly vetted by the programmer to ensure that no references to the scope object actually escaped. If the programmer screws that up, then the implicit conversion to shared will have violated the guarantees that are supposed to go with the object being thread-local, and unlike now, the point of the conversion wouldn't be @trusted, because it would be done implicitly by the compiler based on the assumption that scope and @safe were properly upheld within the function being called (which is true of any @safe function involving scope except that normally that doesn't involve any conversions to or from shared unless the programmer does something @system to make it happen).

On the surface, what Manu is proposing _seems_ sound (assuming that any @trusted code involved is vetted properly), but as Walter points out, it's really easy to screw up threading stuff. And personally, I'm inclined to argue that we're better off having the point where a variable is converted to or from shared always be @system so that it's never invisible, since pretty much anything involving shared needs to be vetted. Also, normally, shared objects either deal with their thread-safety stuff internally (e.g. by having an internal mutex that locks appropriately when accessing the object's members), or they require that you deal with the thread synchronization stuff explicitly (e.g. by directly dealing with the mutex whenever the shared object needs to be accessed). Having a function that's expecting a shared variable be given a thread-local one seems off to me. That's basically what's required when passing objects across threads, but at that point, scope wouldn't be involved, and it would definitely be up to the programmer to make sure that no other references to the object exist before casting it to shared to pass it across threads. It could very well be that Manu has found a totally reasonable use case where it makes sense to temporarily convert a thread-local object to shared without letting it escape, but I'm still inclined to think that the conversion should be vetted by the programmer rather than being considered okay and done implicitly just because scope is involved.

Regardless, as discussed at dconf this year, D's memory model and the exact semantics of shared really need to be properly locked down before we start making changes like this. We know basically what shared is and how it works, and we even have some idea of some of the changes that need to be made (e.g. it's almost certainly the case that reading and writing shared variables needs to become illegal, thus requiring casts to do either, because neither is thread-safe without thread synchronization mechanisms that the programmer needs to put into place and which the compiler doesn't understand well enough to do any conversions for you), but not all of the details have been properly ironed out yet, and the devil is in the details.

- Jonathan M Davis

On Wed, Jun 19, 2019 at 7:00 AM Ola Fosheim Grostad via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Tuesday, 18 June 2019 at 09:01:47 UTC, Manu wrote:
> > The access function returns a scoped lock object with a ref count. You may take many read ref's, a write access has a maximum of 1 ref. You may not take a read and a write lock at the same time.
>
> But, you could take the address of an attribute or object and retain it?  So this is only by convention? Or is the type system able to ensure this?

`scope` has a lot to say in this whole space; it must prevent escaping references that are only intended to have a finite life. We may reach limitations with `scope` today and need to make improvements, but that's fine. Some limitations in this space are fine to work through... but even in lieu of watertight solutions, we can make working solutions which are very helpful and hard to break unless you deliberately go out of your way to do so.

The worst thing that happens is that we stop making progress because a limitation of this sort inhibits development on other axiis. We're often too timid, and it hurts D's velocity immensely.