On Tue, Jun 18, 2019 at 11:30 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 6/17/19 7:46 PM, Manu wrote:
> > Is this valid?
> >
> > int x;
> > void fun(scope ref shared(int) x) { ... }
> > fun(x); // implicit promotion to shared in this case
> >
> > This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
> >
> > With scope, we can guarantee that the reference doesn't escape the callee.
> > Since the argument is local to the calling thread, and since the
> > calling thread can not be running other code at the same time as the
> > call is executing, there is no way for any code to execute with a
> > thread-local assumption while the callee makes shared assumptions.
> >
> > I think this might be safe?
> >
>
> Seems like it would be safe, but what is the use case for it? If you have a scope shared, which can't actually get shared with anything else, what does it buy you?

parallel foreach.

On Wed, Jun 19, 2019 at 12:15 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 18.06.19 15:29, Steven Schveighoffer wrote:
> > On 6/17/19 7:46 PM, Manu wrote:
> >> Is this valid?
> >>
> >> int x;
> >> void fun(scope ref shared(int) x) { ... }
> >> fun(x); // implicit promotion to shared in this case
> >>
> >> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
> >>
> >> With scope, we can guarantee that the reference doesn't escape the
> >> callee.
> >> Since the argument is local to the calling thread, and since the
> >> calling thread can not be running other code at the same time as the
> >> call is executing, there is no way for any code to execute with a
> >> thread-local assumption while the callee makes shared assumptions.
> >>
> >> I think this might be safe?
> >>
> >
> > Seems like it would be safe, but what is the use case for it? If you have a scope shared, which can't actually get shared with anything else, what does it buy you?
> >
> > -Steve
>
> He would write @system code that leaks the reference to other threads temporarily, but ensures those references disappear once the parallel part of the computation has finished. His use case is @safe parallel foreach without having to explicitly qualify the variables in the outer scope as `shared`:
>
> void sum(){ // note: just to illustrate the concept
>      int result=0;
>      foreach(i;iota(1000).parallel){
>          static assert(typeof(result)==shared(int));
>          result.atomic!"+="(i);
>      }
>      static assert(typeof(result)==int);
>      return result;
> }

👍

We must get here.

On Wed, Jun 19, 2019 at 12:20 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 18.06.19 16:14, Timon Gehr wrote:
> > On 18.06.19 16:13, Timon Gehr wrote:
> >>
> >> int sum(){ // note: just to illustrate the concept
> >>      int result=0;
> >>      foreach(i;iota(1000).parallel){
> >>          static assert(is(typeof(result)==shared(int)));
> >>          result.atomic!"+="(i);
> >>      }
> >>      static assert(is(typeof(result)==int));
> >>      return result;
> >> }
> >
> > Return type should have been int, of course...
>
> And there should have been is expressions. Getting tired from pushing back against all the nonsense. Just note that if you can make the above work with useless qualified capturing, you can do so with useful qualified capturing and this is so blatantly obvious that it causes me physical pain that Manu honestly does not see it.

I can't see it because the design specifically inhibits that static
assert in the loop body.
But I will revoke my opinion on this matter; arguing will only slow it
down. If you know how to make that *EXACT* code you wrote above work,
then we are done here and I will buy you a years supply of beer.

Coupled with removing read/write access from shared, that is everything I need to go away and leave you all alone.

On Wednesday, 19 June 2019 at 03:02:30 UTC, Manu wrote:
> On Wed, Jun 19, 2019 at 12:15 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>
>> On 18.06.19 15:29, Steven Schveighoffer wrote:
>> > On 6/17/19 7:46 PM, Manu wrote:
>> >> Is this valid?
>> >>
>> >> int x;
>> >> void fun(scope ref shared(int) x) { ... }
>> >> fun(x); // implicit promotion to shared in this case
>> >>
>> >> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
>> >>
>> >> With scope, we can guarantee that the reference doesn't escape the
>> >> callee.
>> >> Since the argument is local to the calling thread, and since the
>> >> calling thread can not be running other code at the same time as the
>> >> call is executing, there is no way for any code to execute with a
>> >> thread-local assumption while the callee makes shared assumptions.
>> >>
>> >> I think this might be safe?
>> >>
>> >
>> > Seems like it would be safe, but what is the use case for it? If you have a scope shared, which can't actually get shared with anything else, what does it buy you?
>> >
>> > -Steve
>>
>> He would write @system code that leaks the reference to other threads temporarily, but ensures those references disappear once the parallel part of the computation has finished. His use case is @safe parallel foreach without having to explicitly qualify the variables in the outer scope as `shared`:
>>
>> void sum(){ // note: just to illustrate the concept
>>      int result=0;
>>      foreach(i;iota(1000).parallel){
>>          static assert(typeof(result)==shared(int));
>>          result.atomic!"+="(i);
>>      }
>>      static assert(typeof(result)==int);
>>      return result;
>> }
>
> 👍
>
> We must get here.

How is it to know it is supposed to be shared though? opApply is already kind of a disaster with attributes.

On Wed, Jun 19, 2019 at 1:11 PM Manu <turkeyman@gmail.com> wrote:
>
> On Wed, Jun 19, 2019 at 12:20 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >
> > On 18.06.19 16:14, Timon Gehr wrote:
> > > On 18.06.19 16:13, Timon Gehr wrote:
> > >>
> > >> int sum(){ // note: just to illustrate the concept
> > >>      int result=0;
> > >>      foreach(i;iota(1000).parallel){
> > >>          static assert(is(typeof(result)==shared(int)));
> > >>          result.atomic!"+="(i);
> > >>      }
> > >>      static assert(is(typeof(result)==int));
> > >>      return result;
> > >> }
> > >
> > > Return type should have been int, of course...
> >
> > And there should have been is expressions. Getting tired from pushing back against all the nonsense. Just note that if you can make the above work with useless qualified capturing, you can do so with useful qualified capturing and this is so blatantly obvious that it causes me physical pain that Manu honestly does not see it.
>
> I can't see it because the design specifically inhibits that static
> assert in the loop body.
> But I will revoke my opinion on this matter; arguing will only slow it
> down. If you know how to make that *EXACT* code you wrote above work,
> then we are done here and I will buy you a years supply of beer.
>
> Coupled with removing read/write access from shared, that is everything I need to go away and leave you all alone.

I'll just tweak your example one more little bit to make sure absolutely everything I care about is captured correctly:

struct S {
    int result;
    void inc(int i) shared { result.atomic!"+="(i); }
}
int sum(){
    S s;
    foreach(i; iota(1000).parallel){
        static assert(is(typeof(s) == shared(S)));
        s.inc(i);
    }
    static assert(is(typeof(s) == S));
    return s.result;
}

The combinations of primitives at work here will solve every challenge I'm aware of in our engine.

On Wed, Jun 19, 2019 at 1:15 PM Exil via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On Wednesday, 19 June 2019 at 03:02:30 UTC, Manu wrote:
> > On Wed, Jun 19, 2019 at 12:15 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 18.06.19 15:29, Steven Schveighoffer wrote:
> >> > On 6/17/19 7:46 PM, Manu wrote:
> >> >> Is this valid?
> >> >>
> >> >> int x;
> >> >> void fun(scope ref shared(int) x) { ... }
> >> >> fun(x); // implicit promotion to shared in this case
> >> >>
> >> >> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
> >> >>
> >> >> With scope, we can guarantee that the reference doesn't
> >> >> escape the
> >> >> callee.
> >> >> Since the argument is local to the calling thread, and
> >> >> since the
> >> >> calling thread can not be running other code at the same
> >> >> time as the
> >> >> call is executing, there is no way for any code to execute
> >> >> with a
> >> >> thread-local assumption while the callee makes shared
> >> >> assumptions.
> >> >>
> >> >> I think this might be safe?
> >> >>
> >> >
> >> > Seems like it would be safe, but what is the use case for it? If you have a scope shared, which can't actually get shared with anything else, what does it buy you?
> >> >
> >> > -Steve
> >>
> >> He would write @system code that leaks the reference to other threads temporarily, but ensures those references disappear once the parallel part of the computation has finished. His use case is @safe parallel foreach without having to explicitly qualify the variables in the outer scope as `shared`:
> >>
> >> void sum(){ // note: just to illustrate the concept
> >>      int result=0;
> >>      foreach(i;iota(1000).parallel){
> >>          static assert(typeof(result)==shared(int));
> >>          result.atomic!"+="(i);
> >>      }
> >>      static assert(typeof(result)==int);
> >>      return result;
> >> }
> >
> > 👍
> >
> > We must get here.
>
> How is it to know it is supposed to be shared though? opApply is already kind of a disaster with attributes.
>

opApply receives a delegate; you would infer the delegate qualifier to the
lambda as it already does for the loop counters from the arguments.
Ie:
 opApply(scope void delegate() shared) <- infer the shared qualifier to the
foreach body lambda.

On Wed, Jun 19, 2019 at 1:24 PM Manu <turkeyman@gmail.com> wrote:

> On Wed, Jun 19, 2019 at 1:15 PM Exil via Digitalmars-d < digitalmars-d@puremagic.com> wrote:
>
>> On Wednesday, 19 June 2019 at 03:02:30 UTC, Manu wrote:
>> > On Wed, Jun 19, 2019 at 12:15 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> >>
>> >> On 18.06.19 15:29, Steven Schveighoffer wrote:
>> >> > On 6/17/19 7:46 PM, Manu wrote:
>> >> >> Is this valid?
>> >> >>
>> >> >> int x;
>> >> >> void fun(scope ref shared(int) x) { ... }
>> >> >> fun(x); // implicit promotion to shared in this case
>> >> >>
>> >> >> This appears to promote a thread-local to shared. The problem with such promotion is that it's not valid that a thread-local AND a shared reference to the same thing can exist at the same time.
>> >> >>
>> >> >> With scope, we can guarantee that the reference doesn't
>> >> >> escape the
>> >> >> callee.
>> >> >> Since the argument is local to the calling thread, and
>> >> >> since the
>> >> >> calling thread can not be running other code at the same
>> >> >> time as the
>> >> >> call is executing, there is no way for any code to execute
>> >> >> with a
>> >> >> thread-local assumption while the callee makes shared
>> >> >> assumptions.
>> >> >>
>> >> >> I think this might be safe?
>> >> >>
>> >> >
>> >> > Seems like it would be safe, but what is the use case for it? If you have a scope shared, which can't actually get shared with anything else, what does it buy you?
>> >> >
>> >> > -Steve
>> >>
>> >> He would write @system code that leaks the reference to other threads temporarily, but ensures those references disappear once the parallel part of the computation has finished. His use case is @safe parallel foreach without having to explicitly qualify the variables in the outer scope as `shared`:
>> >>
>> >> void sum(){ // note: just to illustrate the concept
>> >>      int result=0;
>> >>      foreach(i;iota(1000).parallel){
>> >>          static assert(typeof(result)==shared(int));
>> >>          result.atomic!"+="(i);
>> >>      }
>> >>      static assert(typeof(result)==int);
>> >>      return result;
>> >> }
>> >
>> > 👍
>> >
>> > We must get here.
>>
>> How is it to know it is supposed to be shared though? opApply is already kind of a disaster with attributes.
>>
>
> opApply receives a delegate; you would infer the delegate qualifier to the
> lambda as it already does for the loop counters from the arguments.
> Ie:
>  opApply(scope void delegate() shared) <- infer the shared qualifier to
> the foreach body lambda.
>

The delegate is effectively a prototype for the lambda, it should be used verbatim.

On 6/18/2019 4:05 PM, Steven Schveighoffer wrote:
> [...]
The rude remarks aren't acceptable here.

On 6/17/2019 4:46 PM, Manu wrote:
> Is this valid?
> 
> int x;
> void fun(scope ref shared(int) x) { ... }
> fun(x); // implicit promotion to shared in this case
> 
> This appears to promote a thread-local to shared. The problem with
> such promotion is that it's not valid that a thread-local AND a shared
> reference to the same thing can exist at the same time.
> 
> With scope, we can guarantee that the reference doesn't escape the callee.
> Since the argument is local to the calling thread, and since the
> calling thread can not be running other code at the same time as the
> call is executing, there is no way for any code to execute with a
> thread-local assumption while the callee makes shared assumptions.
> 
> I think this might be safe?
> 

Using the resulting value of x:

  int x;
  void fun(scope ref shared(int) x) { ... }
  fun(x); // implicit promotion to shared in this case
  int y = x; // add this line

And now there's a problem. The compiler thinks x is thread local, so it doesn't add synchronization to the read of x. Not adding synchronization means that although fun() has terminated all the threads it spawned accessing x, it does not mean the memory caches of x are updated.

While you may have coded fun() to update the caches of x before returning, the compiler can't know that, and so the compiler cannot allow the implicit conversion.

In other words, adding `scope` does not guarantee SC-DRF (Sequential Consistency - Data Race Free).

On 19/06/2019 3:20 PM, Manu wrote:
> I'll just tweak your example one more little bit to make sure
> absolutely everything I care about is captured correctly:
> 
> struct S {
>      int result;
>      void inc(int i) shared { result.atomic!"+="(i); }
> }
> int sum(){
>      S s;
>      foreach(i; iota(1000).parallel){
>          static assert(is(typeof(s) == shared(S)));
>          s.inc(i);
>      }
>      static assert(is(typeof(s) == S));
>      return s.result;
> }
> 
> The combinations of primitives at work here will solve every challenge
> I'm aware of in our engine.

Okay is everyone in agreement that this is a use case for D?
If so, lets get this into bugzilla!