On 14.06.19 20:51, Manu wrote:
> On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
>>> Right, exactly... the compile error should be at the assignment
>>> of the
>>> not-shared closure to the shared delegate. The error would be
>>> something like "can't assign `Context*` to `shared(Context)*`".
>>> So, the function can certainly exist, but for shared, you
>>> should get
>>> an error when you attempt to call it passing the closure to the
>>> function.
>>
>> The shared function attribute means that its context is shared
>> similar to object methods, the error is an attempt to implicitly
>> leak unshared data into that shared context, that's why indeed
>> function itself is incorrect. Declare the variable as shared and
>> it will work.
> 
> No, you've misunderstood. The qualifier does NOT apply to the context
> as it should, that's the issue I'm reporting.

There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?

This is not the first time you do this. Sometimes, you spend some time on your own to form some partially-correct opinion and then you stick to it against all evidence to the contrary.

> Try it with const, it's simpler to understand.

That's because it's a completely different case.

> You can incorrectly
> mutate x from a const context.
> 
> A local function is essentially:
> void localFun(Context*, Args...);
> 
> A const local function should be:
> void localFun(const(Context)*, Args...) const  // <- const applies to
> the context pointer, as usual.
> ...

Yes, but this is not "as usual". In particular, this is not how it works for `immutable` and `shared`, because it makes no sense. With your suggestion it would be impossible to ever call an `immutable` or `shared` local function unless all data that is transitively reachable from the current stack frame is `immutable` or `shared`, respectively. That's clearly useless.

> That does not appear to be the case however.
> For example:
> 
> void test()
> {
>    int x;
>    void fun() const
>    {
>      pragma(msg, typeof(x)); // should print `const(int)` because
> typeof(const(Context*).x) == const(int), but it incorrectly prints
> `int`
>      ++x; // <- should be an error, but the context pointer is not
> const, so this compiles!
>    }
> }
> 
> The context pointer is missing the qualifier.
> 

Yes, in this instance, but your actual blocker is that there is no `shared` inference for local functions.

On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 14.06.19 20:51, Manu wrote:
> > On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
> >>> Right, exactly... the compile error should be at the assignment
> >>> of the
> >>> not-shared closure to the shared delegate. The error would be
> >>> something like "can't assign `Context*` to `shared(Context)*`".
> >>> So, the function can certainly exist, but for shared, you
> >>> should get
> >>> an error when you attempt to call it passing the closure to the
> >>> function.
> >>
> >> The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
> >
> > No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
>
> There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?
>
> This is not the first time you do this. Sometimes, you spend some time on your own to form some partially-correct opinion and then you stick to it against all evidence to the contrary.

Ok... Why would a function that receives a context object, which also explicitly qualifies the context, not apply the qualifier to the context?

> > Try it with const, it's simpler to understand.
>
> That's because it's a completely different case.

Not even remotely.

  Context*
  const(Context)*
  shared(Context)*

Qualifiers hare a regular and consistent semantic when applied to things.
There are no 'cases' here that I can see... what is the 'case' that I'm missing?

> > You can incorrectly
> > mutate x from a const context.
> >
> > A local function is essentially:
> > void localFun(Context*, Args...);
> >
> > A const local function should be:
> > void localFun(const(Context)*, Args...) const  // <- const applies to
> > the context pointer, as usual.
> > ...
>
> Yes, but this is not "as usual". In particular, this is not how it works for `immutable` and `shared`, because it makes no sense.

What do you mean? That's *exactly* how it works with immutable and
shared, and any qualifier applied to any context pointer.
A qualified method accepts a qualified context pointer, and expects
the calling code to supply a valid context argument. There's no magic
here, and there shouldn't be.

> With your
> suggestion it would be impossible to ever call an `immutable` or
> `shared` local function unless all data that is transitively reachable
> from the current stack frame is `immutable` or `shared`, respectively.

Yes, that is **EXACTLY** what my suggestion expects.
The calling code supplies a `Context*`, and all qualifiers other than
`const` would emit the predictable compile error because there is no
implicit conversion from `Context*` to `immutable(Context)*`, etc.
The function itself must still respect the fact that it received a
qualified context, or it's just lying.

> That's clearly useless.

It's not useless at all, because the compiler will still emit the
function, and I can call it by force.
What's _actually_ useless, is having the function accept qualifiers on
the definition that don't do anything. The user shouldn't qualify the
function if they don't want the context to be qualified as they
specify.

> > That does not appear to be the case however.
> > For example:
> >
> > void test()
> > {
> >    int x;
> >    void fun() const
> >    {
> >      pragma(msg, typeof(x)); // should print `const(int)` because
> > typeof(const(Context*).x) == const(int), but it incorrectly prints
> > `int`
> >      ++x; // <- should be an error, but the context pointer is not
> > const, so this compiles!
> >    }
> > }
> >
> > The context pointer is missing the qualifier.
> >
>
> Yes, in this instance, but your actual blocker is that there is no `shared` inference for local functions.

It's not 'inference', placing a qualifier after the function parameter
list specifies an absolute and explicit qualification of the context.
const and shared aren't different 'cases', qualifiers apply to the
context, and if that appears to be useless or invalid, or an obvious
compile error because implicit conversion is impossible, then that's
the expected behaviour.
In the case of shared; I can produce the appropriate calling context
and cast it. But I can't write a valid function right now, because the
context qualifier is disregarded.

On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 14.06.19 20:51, Manu wrote:
> > On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
> >>> Right, exactly... the compile error should be at the assignment
> >>> of the
> >>> not-shared closure to the shared delegate. The error would be
> >>> something like "can't assign `Context*` to `shared(Context)*`".
> >>> So, the function can certainly exist, but for shared, you
> >>> should get
> >>> an error when you attempt to call it passing the closure to the
> >>> function.
> >>
> >> The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
> >
> > No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
>
> There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?

So, what is actually going on? (I believe I'm aware what is going on,
because behaviour is self-evident).
And more importantly, how is it useful? Why is it so useful that it
should it violate default behaviour, and expectation?
I know how it's not useful.

On 15.06.19 04:40, Manu wrote:
> On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>>
>> On 14.06.19 20:51, Manu wrote:
>>> On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d
>>> <digitalmars-d@puremagic.com> wrote:
>>>>
>>>> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
>>>>> Right, exactly... the compile error should be at the assignment
>>>>> of the
>>>>> not-shared closure to the shared delegate. The error would be
>>>>> something like "can't assign `Context*` to `shared(Context)*`".
>>>>> So, the function can certainly exist, but for shared, you
>>>>> should get
>>>>> an error when you attempt to call it passing the closure to the
>>>>> function.
>>>>
>>>> The shared function attribute means that its context is shared
>>>> similar to object methods, the error is an attempt to implicitly
>>>> leak unshared data into that shared context, that's why indeed
>>>> function itself is incorrect. Declare the variable as shared and
>>>> it will work.
>>>
>>> No, you've misunderstood. The qualifier does NOT apply to the context
>>> as it should, that's the issue I'm reporting.
>>
>> There is a bug, but It appears you don't understand what it is. Why do
>> you insist on reporting the bug in a particular wrong way instead of
>> trying to understand what is actually going on?
> ...

(I'm going to ignore your other post, as it is just more rambling.)

Note that this part of the language is extremely poorly implemented in DMD right now. There are quite a few independent bugs plaguing local function and delegate qualifiers. But the fact that you can't access an unshared variable from a `shared` local function is by design; it is not one of those bugs. The fact that `const` on local functions does not work properly has no bearing on `shared` on local functions. DMD implements those things independently.

> So, what is actually going on?

(NOTE: I am first explaining the actual design here, not the buggy implementation. This would make your parallel `foreach` code compile. You can't argue against what is below on the basis that it is obviously nonsense because it prevents your parallel `foreach` from compiling.)

For member functions, the qualifiers affect the this pointer. I.e., you need to construct a differently-qualified receiver to call a `shared` or `immutable` member function. This is not the only behavior that would make sense. Qualifiers on member functions could also just restrict how that member function accesses other members.

For local functions, the qualifiers specify how the context is _accessed_, not how the entire thing is qualified. If your local function is `shared` that means the function may only access `shared` data.

void main(){
    int x; // not shared, yet part of stack frame
    shared(int) y;
    int foo()shared{
        // x=2; // error, x is not shared
        return y; // ok
    }
    // ...
}

Similarly, if your local function is `immutable`, it may only access `immutable` data. If your local function is `const`, it may only access `const` data, but as everything implicitly converts to `const`, it can actually access everything, it may just not modify it, accessed variables are `const`-qualified. (It's a bug in DMD that they are not.)

(Const is a special case because this is the only case where you can interpret what is going on as slapping the `const` qualifier on the entire context. It is not necessarily the best way to think about what is going on.)

This is the way this was intended to work, but apparently it was never fully implemented, leaving behind quite a few type system holes, for example:

void main(){
    int x;
    void bar(){
        x=2;
    }
    void foo()shared{
        bar(); // this shouldn't compile, but it does
    }
}

Local functions should _infer_ those attributes. So if your function (like the implicitly-generated lambda representing your `foreach` body), only accesses `shared` variables, it should be automatically `shared`-qualified.

> (I believe I'm aware what is going on, because behaviour is self-evident).

(You have demonstrated that this is not the case.)

> And more importantly, how is it useful? 

You have argued that there shouldn't be any way to call `shared`-qualified local functions. I don't understand how you can possibly think that behavior is useful. It would preclude your parallel `foreach` code from working!

The intended design is useful in the sense that it would make your parallel `foreach` code work in a compiler-checked thread safe way, because the compiler would simply automatically check that the loop body only accesses shared variables (or variables local to the loop body, of course).

> Why is it so useful that it should it violate default behaviour, and expectation?

It does not.

> I know how it's not useful.
> 

I am not defending the fact that your parallel `foreach` code does not compile! This has to be fixed.

But if it is fixed, and if D passes a shared delegate to your parallel `foreach`, where `shared` means it is @safe to call that delegate from other threads, how would that not be useful?

On Fri, Jun 14, 2019 at 9:10 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 15.06.19 04:40, Manu wrote:
> > On Fri, Jun 14, 2019 at 3:35 PM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>
> >> On 14.06.19 20:51, Manu wrote:
> >>> On Fri, Jun 14, 2019 at 8:05 AM Kagamin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> >>>>
> >>>> On Friday, 14 June 2019 at 09:04:42 UTC, Manu wrote:
> >>>>> Right, exactly... the compile error should be at the assignment
> >>>>> of the
> >>>>> not-shared closure to the shared delegate. The error would be
> >>>>> something like "can't assign `Context*` to `shared(Context)*`".
> >>>>> So, the function can certainly exist, but for shared, you
> >>>>> should get
> >>>>> an error when you attempt to call it passing the closure to the
> >>>>> function.
> >>>>
> >>>> The shared function attribute means that its context is shared similar to object methods, the error is an attempt to implicitly leak unshared data into that shared context, that's why indeed function itself is incorrect. Declare the variable as shared and it will work.
> >>>
> >>> No, you've misunderstood. The qualifier does NOT apply to the context as it should, that's the issue I'm reporting.
> >>
> >> There is a bug, but It appears you don't understand what it is. Why do you insist on reporting the bug in a particular wrong way instead of trying to understand what is actually going on?
> > ...
>
> (I'm going to ignore your other post, as it is just more rambling.)
>
> Note that this part of the language is extremely poorly implemented in DMD right now. There are quite a few independent bugs plaguing local function and delegate qualifiers. But the fact that you can't access an unshared variable from a `shared` local function is by design; it is not one of those bugs. The fact that `const` on local functions does not work properly has no bearing on `shared` on local functions. DMD implements those things independently.
>
> > So, what is actually going on?
>
> (NOTE: I am first explaining the actual design here, not the buggy implementation. This would make your parallel `foreach` code compile. You can't argue against what is below on the basis that it is obviously nonsense because it prevents your parallel `foreach` from compiling.)
>
> For member functions, the qualifiers affect the this pointer. I.e., you need to construct a differently-qualified receiver to call a `shared` or `immutable` member function. This is not the only behavior that would make sense. Qualifiers on member functions could also just restrict how that member function accesses other members.
>
> For local functions, the qualifiers specify how the context is
> _accessed_, not how the entire thing is qualified. If your local
> function is `shared` that means the function may only access `shared` data.
>
> void main(){
>      int x; // not shared, yet part of stack frame
>      shared(int) y;
>      int foo()shared{
>          // x=2; // error, x is not shared
>          return y; // ok
>      }
>      // ...
> }
>
> Similarly, if your local function is `immutable`, it may only access `immutable` data. If your local function is `const`, it may only access `const` data, but as everything implicitly converts to `const`, it can actually access everything, it may just not modify it, accessed variables are `const`-qualified. (It's a bug in DMD that they are not.)
>
> (Const is a special case because this is the only case where you can interpret what is going on as slapping the `const` qualifier on the entire context. It is not necessarily the best way to think about what is going on.)
>
> This is the way this was intended to work, but apparently it was never fully implemented, leaving behind quite a few type system holes, for example:
>
> void main(){
>      int x;
>      void bar(){
>          x=2;
>      }
>      void foo()shared{
>          bar(); // this shouldn't compile, but it does
>      }
> }
>
> Local functions should _infer_ those attributes. So if your function (like the implicitly-generated lambda representing your `foreach` body), only accesses `shared` variables, it should be automatically `shared`-qualified.
>
> > (I believe I'm aware what is going on, because behaviour is self-evident).
>
> (You have demonstrated that this is not the case.)

No, I genuinely understand. You've demonstrated that I know exactly
what was intended, and it's terrible.
With this semantic, it's true I can call the function, but the
function fundamentally doesn't work. There's no point making the
function callable if the function can't do anything. I need a useful
function to call before worrying about how to call it. We can get
there in better ways without a complex web of special-case rules.
This existing semantic expects that the context has shared objects
floating around to interact with. When have you ever declared a shared
object anywhere?

> > And more importantly, how is it useful?
>
> You have argued that there shouldn't be any way to call `shared`-qualified local functions. I don't understand how you can possibly think that behavior is useful. It would preclude your parallel `foreach` code from working!

I can't move on that front without fixing this first.
I have a clear plan to get there, but this design choice is blocking
progress. It seems pointless, can you show me any cases demonstrating
where this case-specific complexity is useful?

> The intended design is useful in the sense that it would make your parallel `foreach` code work in a compiler-checked thread safe way, because the compiler would simply automatically check that the loop body only accesses shared variables (or variables local to the loop body, of course).
>
> > Why is it so useful that it should it violate default behaviour, and expectation?
>
> It does not.

It does though, there is a completely different set of semantics
applied to a local function than to normal methods. That's extremely
surprising.
The weird thing is, both methods AND local functions can be assigned
to delegates... delegates do not distinguish between these 2 distinct
sets of semantics, and I believe that's why these
pipe-through-a-delegate-to-do-invalid-behaviour bugs exist. That would
require even more special-case code to correct.
A better design would be that no special case exists, and then no
special case corrections to special cases should exist either.
This is a bad design. Local functions context pointers should behave
identical to methods, and then delegates, and code that calls
delegates know what semantics they're dealing with.
A 'this' pointer is a 'this' pointer, don't make it more complicated than that.

> > I know how it's not useful.
> >
>
> I am not defending the fact that your parallel `foreach` code does not compile! This has to be fixed.
>
> But if it is fixed, and if D passes a shared delegate to your parallel `foreach`, where `shared` means it is @safe to call that delegate from other threads,

I don't believe that guarantee is possible. `shared` is fundamentally
unsafe, we've argued this for a long time.
It is possible to build safe tools with shared, but at the low level, it is not.

> how would that not be useful?

Because the function can't access anything.
This is the essential case:

struct S
{
  void method() shared;
}

void test()
{
  S s;
  void localFun() shared
  {
    s.method(); // <- s must transitively receive const from the
context pointer, otherwise this doesn't work
  }
}

Current semantics would reject this, because s is not shared, and
therefore not accessible by localFun.
The unnecessary complexity inhibits the only useful thing that a
shared function can do.

On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
> void test()
> {
>   int x;
>   void fun() const
>   {
>     pragma(msg, typeof(x)); // should print `const(int)` because
> typeof(const(Context*).x) == const(int), but it incorrectly prints
> `int`
>     ++x; // <- should be an error, but the context pointer is not
> const, so this compiles!
>   }
> }
>
> The context pointer is missing the qualifier.

It works for immutable qualifier, maybe type check there is too simple and fails to address const.

On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
> The context pointer is missing the qualifier.

I think it can be attributed to bug 1983, which Walter didn't want to fix, so anything that relies on it won't work properly any time soon.

On Fri, Jun 14, 2019 at 11:13 PM Manu <turkeyman@gmail.com> wrote:
>
> struct S
> {
>   void method() shared;
> }
>
> void test()
> {
>   S s;
>   void localFun() shared
>   {
>     s.method(); // <- s must transitively receive const from the
> context pointer, otherwise this doesn't work
>   }
> }

** s must transitively receive *shared* from the context

On 15.06.19 19:00, Kagamin wrote:
> On Friday, 14 June 2019 at 18:51:38 UTC, Manu wrote:
>> The context pointer is missing the qualifier.
> 
> I think it can be attributed to bug 1983,

I don't think this is the case, there are just a lot of different bugs around the same general theme because it was never properly implemented.

> which Walter didn't want to fix,

Do you have a link?

> so anything that relies on it won't work properly any time soon.

I wasn't able to find a pull request with the correct fix.

On Saturday, 15 June 2019 at 06:13:09 UTC, Manu wrote:
> struct S
> {
>   void method() shared;
> }
>
> void test()
> {
>   S s;
>   void localFun() shared
>   {
>     s.method(); // <- s must transitively receive const from the
> context pointer, otherwise this doesn't work
>   }
> }
>
> Current semantics would reject this, because s is not shared, and
> therefore not accessible by localFun.
> The unnecessary complexity inhibits the only useful thing that a
> shared function can do.

Declare `s` as shared and it will work. It would receive qualifier from context, but unshared variables can't be captured by shared closure to begin with because unshared data can't part of shared context, because that would break the guarantee that unshared data shouldn't be shared. Currently compiler implements check which variables can be captured and which can't.