On Monday, 17 June 2019 at 11:24:59 UTC, Nicholas Wilson wrote:
> On Monday, 17 June 2019 at 10:20:29 UTC, Ola Fosheim Grøstad
>> In C++ it is an add-on, isn't it?
>
> It's a standard library function.

Ah yes, in C++17 it is. I was thinking of OpenMP, but I guess the OMP pragma can be wrapped up in the std lib API now.

> There is also the issue that access through the (once its fixed) qualified captured context, then cannot implicitly convert to shared (as oppose to const since everything will implicitly convert to const). This will be alleviated under the new shared.

Thanks for the explanation. So my understanding is that this will all be fixed when the semantics of "shared" has been finally decided on?

On Monday, 17 June 2019 at 11:38:59 UTC, Ola Fosheim Grøstad wrote:
>> There is also the issue that access through the (once its fixed) qualified captured context, then cannot implicitly convert to shared (as oppose to const since everything will implicitly convert to const). This will be alleviated under the new shared.
>
> Thanks for the explanation. So my understanding is that this will all be fixed when the semantics of "shared" has been finally decided on?

The qualified context thing should be fixed ASAP (unfortunately the compiler rabbit hole is rather deep...), since it is objectively broken, not just with shared, but const and immutable as well.

The nicely capturing unshared locals as shared within the local function will indeed have to wait until shared is respecified. The way I think it will go is to add a `-preview=shared` and put the changes behind that. Probably starting with disabling read and write access to shared variables without casting as this was pretty much universally agreed upon at dconf as a necessary, if not sufficient, condition for thread safety.

Manu (and maybe others) will then evaluate the model and see what breaks or is missing and then we repeat the design cycle.

There is also a (eventually series of) dlang.org PRs to define a memory model which will define more of the semantics of shared and multithreading.

On Monday, 17 June 2019 at 11:55:07 UTC, Nicholas Wilson wrote:
> starting with disabling read and write access to shared variables without casting as this was pretty much universally agreed upon at dconf as a necessary, if not sufficient, condition for thread safety.
>
+1 This is the way to go!

Please link me to your work in progress I'll try to help.

On 17.06.19 12:18, Nicholas Wilson wrote:
> 
> That is all moot if mutable promotes to shared under "New Shared"™,

I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one.

What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`.

threadsafe(int) <- inaccessible
threadsafe(shared(int)) <- same as shared(int), accessible

The "New Shared" vision is to remove type qualifier support for `shared` and to instead move `shared` into druntime in a restricted form. (Last time I discussed this with Manu, I believe he was adamant that the language shouldn't distinguish between shared and unshared data at all, so we had a long unproductive debate.)
Then `shared` is repurposed to mean something completely different.


> I don't think anybody disagrees that qualified local functions should work.

Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!

> It would be useful to decouple this from the the current discussion to avoid derailing.

I think it's related to why Manu is so confused.
Basically, there are two cases for local context qualifiers:

1. can implicitly promote mutable variable (`const`, `threadsafe`/"New Shared")
2. cannot implicitly promote mutable variable (`immutable`, (old) `shared`).

So far, Manu only understands case 1, because there is a crutch: it can be interpreted by creating a local context struct containing all local variables in the stack frame, of which the local function is a method. But case 2 exists even if we move from `shared` to `threadsafe`. Because Manu only understands case 1, he considers the fact that `shared` -> `threadsafe` /"New Shared" is a move from case 2 to case 1 as evidence that old shared is broken while `threadsafe`/"New Shared" is not.

This is the argument that is complete nonsense that I am so annoyed about, because it leads to `immutable` capturing not working for no reason at all! Skip step 1, where you break local function qualifiers.

On Mon, Jun 17, 2019 at 8:25 PM Ola Fosheim Grøstad via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On Monday, 17 June 2019 at 09:47:30 UTC, Manu wrote:
> > I mean, if it comes to that... I really hope not.
> > It would make it 100x harder to make the case to my company if
> > I start
> > telling them that I couldn't produce a working demo without
> > forking the
> > language :/
>
> Is parallell for mandatory though? In C++ it is an add-on, isn't it?

Define 'mandatory'? ...because we currently use C++. But it would be a
powerful enabler, and one of the very specific challenges we have with
C++ implementations.
I mean, our project needs parallel-for without any question, and the
smooth implementation that D teases us with is very attractive!
If I hack some crappy implementation using meta and lambdas, then I
have written EXACTLY what you write in C++, and there is no case to be
made. I need to show advantages of D, but parity.
I have an unusual opportunity to make a very strong case for D, but I
can only do that with a great showcase. I spent hours in a meeting the
other day arguing various terribly ways to implement it in C++, and
the take-away from the meeting was basically "well I guess we have to
choose this crappy compromise", which mostly revolve around naming
conventions.

Everything I need to make a strong case revolves around shared; and
since nobody on this forum seems to know what to do with shared (we've
been arguing about it and doing exactly nothing for the 10 years I've
been watching), I think I'm uniquely positioned to move the bar
forward here and make something that's actually useful, and
demonstrated what shared is actually for.
Reality is, most people don't write highly multi-threaded code, maybe
they spin a worker thread or something, but that's not what the future
looks like.

> Anyway, I am back to toying with creating a unicode-syntax. Stuff like inheriting structs could easily be handled on the syntactical level, so it does not require a fork, but things like parallell for has to be done at the semantic level.

Inheriting structs doesn't block anything... I just started that thread to re-iterate how unbelievably tired I am of not having that tool. I hate the struct inheritance workaround so much.

On Monday, 17 June 2019 at 20:11:32 UTC, Manu wrote:
> I think I'm uniquely positioned to move the bar
> forward here and make something that's actually useful, and
> demonstrated what shared is actually for.
> Reality is, most people don't write highly multi-threaded code, maybe
> they spin a worker thread or something, but that's not what the future
> looks like.

Yes, I don't think many people write highly multi-threaded code. I think the model that is easiest to understand is that you have  a queue of work to be done and then just let worker threads grab work from that queue. More complicated structures require a lot of planning and it is difficult to be 100% certain that you never can get a deadlock as the program design evolves over time.

I think it is difficult to say what they future looks like, it involves many cores, I think, but perhaps also more local memory per core. So… maybe the actor model will take off at some point, but I don't know. I think the actor model is easier to grasp, so it might win out even if it adds overhead.

Right now there seems to be many different hardware mechanisms that provide parallel computation that are rather specialized. SIMD, FPGA, GPU, neural network oriented cores, etc.

On 17.06.19 08:15, Manu wrote:
>     Closures with an immutable-qualified context can only
>     capture immutable variables. What is so surprising about this?
> 
> 
> What's the use of that? Can you point me at any code like that?

No, I don't have in my head an index of all existing D code keyed on which features it is using (not even my own), but I know how to design a programming language correctly. One reason why I don't have a lot of examples demonstrating usage of qualifiers is that I avoid them as well as I can -- their implementation is broken, and currently I have no interest in dealing with all of those bugs.

If I spent an unreasonable amount of time I would probably be able to produce plausible examples, but this would be a waste of my time, because it is not what I am good at or enjoy doing.

It would be a lot more productive if you produced the examples and trusted me more when I explain the language design considerations. I also want the language to improve, this is why I push back against bad aspects of otherwise good ideas. It does not mean I oppose the general direction things are moving.

> Make a const local function if you want to stop it mutating stuff...

That doesn't stop others from mutating. The _only way_ to get a strongly pure delegate literal is to qualify its context immutable. I hope this is reason enough without a concrete usage example.

> what's interesting about an immutable local function?
> ...

You can call an immutable-qualified delegate only if the context pointer is immutable-qualified. If it is const-qualified, you can't do that without breaking the type system.

It would be so weird if the only way to make a strongly pure delegate would be to create a local `static struct`, explicitly capture the `immutable` variables in it's members, and take the address of an `immutable` member function of the resulting immutable `struct` instance.

Maybe this way of thinking about it helps: You don't capture the entire stack frame and the context does not point to the entire stack frame. It only points to the variables that were actually captured.


>      > inout is fine too, just like const.
> 
>     Absolutely not. You can't implicitly promote stuff to `inout`.
> 
> 
> What are you taking about? inout accepts mutable, const, or immutable. I don't know what you could mean?

What I meant is `inout` does not work "just like const". Maybe this is not what you meant to imply. Conceptually, `inout` is a polymorphic parameter. It represents an actual qualifier at each specific point in the program, you just don't know what it is. You can't implicitly convert a mutable variable into an `inout` variable, because it is possible that `inout` means `immutable` during the actual execution of the program.

> Context objects are mutable, unless called via another local function where they would have gained the respective qualifier, and that would interact with inout correctly, eliminating some existing false-mutability bugs.
> ...

I hate how complicated it is to explain why this is nonsense, especially because it is only a tangent in your post, but I'll bite. `inout` strikes again. (BTW: How much time do you spend thinking about your posts in this thread? I am wasting hours upon hours. I can't keep this up much longer, I have other things I need to do... Please start figuring out the holes in your suggestions on your own. Please be more careful with the assertions that you make.)

The issue with `inout` is that there is scope confusion: there can be multiple enclosing function scopes, all of which have an `inout` qualifier, and all of those different polymorphic parameters have the same name: "inout". Right now the language deals with this (somewhat unsuccessfully implemented in DMD!), by effectively disallowing nested `inout` functions (their `inout` is treated as being the same as that of the enclosing function).

Therefore, right now there is no way to "interact with inout correctly" in the way you suggest, the `inout` qualifier on a nested function cannot be instantiated with a concrete value, because this happens only once the _outermost_ function gets called.

Of course, you will now just say that this is all bollocks and we should change all of that too.

So let's summarize. What you are saying is:
1. Any local function can capture any local variable from an enclosing scope.

2. The type of those variables pick up the context qualifier of the local functions trough which they are captured.

3. `inout` local functions can be called from other local functions and their context `inout` is instantiated with the qualifier of the other local function.

But you can't actually qualify a `foo` context with `bar`s `inout`, because you would immediately get `inout` confusion. Of course, the compiler would still happily do it. So your proposal would lead to at least one `inout` bug:

int* foo(inout(int)* x)@safe{
    inout(int)* bar()inout{
        return x; // ok, type of x is inout(inout(int)*)
    }
    int* baz(){ return bar(); }
    return baz();
}

void main()@safe{
    immutable(int)* x=...;
    int* y=foo(x);
    // x and y alias, UB
    assert(x is y);
}

(The function `baz` is just to make it very explicit that this is what you propose, I think you agree that `foo` could return bar(dummy) directly with the same result.)

The right solution would actually be to have proper scoping for `inout`, where additionally you can have multiple names for `inout`, e.g. `inout!"foo"` and `inout!"bar"`. This would fix every `inout` false-mutability bug, with your weird context interpretation or without. But then the correct design is that a function with `inout` context would only be able to capture variables with the right type of `inout`, the one as which its context is qualified. Otherwise you couldn't call `inout` local functions.


>      > shared is the interesting one; we shouldn't be able to pass the
>      > capture to the function because Context* -> shared(Context)*, but we
>      > can start to talk about ways this can work.
>      > One way is that the promotion is actally perfectly valid! Because you
>      > are calling a local function from the local thread; so the shared
>      > function will have a shared reference to the local context only for
>      > the life of the function call, and when it returns, the shared
>      > reference must end with the function. We can do this by declaring the
>      > local function: `void localFun(scope shared(Context)* ctx);`
>      > A shared local function is valid so long as that function does NOT
>      > escape any of those shared references. It might want to attribute
>      > `return` too.
> 
>     This idea is not at all contingent on the nonsensical
> 
> 
> How is uniformity nonsense?

Probably I should have been more careful in my first post. I did explicitly say that the local function meaning could be useful for member functions and not the other way around, but I guess you ignored that and just concluded "non-uniform".

It's not really all that non-uniform, I can implement `immutable` capturing manually like this:

// version with closures:
int delegate()immutable bar(){
    immutable(int) x = 3;
    int foo()immutable{
        return x;
    }
    return &foo;
}
// version without closures:
int delegate()immutable bar(){
    struct StackFrame{
        immutable(int) x = 3;
    }
    auto theFrame=new StackFrame();
    struct Closure{
        int* xp;
        int foo()immutable{
            return *xp;
        }
    }
    return &new immutable(Closure)(&theFrame.x).foo;
}

Of course, the compiler will do with one heap allocation to get an equivalent result, which is currently impossible to do with member functions.

> Are you saying that qualified methods are nonsense?

Qualifying the `this` pointer is not nonsense, because you can actually get a value of that type. Qualifying the full stack frame is nonsense, because there is no way to create an instance of a qualified stack frame, except for qualifiers where you can implicitly promote an unqualified stack frame, and there is a very useful way to interpret qualified capturing, which is easy to understand (see above).

However, I do indeed dislike that there is no way to say a member function will only access `immutable` members:

struct S{
    immutable x;
    int y;
    int foo(){ // only accesses immutable data
        return x;
    }
}

void main(){
     // could work, but typesystem can't show it:
    int delegate()immutable dg=&new S(1,2).foo;
}

There is a workaround:

struct T{
    int x;
    int foo()immutable{
        return x;
    }
}
struct S{
    immutable(T) t;
    int y;
    this(int x,int y){ t=T(x); y=y; }
}

void main(){
    int delegate()immutable dg=&new S(1,2).t.foo;
}

This workaround would also work in a version of the language that implements your set of rules (with `inout` bugs and all):

But I'd much rather write this than the above:

void main(){
    immutable(int) x=1;
    int y=2;
    int delegate()immutable dg=()=>x;
    writeln(dg());
}

With your suggested changes, this would not compile, for no benefit at all. It would stop qualifier inference for nested functions in its tracks.

> How could applying proven, uniform, and predictable rules be considered nonsense?

The nonsense is not in how you apply the rules (that part is fine, you correctly derive the consequences), it is in how you select the rules and how you justify selecting those rules.

> Can you show me how these special-case rules are useful?

I have argued above that there is no special case.

> Qualified local functions are extremely rare;

Because they don't work properly, which you have already noticed. E.g., above I should be allowed to return `int delegate()` instead of `int delegate()immutable`, but the compiler won't let me. It's full of bugs. I would use qualifiers if they worked, and I would use them more if I had to write multithreaded code.

> I suspect someone just dun some design that sounded cool and felt chuffed, but probably never used it.
> 
>     parts of your
>     vision, but I don't know if there is a simple way to make this work. It
>     will require more care.
> 
> 
> I agree there's certainly no simple way to do it assuming the current design; 

I have shown you how to manually construct `immutable` closures with nothing but structs and member functions. I hope that clarifies that this assertions is plain nonsense.

> I did try and imagine a solution extensively under the existing implementation for quite some time, but I don't believe it's possible without effectively transforming it towards the default semantics.
> ...which isn't actually surprising! You shouldn't be surprised that uniform semantics interact properly with all the other existing language. If the design just worked like any normal qualified this pointer, there'd be nothing to think about other than how to call the shared function, which I think is a problem we can defeat. Everything else would work elegantly with no special code in the compiler.
> ...

There would be the same amount of special code in the compiler, because stack frames are not structs.

>      >>From there, parallel foreach can be implemented as a @trusted
>     function.
>      >
> 
>     So your evil master plan is:
> 
>     1. Make qualified local functions useless.
> 
> 
> Not at all, const and inout work as expected

I agree: `const` works fine, `inout` continues to be buggy, as expected.

> (surely the 99% case),

I hate 99% features.

> and reflective meta will actually work too.

When does it not work? (I am not doubting that it does not work, but I very much doubt it is because qualified closures are designed usefully.)

> Existing bugs are all eliminated instantly.

That's plain snake oil and completely untrue. Have you ever seen the DMD source code?

> `immutable` has no useful meaning anyway

Wtf.

> (just use const? The thing about local functions is that they're local. It's not API material),

Yes, it is. Qualified delegate types can appear in APIs. Your very first example in this thread is an example of how it is API material.

> shared is the only interesting case that's affected, and I can't imagine another way to make it useful.
> ...

I hope after reading this and the previous post you will notice that there is actually no blocker.

>     2. Make qualified local functions more useful.
> 
> 
> If you can do this without 1, then great.

I can.

> I couldn't imagine a solution. The problem is specifically the non-uniformity and special case rules.
> ...

Do you still feel that way now?

>     3. Profit.
> 
>     Just skip phase 1, please.
> 
> 
> I've asked a couple of times for a demonstration of how existing semantics are useful? It's certainly complex, non-uniform, and surprising.
> The closure has un-qualified objects in it despite the function qualification, which means typeof() doesn't work right, and derivative meta that depends on that is broken by consequence.
> ...

The closure does not have unqualified objects in it. Why is it a problem that it is not necessarily contiguous in memory because the compiler optimizes away allocations?

> Explain to me how qualifying the context makes it useless? How does it make anything that works right now stop working?

I have shown a few examples in this post and argued why they are useful. (E.g., strongly pure delegates.)

> I think that makes it uniform, only meaningfully affects shared (in a positive way), and removes a mountain of complex special case code.
> ...

Step 1 only removes functionality, it does not eliminate any blockers for anything, including your proposed redefinition of `shared` to mean `threadsafe`.

> But you know what, do what you want, I'll shut up... If it just gets fixed, I won't complain.

Having wasted a lot of time on this thread, it would still be great if you made an effort to understand my points.

On 17.06.19 22:11, Manu wrote:
> 
> Everything I need to make a strong case revolves around shared; and
> since nobody on this forum seems to know what to do with shared (we've
> been arguing about it and doing exactly nothing for the 10 years I've
> been watching),

Lack of activity doesn't imply lack of knowledge, it can just be lack of management.

There is too little activity for most issues with the language that require:
a) breaking a ton of code
b) adding a new data qualifier

It is worse if a) and b) are combined with a high specification and implementation effort and nobody who can spend enough time on the compiler has an in-depth understanding of what needs to be done in every corner of the language.

I know multiple options for what to do, but they involve either a) or b). So do your suggestions. This is why there hasn't been enough activity, it is not a lack of theoretical understanding from every single forum poster.

> I think I'm uniquely positioned to move the bar
> forward here and make something that's actually useful, and
> demonstrated what shared is actually for.

I think you have useful perspectives on the application side and the prioritization of issues to help adoption in your industry, but this does not always translate to useful insights on language design. Our conversations should involve more of you talking about important applications and me talking about language design.

Of course, this is all useless unless someone writes DIPs and someone implements them.

On Tue, Jun 18, 2019 at 5:40 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 17.06.19 12:18, Nicholas Wilson wrote:
> >
> > That is all moot if mutable promotes to shared under "New Shared"™,
>
> I was discussing the current language. And even then, with old shared as well as new shared, my interpretation is the correct one.
>
> What "New Shared" it is actually trying to do is to introduce a new `threadsafe` qualifier. This doesn't conflict with `shared`. Roughly speaking, `threadsafe` is to unshared and `shared` what `const` is to mutable and `immutable`.
>
> threadsafe(int) <- inaccessible
> threadsafe(shared(int)) <- same as shared(int), accessible

If this threadsafe concept is something that exists, I am not a party to any such conversation.

There was some discussion about promotion possibility (but that never talked about 'threadsafe'), and that was universally rejected.

Making shared inhibit read/write needs to happen regardless of anything else to work according to current rules.

> The "New Shared" vision is to remove type qualifier support for `shared`
> and to instead move `shared` into druntime in a restricted form. (Last
> time I discussed this with Manu, I believe he was adamant that the
> language shouldn't distinguish between shared and unshared data at all,
> so we had a long unproductive debate.)
> Then `shared` is repurposed to mean something completely different.

I don't know what you're talking about.
Shared needs to have read/write access removed... I feel like that was
universally agreed.
>From there, I can get to work.

> > I don't think anybody disagrees that qualified local functions should work.
>
> Yes, they do. Manu said you shouldn't be able to call an `immutable`-qualified local function! This is extremely weird, because it is always valid to drop a context qualifier!
>
> > It would be useful to decouple this from the the current discussion to avoid derailing.
>
> I think it's related to why Manu is so confused.

I don't think I'm as confused as you'd like to think. I understand I'm proposing to obliterate a thing. It's a complex and surprising thing, and as far as I can tell, it's useless, and only a point of friction and bugs. Can you show how it's useful?

> Basically, there are two cases for local context qualifiers:
>
> 1. can implicitly promote mutable variable (`const`, `threadsafe`/"New
> Shared")
> 2. cannot implicitly promote mutable variable (`immutable`, (old) `shared`).
>
> So far, Manu only understands case 1, because there is a crutch: it can be interpreted by creating a local context struct containing all local variables in the stack frame, of which the local function is a method. But case 2 exists even if we move from `shared` to `threadsafe`. Because Manu only understands case 1, he considers the fact that `shared` -> `threadsafe` /"New Shared" is a move from case 2 to case 1 as evidence that old shared is broken while `threadsafe`/"New Shared" is not.
>
> This is the argument that is complete nonsense that I am so annoyed about, because it leads to `immutable` capturing not working for no reason at all! Skip step 1, where you break local function qualifiers.

Show me one example of an immutable local function in the wild; explain how it's useful? It's a meaningless concept; how can a functions capture be immutable? We have no language to qualify the capture, and certainly not immutable; the callstack is mutable.

But I said before, solve however you like.

On Tue, Jun 18, 2019 at 8:35 AM Timon Gehr via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
> On 17.06.19 22:11, Manu wrote:
> >
> > Everything I need to make a strong case revolves around shared; and since nobody on this forum seems to know what to do with shared (we've been arguing about it and doing exactly nothing for the 10 years I've been watching),
>
> Lack of activity doesn't imply lack of knowledge, it can just be lack of management.
>
> There is too little activity for most issues with the language that require:
> a) breaking a ton of code
> b) adding a new data qualifier
>
> It is worse if a) and b) are combined with a high specification and implementation effort and nobody who can spend enough time on the compiler has an in-depth understanding of what needs to be done in every corner of the language.
>
> I know multiple options for what to do, but they involve either a) or b). So do your suggestions. This is why there hasn't been enough activity, it is not a lack of theoretical understanding from every single forum poster.

It took a mammoth effort to arrive at agreement that shared can't
read/write. That's uncontroversial, trivial to implement, and won't
break anything that's not already critically broken.
Can we start there?

> > I think I'm uniquely positioned to move the bar
> > forward here and make something that's actually useful, and
> > demonstrated what shared is actually for.
>
> I think you have useful perspectives on the application side and the prioritization of issues to help adoption in your industry, but this does not always translate to useful insights on language design. Our conversations should involve more of you talking about important applications and me talking about language design.
>
> Of course, this is all useless unless someone writes DIPs and someone implements them.

Just write a patch. I will use a forked language and prove that it works or not. I would gladly never post in this forum again in my life; I hate this process more more than I can describe.